Digital twinborn scene illumination dynamic tuning method and device, electronic equipment and medium

Abstract

According to the digital twinborn scene illumination dynamic tuning method and device, the electronic equipment and the medium provided by the invention, the digital twinborn scene is rendered through the plurality of illumination parameters of the image rendering engine, and the rendered digital twinborn scene is obtained; acquiring a virtual image in the rendered digital twin scene based on equipment acquisition parameters corresponding to the real image; weighting the pixel loss value and the structural similarity loss value between the real image and the virtual image to obtain a difference value between the virtual image and the real image; and if the difference value exceeds a preset difference threshold value, adjusting each illumination parameter based on the difference value to obtain a plurality of adjusted illumination parameters. The illumination parameters are adjusted through the difference value between the real image and the virtual image, automatic adjustment of the illumination parameters is achieved, it is ensured that the illumination effect of the digital twinborn scene is consistent with the illumination effect of the real scene, and the illumination parameter adjustment efficiency and precision of the digital twinborn scene are effectively improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, electronic device and medium for dynamic optimization of lighting in a digital twin scene. Background Technology

[0002] With the emergence of concepts such as smart cities, digital twins, and digital earth, urban construction in virtual scenes has entered an era of rapid development. To ensure that the display effects of objects in virtual scenes are consistent with those in real scenes, the lighting parameters used by Unreal Engine when rendering digital twin scenes are particularly important.

[0003] Because different scenes have different lighting parameters, when Unreal Engine renders different digital twin scenes, technicians need to manually adjust the lighting parameters of Unreal Engine. However, Unreal Engine has many lighting parameters when rendering digital twin scenes, and manual adjustment is not only inefficient, but also prone to low accuracy in adjusting the lighting parameters.

[0004] The low efficiency and accuracy of adjusting lighting parameters in digital twin scenarios is a problem that needs to be solved. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a method, apparatus, electronic device and medium for dynamic optimization of lighting in digital twin scenes, so as to solve the problem of low efficiency and accuracy of lighting parameter adjustment in digital twin scenes in related technologies.

[0006] In a first aspect, embodiments of this application provide a method for dynamic lighting optimization in a digital twin scene. The method is applied to an image rendering engine, adjusting the lighting parameters of the image rendering engine based on the difference in lighting effects between the real scene and the digital twin scene. The method includes:

[0007] Step S100: Obtain a real image of the real scene and the acquisition parameters of the device that acquires the real image; the real image is used to construct the digital twin scene;

[0008] Step S200: Render the digital twin scene based on multiple lighting parameters of the image rendering engine to obtain the rendered digital twin scene; the multiple lighting parameters include lighting parameters, anti-aliasing parameters, and material resolution;

[0009] Step S300: Based on the parameters collected by the device, obtain the virtual image corresponding to the real image in the rendered digital twin scene;

[0010] Step S400: Determine the difference value between the virtual image and the real image;

[0011] Step S500: If the difference value exceeds the preset difference threshold, then based on the difference value, each illumination parameter is adjusted to obtain multiple adjusted illumination parameters.

[0012] In one embodiment, adjusting the plurality of illumination parameters currently involves performing a t-th adjustment on the illumination parameters, where t is a positive integer. The adjustment of each illumination parameter based on the difference value yields the adjusted plurality of illumination parameters, including:

[0013] Step S510: Based on the adjusted multiple illumination parameters, repeat steps S200 to S500.

[0014] Step S520: Determine the first difference value between the virtual image and the real image at the (t-1)th adjustment, and the second difference value between the virtual image and the real image at the tth adjustment;

[0015] Step S530: Based on the rate of change between the first difference value and the second difference value, and the multiple lighting parameters after the (t-1)th adjustment, determine the multiple lighting parameters after the tth adjustment, until the difference value between the virtual image and the real image does not exceed the preset difference threshold.

[0016] In one embodiment, step S530 includes:

[0017] Step S531a: Determine the adjustment coefficient for each illumination parameter in the t-th adjustment;

[0018] Step S532a: Based on the rate of change between the first difference value and the second difference value and the adjustment coefficient of each illumination parameter in the t-th adjustment, determine the amount of change of each illumination parameter in the t-th adjustment;

[0019] Step S533a: Based on the multiple illumination parameters after the (t-1)th adjustment and the change of each illumination parameter in the tth adjustment, determine the multiple illumination parameters after the tth adjustment.

[0020] In one embodiment, step S531a includes:

[0021] Step S5311: Determine the gradient change rate of each illumination parameter after the (t-1)th adjustment;

[0022] Step S5312: Based on the gradient change rate of each illumination parameter after the (t-1)th adjustment, determine the adjustment coefficient of the corresponding illumination parameter in the tth adjustment.

[0023] In one embodiment, the digital twin scene is generated by reconstructing real images from multiple different perspectives;

[0024] Step S100 includes: acquiring a single-view real image of the real scene from each viewpoint, and the single-view device acquisition parameters corresponding to the single-view real image.

[0025] Step S300 includes: acquiring a single-view virtual image in the rendered digital twin scene based on the acquisition parameters of each single-view device;

[0026] Step S400 includes: Step S410, determining the single-view difference value between the single-view real image and the corresponding single-view virtual image under each viewpoint; Step S420, determining the global difference value between the virtual image and the real image based on the single-view weight value of each viewpoint and the corresponding single-view difference value.

[0027] Step S500 includes: if the global difference value exceeds the preset difference threshold, then based on all single-view difference values, each illumination parameter is adjusted to obtain multiple adjusted illumination parameters.

[0028] In one embodiment, step S420 further includes:

[0029] The corresponding single-view weight value is determined based on the proportion of each single-view difference value in the sum of all single-view difference values.

[0030] In one embodiment, step S520 includes: determining a first difference value of the target view between the single-view virtual image and the corresponding single-view real image under the target view at the (t-1)th adjustment, and a second difference value of the target view between the single-view virtual image and the corresponding single-view real image under the target view at the tth adjustment; the target view is any one of a plurality of different viewpoints;

[0031] Step S530 includes:

[0032] Step S531b: Determine the adjustment coefficient for each illumination parameter in the t-th adjustment;

[0033] Step S532b: Based on the rate of change between the first difference value of the target view and the second difference value of the target view and the single view weight value under the target view, determine the gradient value of the t-th adjustment under the target view;

[0034] Step S533b: Determine the global gradient value for the t-th adjustment based on the average gradient of the t-th adjustment across all views;

[0035] Step S534b: Based on the global gradient value of the t-th adjustment and the adjustment coefficient of each illumination parameter in the t-th adjustment, determine the amount of change of each illumination parameter in the t-th adjustment;

[0036] Step S535b: Based on the multiple illumination parameters after the (t-1)th adjustment and the change of each illumination parameter in the tth adjustment, determine the multiple illumination parameters after the tth adjustment.

[0037] In one embodiment, S400 includes:

[0038] Step S401: Determine the pixel loss value and structural similarity loss value between the virtual image and the real image, respectively;

[0039] Step S402: Based on the pixel loss value and the structural similarity loss value, determine the difference value between the virtual image and the real image.

[0040] Secondly, embodiments of this application provide a dynamic lighting optimization device for a digital twin scene. The device is applied to an image rendering engine and adjusts the lighting parameters of the image rendering engine based on the difference in lighting effects between the real scene and the digital twin scene. The device includes:

[0041] The first acquisition module is used to acquire real images of the real scene and acquisition parameters of the device that acquires the real images; the real images are used to construct the digital twin scene.

[0042] The rendering module is used to render the digital twin scene based on multiple lighting parameters of the image rendering engine to obtain the rendered digital twin scene; the multiple lighting parameters include lighting parameters, anti-aliasing parameters, and material resolution;

[0043] The second acquisition module acquires the virtual image corresponding to the real image in the rendered digital twin scene based on the parameters collected by the device.

[0044] A difference determination module is used to determine the difference value between the virtual image and the real image;

[0045] The parameter adjustment module is used to adjust each illumination parameter based on the difference value if the difference value exceeds a preset difference threshold, so as to obtain multiple adjusted illumination parameters.

[0046] Thirdly, embodiments of this application also provide an electronic device, including: a memory and a processor, wherein the memory stores a computer program executable by the processor, and when the computer program is executed by the processor, it performs the digital twin scene lighting dynamic optimization method as described in the first aspect above.

[0047] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, perform the digital twin scene lighting dynamic optimization method as described in the first aspect above.

[0048] In the above implementation process, the digital twin scene is rendered according to the lighting parameters in the image rendering engine to obtain the rendered digital twin scene. Then, a virtual image corresponding to the real image is acquired in the rendered digital twin scene based on the device acquisition parameters of the real scene. Further, it is determined whether the difference between the real image and the virtual image exceeds a preset difference threshold. If the difference exceeds the preset difference threshold, the lighting parameters in the image rendering engine are adjusted according to this difference, thereby achieving automatic adjustment of the lighting parameters. Under the adjusted lighting parameters, the difference between the virtual image and the real image does not exceed the preset difference threshold; that is, under the adjusted lighting parameters, the lighting effect of the digital twin scene is consistent with the lighting effect of the real scene. Furthermore, this application achieves the consistency of the lighting effect of the digital twin scene with the real scene by adjusting the lighting parameters of the image rendering engine, rather than adjusting the parameters of the image itself. While ensuring consistent lighting effects, it also ensures that the parameters of the constructed digital twin scene are consistent with those of the real scene. Attached Figure Description

[0049] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating a method for dynamic lighting optimization in a digital twin scene provided in an embodiment of this application.

[0051] Figure 2 This is a schematic diagram of the structure of a digital twin scene lighting dynamic optimization device provided in an embodiment of this application;

[0052] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0053] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this application.

[0055] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0056] With the emergence of concepts such as smart cities, digital twins, and digital earth, urban construction in virtual scenes has entered an era of rapid development. To ensure that the display effects of objects in virtual scenes are consistent with those in real scenes, the lighting parameters used by Unreal Engine when rendering digital twin scenes are particularly important.

[0057] Because different scenes have different lighting parameters, when Unreal Engine renders different digital twin scenes, technicians need to manually adjust the lighting parameters of Unreal Engine. However, Unreal Engine has many lighting parameters when rendering digital twin scenes, and manual adjustment is not only inefficient, but also prone to low accuracy in adjusting the lighting parameters.

[0058] The low efficiency and accuracy of adjusting lighting parameters in digital twin scenarios is a problem that needs to be solved.

[0059] Please see Figure 1 , Figure 1 This is a flowchart illustrating a method for dynamically optimizing lighting in a digital twin scene, as provided in an embodiment of this application. This method is applied to an image rendering engine. It primarily adjusts the lighting parameters of the image rendering engine based on the difference in lighting effects between the real scene and the digital twin scene. The method may include the following steps:

[0060] Step S100: Obtain the real image of the real scene and the acquisition parameters of the device that acquires the real image.

[0061] Real images are used to construct digital twin scenarios.

[0062] For example, a real image of a real scene can be acquired by an image acquisition device, which can be a drone. The real image can be an oblique photographic image acquired by the drone. Since the method of this application is mainly used to adjust the lighting parameters of the image rendering engine according to the difference in lighting effects between the real scene and the digital twin scene, the real scene is mainly a scene during the day when it is sunny.

[0063] The collected real images can then be used to construct a digital twin scene corresponding to the real scene. It should be noted that the digital twin scene can be constructed in an image rendering engine or in other virtual scene building software, without any restrictions.

[0064] Furthermore, the image rendering engine can acquire real images of the real scene, as well as the device acquisition parameters when the image acquisition device acquires the real image. Specifically, the device acquisition parameters can be external parameters of the image acquisition device, such as the position and orientation of the drone.

[0065] Step S200: Render the digital twin scene based on multiple lighting parameters of the image rendering engine to obtain the rendered digital twin scene.

[0066] Multiple lighting parameters include lighting parameters, anti-aliasing parameters, and material resolution.

[0067] For example, when rendering a digital twin scene, the image rendering engine needs to render the lighting effects of the digital twin scene using multiple lighting parameters within the image rendering engine. The image rendering engine typically includes initial values ​​for these lighting parameters. During the initial rendering of the digital twin scene, the image rendering engine can render the scene based on these initial values. Specifically, the multiple lighting parameters of the image rendering engine may include lighting and shadow parameters, anti-aliasing parameters, and material resolution, etc.

[0068] It should be noted that in this embodiment, only the lighting parameters are listed as light and shadow parameters, anti-aliasing parameters, and material resolution. In actual applications, the lighting parameters may also include parameters for adjusting natural light, sky light, volumetric clouds, fog, and clouds in the scene, but do not involve adjustments to the image material or the shadows themselves.

[0069] Step S300: Based on the device acquisition parameters, obtain the virtual image corresponding to the real image in the rendered digital twin scene.

[0070] For example, the image rendering engine may include a virtual image acquisition device. The image rendering engine can control the virtual image acquisition device to acquire images in the digital twin scene according to the device acquisition parameters of the real image acquisition device, so as to obtain a virtual image, thereby making the device acquisition parameters of the real image and the corresponding virtual image the same.

[0071] Step S400: Determine the difference between the virtual image and the real image.

[0072] Furthermore, the difference value between the virtual image and the corresponding real image is determined. Specifically, the difference value can be the pixel loss value and / or structural loss value between the virtual image and the real image.

[0073] Step S500: If the difference value exceeds the preset difference threshold, then each illumination parameter is adjusted based on the difference value to obtain multiple adjusted illumination parameters.

[0074] Furthermore, the difference value is compared with a preset difference value. If the difference value does not exceed the preset difference threshold, there is no need to adjust the lighting parameters in the image rendering engine. If the difference value exceeds the preset difference threshold, each lighting parameter is adjusted according to the difference value to obtain multiple adjusted lighting parameters. Specifically, when the difference value exceeds the preset difference threshold, each lighting parameter can be adjusted according to a preset adjustment value. The preset adjustment value corresponding to each lighting parameter can be the same or different.

[0075] It should be noted that the preset threshold can be set adaptively according to the actual situation. The preset threshold can be 0.2, 0.3, or other values, and there are no restrictions here.

[0076] If the lighting parameters are adjusted and the digital twin scene is rendered according to the adjusted lighting parameters, and steps S200 to S500 are executed repeatedly until the difference between the virtual image and the real image does not exceed the preset difference threshold.

[0077] In the above implementation process, the digital twin scene is rendered according to the lighting parameters in the image rendering engine to obtain the rendered digital twin scene. Then, a virtual image corresponding to the real image is acquired in the rendered digital twin scene based on the device acquisition parameters of the real scene. Further, it is determined whether the difference between the real image and the virtual image exceeds a preset difference threshold. If the difference exceeds the preset difference threshold, the lighting parameters in the image rendering engine are adjusted according to this difference, thereby achieving automatic adjustment of the lighting parameters. Under the adjusted lighting parameters, the difference between the virtual image and the real image does not exceed the preset difference threshold; that is, under the adjusted lighting parameters, the lighting effect of the digital twin scene is consistent with the lighting effect of the real scene. Furthermore, this application achieves the consistency of the lighting effect of the digital twin scene with the real scene by adjusting the lighting parameters of the image rendering engine, rather than adjusting the parameters of the image itself. While ensuring consistent lighting effects, it also ensures that the parameters of the constructed digital twin scene are consistent with those of the real scene, effectively reducing the performance consumption during the adjustment of the lighting effect of the digital twin scene and improving the efficiency of lighting parameter adjustment.

[0078] In one embodiment, adjusting multiple illumination parameters involves performing a t-th adjustment on the illumination parameters, where t is a positive integer. Based on the difference value, each illumination parameter is adjusted to obtain the adjusted multiple illumination parameters, including:

[0079] Step S510: Based on the adjusted multiple illumination parameters, repeat steps S200 to S500.

[0080] Step S520: Determine the first difference value between the virtual image and the real image at the (t-1)th adjustment, and the second difference value between the virtual image and the real image at the tth adjustment.

[0081] Step S530: Based on the rate of change between the first difference value and the second difference value, and the multiple lighting parameters after the (t-1)th adjustment, determine the multiple lighting parameters after the tth adjustment, until the difference value between the virtual image and the real image does not exceed the preset difference threshold.

[0082] For example, multiple illumination parameters can be adjusted multiple times to obtain multiple illumination parameters after each adjustment, and steps S200 to S500 can be executed cyclically.

[0083] If the current adjustment of multiple lighting parameters is to perform the t-th adjustment of the lighting parameters, when t is 1, an initial adjustment value is set for each lighting parameter. This initial adjustment value can be 0.001 or smaller, without restriction. Further, based on the initial value of each lighting parameter and its corresponding initial adjustment value, the lighting parameters are adjusted one by one to obtain the adjusted lighting parameters.

[0084] It should be noted that the initial adjustment value for each lighting parameter can be the same or different, and the size of the initial adjustment value can be adaptively set according to the actual situation, without any restrictions here.

[0085] When t is greater than 1, the digital twin model is rendered using the lighting parameters at the (t-1)th adjustment to obtain the digital twin scene after the (t-1)th adjustment. The virtual image at the (t-1)th adjustment is obtained from the digital twin scene after the (t-1)th adjustment based on the device acquisition parameters. Furthermore, the first difference value between the virtual image at the (t-1)th adjustment and the real image with the same device acquisition parameters is determined.

[0086] After adjusting the lighting parameters for the (t-1)th time, the lighting parameters after the (t-1)th adjustment can be obtained, which is the lighting parameters at the tth adjustment. The digital twin model is rendered using the lighting parameters at the tth adjustment to obtain the digital twin scene rendered at the tth adjustment. Based on the device acquisition parameters, the virtual image at the tth adjustment is obtained from the digital twin scene rendered at the tth adjustment. Furthermore, the second difference value between the virtual image at the tth adjustment and the real image with the same device acquisition parameters is determined.

[0087] Then, the rate of change between the first difference value and the second difference value is determined. Specifically, the rate of change corresponding to the target illumination parameter can be determined by the ratio of the difference between the first difference value and the second difference value to the change of the target illumination parameter at the (t-1)th time. The target illumination parameter is any one of multiple illumination parameters. Therefore, the rate of change corresponding to all illumination parameters can be determined in this way.

[0088] Furthermore, the change in the target illumination parameter at the tth time can be determined based on the rate of change corresponding to the target illumination parameter. Then, based on the change in the target illumination parameter at the tth time and the target illumination parameter after the (t-1)th adjustment, the target illumination parameter after the tth adjustment can be determined.

[0089] Similarly, this method can be used to determine all lighting parameters after the t-th adjustment, thereby achieving the t-th adjustment of the lighting parameters.

[0090] In the above implementation process, when the number of adjustments is greater than 1, the change amount of the current adjustment is determined based on the rate of change between the first difference value of the previous adjustment and the second difference value of the current adjustment. Then, the current illumination parameters are adjusted based on the change amount of the current adjustment, thereby realizing the adjustment of the current illumination parameters. Furthermore, adjusting the illumination parameters based on the rate of change of the difference values ​​between two consecutive adjustments can improve the accuracy of the adjustment.

[0091] In one embodiment, step S530 includes:

[0092] Step S531a: Determine the adjustment coefficient for each illumination parameter in the t-th adjustment.

[0093] Step S532a: Based on the rate of change between the first difference value and the second difference value and the adjustment coefficient of each illumination parameter in the tth adjustment, determine the amount of change of each illumination parameter in the tth adjustment.

[0094] Step S533a: Based on the multiple illumination parameters after the (t-1)th adjustment and the change of each illumination parameter in the tth adjustment, determine the multiple illumination parameters after the tth adjustment.

[0095] For example, each time the illumination parameters are adjusted, the amount of change in each adjustment can be corrected by adjusting the coefficient, thereby ensuring the uniformity of the adjustment gradient during cyclic adjustment.

[0096] Specifically, the adjustment coefficient for each illumination parameter in the t-th adjustment can be determined. Then, the product of the rate of change of the first difference value and the second difference value and the adjustment coefficient for each illumination parameter in the t-th adjustment is determined as the change amount of the corresponding illumination parameter in the t-th adjustment. Finally, based on the multiple illumination parameters after the (t-1)-th adjustment and the change amount of each illumination parameter in the t-th adjustment, the multiple illumination parameters after the t-th adjustment are determined.

[0097] In the above implementation process, each time the illumination parameters are adjusted, the amount of change in each adjustment can be corrected by adjusting the coefficient, thereby ensuring the uniformity of the adjustment gradient during the cyclic adjustment process.

[0098] In one embodiment, step S531a includes:

[0099] Step S5311: Determine the gradient change rate of each illumination parameter after the (t-1)th adjustment.

[0100] Step S5312: Based on the gradient change rate of each illumination parameter after the (t-1)th adjustment, determine the adjustment coefficient of the corresponding illumination parameter in the tth adjustment.

[0101] For example, during the cyclic adjustment of the illumination parameters, the change of each illumination parameter in the first t-1 adjustments is determined, the slope of the change of the corresponding illumination parameter after the t-1 adjustment is determined based on the change in the first t-1 adjustments, and the slope is determined as the gradient change rate of the corresponding illumination parameter after the t-1 adjustment.

[0102] Furthermore, the gradient change rate of each illumination parameter after the (t-1)th adjustment is determined as the adjustment coefficient of the corresponding illumination parameter in the tth adjustment. Specifically, the adaptive dynamic adjustment of the adjustment coefficient can be achieved through the Adam (Adaptive Moment Estimation) algorithm and / or the RMSProp (root mean square propagation) algorithm.

[0103] In the above implementation process, the adjustment coefficient of the corresponding illumination parameter in the tth adjustment is determined by the gradient change rate of each illumination parameter after the (t-1)th adjustment, thereby realizing the adaptive adjustment of the adjustment coefficient.

[0104] In one embodiment, the digital twin scene is generated by reconstructing real images from multiple different perspectives.

[0105] Step S100 includes: acquiring a single-view real image of the real scene from each viewpoint, and the single-view device acquisition parameters corresponding to the single-view real image.

[0106] Step S300 includes: acquiring a single-view virtual image in the rendered digital twin scene based on the acquisition parameters of each single-view device.

[0107] Step S400 includes: Step S410, determining the single-view difference value between the single-view real image and the corresponding single-view virtual image under each viewpoint; Step S420, determining the global difference value between the virtual image and the real image based on the single-view weight value of each viewpoint and the corresponding single-view difference value.

[0108] Step S500 includes: if the global difference value exceeds the preset difference threshold, then based on all single-view difference values, each illumination parameter is adjusted to obtain multiple adjusted illumination parameters.

[0109] For example, when constructing a digital twin scene, it can be constructed using a single panoramic image or using real images of different regions from different perspectives. Therefore, the real images obtained by the image rendering engine can include real images from multiple different perspectives, and these multiple perspectives can form a global view of the real scene.

[0110] Specifically, the image rendering engine can obtain single-view real images of the real scene from each perspective, as well as the single-view device acquisition parameters corresponding to each single-view real image.

[0111] After constructing a digital twin model based on multiple single-view real images, the overall digital twin scene can be rendered using an image rendering engine to obtain the rendered digital twin scene. Then, based on the single-view device acquisition parameters corresponding to each single-view real image, a virtual image acquisition device is controlled to obtain the corresponding single-view virtual image from the rendered digital twin scene.

[0112] Then, the single-view difference value between the single-view real image and the corresponding single-view virtual image under each viewpoint is determined, and the global difference value between the virtual image and the real image is determined according to the single-view weight value of each viewpoint and the corresponding single-view difference value.

[0113] Specifically, suppose we have N different viewpoints and corresponding ground truth images. We can set a virtual camera for each viewpoint and obtain the corresponding virtual image (RenderingImage) in the rendered digital twin scene. Then, we compare the virtual image (RenderingImage) with the corresponding ground truth image (Ground Truth) to calculate the loss for each viewpoint.

[0114] Let L be the LOSS from the i-th perspective. i (θ), where θ represents the current set of global illumination parameters. Specifically, the LOSS function can be pixel error (e.g., L2Loss), structural similarity (SSIM), or other suitable error metrics.

[0115] L i (θ)=Loss(I rendered,i I GT,i (1)

[0116] Among them, I rendered，i It is the virtual image rendered from the i-th viewpoint, I GT，i It is the ground truth image from the i-th perspective, L i (θ) represents the difference between the virtual image (Rendering Image) and the corresponding real image (Ground Truth) from the i-th viewpoint.

[0117] Assign a single-view weight α to each perspective's loss. i And calculate the weighted average loss for all viewpoints. This global loss reflects the combined requirements of all viewpoints for lighting parameters:

[0118]

[0119] Among them, L total (θ) represents the global difference between the virtual image and the real image, α i Let represent the single-view weight value of the i-th viewpoint, and N represent the total number of viewpoints.

[0120] Furthermore, the global difference value is compared with a preset difference threshold. If the global difference value exceeds the preset difference threshold, each lighting parameter is adjusted based on all single-view difference values ​​to obtain multiple adjusted lighting parameters.

[0121] In the above implementation process, by generating single-view difference values ​​between real images and corresponding virtual images from multiple perspectives of the digital twin scene, a global difference value is determined, thereby improving the accuracy of the difference comparison between the real scene and the digital twin scene. The size between the global difference value and the preset difference threshold is used to determine whether to adjust the lighting parameters of the image rendering engine. When adjusting the lighting parameters, the lighting parameters are adjusted by the single-view difference value, which improves the accuracy of the lighting parameter adjustment.

[0122] In one embodiment, step S420 further includes:

[0123] The corresponding single-view weight value is determined based on the proportion of each single-view difference value in the sum of all single-view difference values.

[0124] For example, the proportion of each single-view difference value in the sum of all single-view difference values ​​can be determined as the corresponding single-view weight value. The specific expression is as follows:

[0125]

[0126] In the above implementation process, the corresponding single-view weight value is determined according to the proportion of each single-view difference value in the sum of all single-view difference values. This realizes the dynamic adjustment of the single-view weight value, reduces the impact of high-error views on global parameters, and further improves the accuracy of lighting parameter adjustment.

[0127] In one embodiment, step S520 includes: determining a first difference value of the target view between the single-view virtual image and the corresponding single-view real image under the target view at the (t-1)th adjustment, and a second difference value of the target view between the single-view virtual image and the corresponding single-view real image under the target view at the tth adjustment; the target view is any view among a plurality of different viewpoints.

[0128] Step S530 may also include the following steps:

[0129] Step S531b: Determine the adjustment coefficient for each illumination parameter in the t-th adjustment.

[0130] Step S532b: Based on the rate of change between the first difference value and the second difference value of the target viewpoint and the single-viewpoint weight value under the target viewpoint, determine the gradient value of the t-th adjustment under the target viewpoint.

[0131] Step S533b: Determine the global gradient value for the t-th adjustment based on the average gradient of the t-th adjustment across all views.

[0132] Step S534b: Based on the global gradient value of the t-th adjustment and the adjustment coefficient of each illumination parameter in the t-th adjustment, determine the amount of change of each illumination parameter in the t-th adjustment.

[0133] Step S535b: Based on the multiple illumination parameters after the (t-1)th adjustment and the change of each illumination parameter in the tth adjustment, determine the multiple illumination parameters after the tth adjustment.

[0134] For example, when determining the difference value under multiple perspectives, the first difference value L of the target perspective between the single-view virtual image and the corresponding single-view real image under the target perspective at the (t-1)th adjustment can be determined by expression (1). i(θ), and the second difference value L of the target view between the single-view virtual image and the corresponding single-view real image at the t-th adjustment. i (θ+δθ). Wherein, the target viewpoint is any one of multiple different viewpoints, and δθ represents the change at the (t-1)th adjustment.

[0135] Furthermore, the rate of change between the first and second difference values ​​under the i-th perspective can be determined using expression (5):

[0136]

[0137] in, This represents the rate of change between the first difference value and the second difference value from the i-th perspective.

[0138] The adjustment coefficient for each illumination parameter in the t-th adjustment can be determined through steps S5311 and S5312. ηη Furthermore, the product of the rate of change between the first difference value and the second difference value of the target viewpoint and the single-viewpoint weight value under the target viewpoint is determined as the gradient value of the t-th adjustment under the target viewpoint.

[0139] The global gradient value for the t-th adjustment is determined based on the average gradient of all views at the t-th adjustment. Specifically, the global gradient value for the t-th adjustment can be determined by the following expression:

[0140]

[0141] in, This represents the global gradient value after the t-th adjustment.

[0142] Furthermore, based on the global gradient value of the t-th adjustment and the adjustment coefficient of each illumination parameter in the t-th adjustment, the change amount of each illumination parameter in the t-th adjustment is determined, and based on the multiple illumination parameters after the (t-1)-th adjustment and the change amount of each illumination parameter in the t-th adjustment, the multiple illumination parameters after the t-th adjustment are determined.

[0143] Specifically, the multiple lighting parameters after the t-th adjustment can be determined using the following expression:

[0144]

[0145] Where, θ t Let represent the illumination parameters after the t-th adjustment, and η represent the adjustment coefficient η for the t-th adjustment.

[0146] In the above implementation process, the gradient value of the adjustment under a single viewpoint is determined based on the rate of change of the single-viewpoint difference value during the two adjustments and the corresponding single-viewpoint weight value. The global gradient value is determined based on the average value of the adjustment gradient values ​​under all views. Then, the amount of adjustment is determined based on the global gradient value and the adjustment coefficient. Finally, the lighting parameters are adjusted based on the amount of adjustment. The influence of the gradient values ​​of multiple views on the global effect is comprehensively considered, and the lighting parameters are adjusted based on the global gradient value, which improves the accuracy of the lighting parameter adjustment.

[0147] In one embodiment, S400 includes:

[0148] Step S401: Determine the pixel loss value and structural similarity loss value between the virtual image and the real image respectively.

[0149] Step S402: Based on pixel loss values ​​and structural similarity loss values, determine the difference between the virtual image and the real image.

[0150] For example, when determining the difference between a virtual image and a real image, the pixel loss value between the virtual image and the real image can be determined according to the L1 loss function, and the structural loss value between the virtual image and the real image can be determined according to the SSIM (structural similarity) loss function.

[0151] Furthermore, the difference between the virtual image and the real image is obtained by weighting the pixel loss value and the structural similarity loss value.

[0152] In the above implementation process, the difference between the virtual image and the real image is determined based on the pixel loss value and structural similarity loss value between the virtual image and the real image, which improves the accuracy of the difference value determination and thus facilitates the improvement of the accuracy of the lighting parameter adjustment.

[0153] Please see Figure 2 , Figure 2 This is a schematic diagram of a digital twin scene lighting dynamic optimization device provided in an embodiment of this application. The device is applied to an image rendering engine and adjusts the lighting parameters of the image rendering engine according to the difference in lighting effects between the real scene and the digital twin scene, such as... Figure 2 The digital twin scene lighting dynamic optimization device shown includes:

[0154] The first acquisition module 201 is used to acquire real images of the real scene and the acquisition parameters of the device that acquires the real images; the real images are used to construct a digital twin scene.

[0155] The rendering module 202 is used to render the digital twin scene based on multiple lighting parameters of the image rendering engine to obtain the rendered digital twin scene; the multiple lighting parameters include lighting parameters, anti-aliasing parameters, and material resolution;

[0156] The second acquisition module 203 acquires the virtual image corresponding to the real image in the rendered digital twin scene based on the device acquisition parameters;

[0157] The difference determination module 204 is used to determine the difference value between the virtual image and the real image;

[0158] The parameter adjustment module 205 is used to adjust each illumination parameter based on the difference value if the difference value exceeds the preset difference threshold, so as to obtain multiple adjusted illumination parameters.

[0159] In one embodiment, adjusting multiple illumination parameters is performed as adjusting the illumination parameters for the t-th time, where t is a positive integer. The parameter adjustment module 205 is specifically used to perform the following steps:

[0160] Step S510: Based on the adjusted multiple illumination parameters, repeat steps S200 to S500.

[0161] Step S520: Determine the first difference value between the virtual image and the real image at the (t-1)th adjustment, and the second difference value between the virtual image and the real image at the tth adjustment;

[0162] Step S530: Based on the rate of change between the first difference value and the second difference value, and the multiple lighting parameters after the (t-1)th adjustment, determine the multiple lighting parameters after the tth adjustment, until the difference value between the virtual image and the real image does not exceed the preset difference threshold.

[0163] In one embodiment, the parameter adjustment module 205 is specifically used to perform the following steps:

[0164] Step S531a: Determine the adjustment coefficient for each illumination parameter in the t-th adjustment;

[0165] Step S532a: Based on the rate of change between the first difference value and the second difference value and the adjustment coefficient of each illumination parameter in the tth adjustment, determine the amount of change of each illumination parameter in the tth adjustment;

[0166] Step S533a: Based on the multiple illumination parameters after the (t-1)th adjustment and the change of each illumination parameter in the tth adjustment, determine the multiple illumination parameters after the tth adjustment.

[0167] In one embodiment, the parameter adjustment module 205 is specifically used to perform the following steps:

[0168] Step S5311: Determine the gradient change rate of each illumination parameter after the (t-1)th adjustment;

[0169] Step S5312: Based on the gradient change rate of each illumination parameter after the (t-1)th adjustment, determine the adjustment coefficient of the corresponding illumination parameter in the tth adjustment.

[0170] In one embodiment, the digital twin scene is generated by reconstructing real images from multiple different perspectives;

[0171] The first acquisition module 201 is specifically used to: acquire single-view real images of the real scene under each viewpoint, and the single-view device acquisition parameters corresponding to the single-view real images;

[0172] The second acquisition module 203 is specifically used to: acquire single-view virtual images in the rendered digital twin scene based on the acquisition parameters of each single-view device;

[0173] The difference determination module 204 is specifically used to perform the following steps: Step S410, determine the single-view difference value between the single-view real image and the corresponding single-view virtual image under each viewpoint; Step S420, determine the global difference value between the virtual image and the real image based on the single-view weight value of each viewpoint and the corresponding single-view difference value.

[0174] The parameter adjustment module 205 is specifically used to: if the global difference value exceeds the preset difference threshold, adjust each lighting parameter based on all single-view difference values ​​to obtain multiple adjusted lighting parameters.

[0175] In one embodiment, the difference determination module 204 is specifically used for:

[0176] The corresponding single-view weight value is determined based on the proportion of each single-view difference value in the sum of all single-view difference values.

[0177] In one embodiment, the parameter adjustment module 205 is specifically used to perform the following steps:

[0178] Step S520 includes: determining the first difference value of the target view between the single-view virtual image and the corresponding single-view real image under the target view at the (t-1)th adjustment, and the second difference value of the target view between the single-view virtual image and the corresponding single-view real image under the target view at the tth adjustment; the target view is any view among multiple different viewpoints;

[0179] Step S530 includes:

[0180] Step S531b: Determine the adjustment coefficient for each illumination parameter in the t-th adjustment;

[0181] Step S532b: Based on the rate of change between the first difference value and the second difference value of the target viewpoint and the single viewpoint weight value under the target viewpoint, determine the gradient value of the t-th adjustment under the target viewpoint;

[0182] Step S533b: Determine the global gradient value for the t-th adjustment based on the average gradient of the t-th adjustment across all views;

[0183] Step S534b: Based on the global gradient value of the t-th adjustment and the adjustment coefficient of each illumination parameter in the t-th adjustment, determine the amount of change of each illumination parameter in the t-th adjustment;

[0184] Step S535b: Based on the multiple illumination parameters after the (t-1)th adjustment and the change of each illumination parameter in the tth adjustment, determine the multiple illumination parameters after the tth adjustment.

[0185] In one embodiment, the difference determination module 204 is specifically used to perform the following steps:

[0186] Step S401: Determine the pixel loss value and structural similarity loss value between the virtual image and the real image respectively;

[0187] Step S402: Based on pixel loss values ​​and structural similarity loss values, determine the difference between the virtual image and the real image.

[0188] It should be understood that this embodiment of the digital twin scene lighting dynamic optimization method is capable of executing the various steps involved in the above method embodiments. The specific functions of the digital twin scene lighting dynamic optimization device can be found in the description above; to avoid repetition, detailed descriptions are appropriately omitted here. The digital twin scene lighting dynamic optimization device includes at least one software function module that can be stored in memory or embedded in the device's operating system (OS) in the form of software or firmware.

[0189] Please refer to Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 300 provided in this application includes a processor 301 and a memory 302. These components are interconnected and communicate with each other via a communication bus 303 and / or other forms of connection mechanisms (not shown). The memory 302 stores a computer program executable by the processor 301. When the computer program is executed by the processor 301, it performs the digital twin scene lighting dynamic optimization method described in the first aspect above.

[0190] This application also provides a computer-readable storage medium storing computer program instructions, which, when executed by processor 301, perform the digital twin scene lighting dynamic optimization method described in the first aspect above.

[0191] The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0192] It should be understood that the disclosed apparatus / systems and methods can also be implemented in other ways, as provided in the embodiments of this application. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0193] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0194] The above description is only an optional implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application.

Claims

1. A method for dynamic optimization of lighting in a digital twin scene, characterized in that, The method is applied to an image rendering engine, adjusting the lighting parameters of the image rendering engine based on the difference in lighting effects between the real scene and the digital twin scene. The method includes: Step S100: Obtain the real image of the real scene and the acquisition parameters of the device that acquires the real image; the real image is used to construct the digital twin scene; Step S200: Render the digital twin scene based on multiple lighting parameters of the image rendering engine to obtain the rendered digital twin scene; the multiple lighting parameters include lighting parameters, anti-aliasing parameters, and material resolution; Step S300: Based on the parameters collected by the device, obtain the virtual image corresponding to the real image in the rendered digital twin scene; Step S400: Determine the difference value between the virtual image and the real image; Step S500: If the difference value exceeds the preset difference threshold, then based on the difference value, each illumination parameter is adjusted to obtain multiple adjusted illumination parameters.

2. The method according to claim 1, characterized in that, The current adjustment of multiple illumination parameters involves performing the t-th adjustment on each illumination parameter, where t is a positive integer. Based on the difference value, each illumination parameter is adjusted to obtain the adjusted multiple illumination parameters, including: Step S510: Based on the adjusted multiple illumination parameters, repeat steps S200 to S500. Step S520: Determine the first difference value between the virtual image and the real image at the (t-1)th adjustment, and the second difference value between the virtual image and the real image at the tth adjustment; Step S530: Based on the rate of change between the first difference value and the second difference value, and the multiple lighting parameters after the (t-1)th adjustment, determine the multiple lighting parameters after the tth adjustment, until the difference value between the virtual image and the real image does not exceed the preset difference threshold.

3. The method according to claim 2, characterized in that, Step S530 includes: Step S531a: Determine the adjustment coefficient for each illumination parameter in the t-th adjustment; Step S532a: Based on the rate of change between the first difference value and the second difference value and the adjustment coefficient of each illumination parameter in the t-th adjustment, determine the amount of change of each illumination parameter in the t-th adjustment; Step S533a: Based on the multiple illumination parameters after the (t-1)th adjustment and the change of each illumination parameter in the tth adjustment, determine the multiple illumination parameters after the tth adjustment.

4. The method according to claim 3, characterized in that, Step S531a includes: Step S5311: Determine the gradient change rate of each illumination parameter after the (t-1)th adjustment; Step S5312: Based on the gradient change rate of each illumination parameter after the (t-1)th adjustment, determine the adjustment coefficient of the corresponding illumination parameter in the tth adjustment.

5. The method according to claim 2, characterized in that, The digital twin scene is generated by reconstructing real images from multiple different perspectives; Step S100 includes: acquiring a single-view real image of the real scene from each viewpoint, and the single-view device acquisition parameters corresponding to the single-view real image. Step S300 includes: acquiring a single-view virtual image in the rendered digital twin scene based on the acquisition parameters of each single-view device; Step S400 includes: Step S410, determining the single-view difference value between the single-view real image and the corresponding single-view virtual image under each viewpoint; Step S420, determining the global difference value between the virtual image and the real image based on the single-view weight value of each viewpoint and the corresponding single-view difference value. Step S500 includes: if the global difference value exceeds the preset difference threshold, then based on all single-view difference values, each illumination parameter is adjusted to obtain multiple adjusted illumination parameters.

6. The method according to claim 5, characterized in that, Step S420 further includes: The corresponding single-view weight value is determined based on the proportion of each single-view difference value in the sum of all single-view difference values.

7. The method according to claim 5, characterized in that, Step S520 includes: determining a first difference value of the target view between the single-view virtual image and the corresponding single-view real image under the target view at the (t-1)th adjustment, and a second difference value of the target view between the single-view virtual image and the corresponding single-view real image under the target view at the tth adjustment; the target view is any view among multiple different viewpoints; Step S530 includes: Step S531b: Determine the adjustment coefficient for each illumination parameter in the t-th adjustment; Step S532b: Based on the rate of change between the first difference value of the target view and the second difference value of the target view and the single view weight value under the target view, determine the gradient value of the t-th adjustment under the target view; Step S533b: Determine the global gradient value for the t-th adjustment based on the average gradient of the t-th adjustment across all views; Step S534b: Based on the global gradient value of the t-th adjustment and the adjustment coefficient of each illumination parameter in the t-th adjustment, determine the amount of change of each illumination parameter in the t-th adjustment; Step S535b: Based on the multiple illumination parameters after the (t-1)th adjustment and the change of each illumination parameter in the tth adjustment, determine the multiple illumination parameters after the tth adjustment.

8. The method according to any one of claims 1-7, characterized in that, The S400 includes: Step S401: Determine the pixel loss value and structural similarity loss value between the virtual image and the real image, respectively; Step S402: Based on the pixel loss value and the structural similarity loss value, determine the difference value between the virtual image and the real image.

9. A digital twin scene lighting dynamic optimization device, characterized in that, The device is applied to an image rendering engine to adjust the lighting parameters of the image rendering engine based on the difference in lighting effects between the real scene and the digital twin scene. The device includes: The first acquisition module is used to acquire real images of the real scene and acquisition parameters of the device that acquires the real images; the real images are used to construct the digital twin scene. The rendering module is used to render the digital twin scene based on multiple lighting parameters of the image rendering engine to obtain the rendered digital twin scene; the multiple lighting parameters include lighting parameters, anti-aliasing parameters, and material resolution; The second acquisition module acquires the virtual image corresponding to the real image in the rendered digital twin scene based on the parameters collected by the device. A difference determination module is used to determine the difference value between the virtual image and the real image; The parameter adjustment module is used to adjust each illumination parameter based on the difference value if the difference value exceeds a preset difference threshold, so as to obtain multiple adjusted illumination parameters.

10. An electronic device, characterized in that, The electronic device includes: Memory; processor; The memory stores a computer program executable by the processor. When the computer program is executed by the processor, it performs the digital twin scene lighting dynamic optimization method according to any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, perform the digital twin scene lighting dynamic optimization method according to any one of claims 1-8.

Images