Image Generation Method, Device and Electronic Device Based on Digital Twin Rendering Engine

Abstract

The embodiments of the present application provide an image generation method, device, and electronic device based on a digital twin rendering engine. If the object to be displayed is in a relatively far distance range, the resolution of the target object model of the object to be displayed is relatively low. If the object to be displayed is in a relatively near distance range, the resolution of the target object model of the object to be displayed is relatively high. That is, the object to be displayed that is farther from the viewer has a lower resolution and a more single texture, and the object to be displayed that is closer to the viewer has a higher resolution and a more abundant texture. Therefore, in the generated image, the situation of rendering multiple different colors in one pixel can be reduced, and the moiré phenomenon in the image can be reduced. Moreover, the image after removing the moiré generated in the embodiments of the present application is a lossless image and will not become blurred after processing.

Description

Technical Field

[0001] The present application relates to the field of image processing technologies, and particularly to an image generation method, apparatus, and electronic device based on a digital twin rendering engine. Background Art

[0002] Digital twin is a simulation process that integrates multidisciplinary, multi-physical quantity, multi-scale, and multi-probability data such as physical models, sensor data, and operation history data, and completes mapping in a virtual space to reflect the entire life cycle process of the corresponding physical equipment. For the real-time rendering part of the digital twin three-dimensional visualization engine, it is necessary to have the capabilities of image data acquisition and rendering processing of image data. In the case of comprehensive display of multi-source data, moiré flickering often occurs, affecting the sensory experience of image data acquisition, analysis, judgment, and visual presentation.

[0003] The main reasons for generating moiré include a large color difference between adjacent pixels and a single pixel rendering multiple different colors. In the digital twin three-dimensional visualization engine, when patterns such as repeated lines, circles, or dots overlap with imperfect alignment, a new, irregular dynamic pattern will appear, and this phenomenon is called moiré. Moiré will appear in 3D models and texture maps in a three-dimensional scene. When two original patterns move relative to each other, the moiré changes the shape and frequency of its elements.

[0004] In the prior art, the solutions to moiré in real-time rendering engines mainly include schemes for adjusting space and frequency such as control filtering and upsampling interpolation. The main purpose of upsampling or image interpolation is to enlarge the original image so that it can be displayed on a display device with a higher resolution. However, the magnification operation of the image does not bring more detailed information about the image, so the quality of the image will inevitably be affected, and there is still a problem of moiré flickering in the image. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide an image generation method, apparatus, and electronic device based on a digital twin rendering engine to reduce moiré phenomena in images. The specific technical solutions are as follows:

[0006] In a first aspect, the embodiments of the present application provide an image generation method based on a digital twin rendering engine, and the method includes:

[0007] Obtain the target distance of each object to be displayed in the three-dimensional scene from the rendering camera;

[0008] For each object to be displayed, determine the distance interval to which the target distance of the object to be displayed belongs within a plurality of pre-set distance intervals as the target distance interval of the object to be displayed;

[0009] For each object to be displayed, determine the object model of the detail level corresponding to the target distance interval of the object to be displayed as the target object model of the object to be displayed; wherein, different distance intervals correspond to different detail levels, and for any distance interval, the longer the distance represented by the distance interval, the lower the detail level corresponding to the distance interval; for any detail level, the lower the detail level, the lower the resolution of the object model at the detail level.

[0010] Generate an image to be displayed based on the target object models of the respective objects to be displayed.

[0011] In a possible implementation manner, the generating an image to be displayed based on the target object models of the respective objects to be displayed includes:

[0012] Obtain the poses of the respective objects to be displayed in the image to be displayed;

[0013] For each object to be displayed, render the target object model of the object to be displayed according to the pose of the object to be displayed in the image to be displayed and the target distance of the object to be displayed, and generate the pixel area of the object to be displayed in the image to be displayed.

[0014] Generate an image to be displayed based on the pixel areas of the respective objects to be displayed in the image to be displayed.

[0015] In a possible implementation manner, the method further includes:

[0016] Obtain the image texture maps of the respective objects in the three-dimensional scene collected at different distances, the collection distances of the image texture maps of the respective objects, and the preset distance intervals;

[0017] For each image texture map, divide the image texture map into the image set corresponding to the distance interval to which the collection distance of the image texture map belongs;

[0018] For each distance interval, establish the object models of the respective objects at the detail level corresponding to the distance interval according to the image texture maps of the respective objects in the image set corresponding to the distance interval.

[0019] In a possible implementation manner, the establishing the object models of the respective objects at the detail level corresponding to the distance interval according to the image texture maps of the respective objects in the image set corresponding to the distance interval includes:

[0020] For each distance interval, select multiple batches of sample data from the image texture maps of the respective objects in the distance interval, and use the sample data of each batch to respectively determine multiple primary object models of each object at the detail level corresponding to the distance interval;

[0021] For each object, the primary object models of the object at the corresponding level of detail in the same distance interval are respectively mixed to obtain the object model of the object at the corresponding level of detail in each distance interval.

[0022] In a possible implementation manner, the method further includes:

[0023] For the moiré pattern area in the image texture map with moiré patterns, calculate the color difference between adjacent pixels in the moiré pattern area;

[0024] For adjacent pixels with a color difference greater than a preset difference threshold, calculate a difference color based on the colors of the two pixels in the adjacent pixels, and insert pixels with the difference color between the adjacent pixels.

[0025] In a second aspect, an embodiment of the present application provides an image generation device based on a digital twin rendering engine, and the device includes:

[0026] A distance acquisition module, configured to acquire the target distance of each object to be displayed in the three-dimensional scene from the rendering camera;

[0027] An interval determination module, configured to, for each object to be displayed, determine the distance interval to which the target distance of the object to be displayed belongs in a plurality of preset distance intervals as the target distance interval of the object to be displayed;

[0028] A model determination module, configured to, for each object to be displayed, determine the object model of the level of detail corresponding to the target distance interval of the object to be displayed as the target object model of the object to be displayed; wherein, different distance intervals correspond to different levels of detail, for any distance interval, the longer the distance represented by the distance interval, the lower the level of detail corresponding to the distance interval; for any level of detail, the lower the level of detail, the lower the resolution of the object model at the level of detail;

[0029] An image generation module, configured to generate an image to be displayed based on the target object models of the objects to be displayed.

[0030] In a possible implementation manner, the image generation module is specifically configured to:

[0031] Acquire the poses of the objects to be displayed in the image to be displayed;

[0032] For each object to be displayed, render the target object model of the object to be displayed according to the pose of the object to be displayed in the image to be displayed and the target distance of the object to be displayed, and generate a pixel area of the object to be displayed in the image to be displayed;

[0033] Generate the image to be displayed based on the pixel regions of the objects to be displayed in the image to be displayed.

[0034] In a possible implementation manner, the device further includes:

[0035] A texture map acquisition module, configured to acquire the image texture maps of the objects in the three-dimensional scene collected at different distances, the acquisition distances of the image texture maps of the objects, and preset distance intervals.

[0036] A texture map division module, configured to divide each image texture map into the image set corresponding to the distance interval to which the acquisition distance of the image texture map belongs.

[0037] A model establishment module, configured to establish object models of the objects at the corresponding level of detail for each distance interval according to the image texture maps of the objects in the image set corresponding to the distance interval.

[0038] In a possible implementation manner, the model establishment module is specifically configured to:

[0039] For each distance interval, select multiple batches of sample data from the image texture maps of the objects in the distance interval, and use the sample data of each batch to respectively determine multiple primary object models of each object at the corresponding level of detail in the distance interval.

[0040] For each object, respectively perform model mixing on the primary object models of the object at the corresponding level of detail in the same distance interval to obtain the object model of the object at the corresponding level of detail in each distance interval.

[0041] In a possible implementation manner, the device further includes a color compensation module, configured to:

[0042] For the moiré region in the image texture map with moiré, calculate the color difference between adjacent pixels in the moiré region.

[0043] For adjacent pixels with a color difference greater than a preset difference threshold, calculate a difference color based on the colors of the two pixels in the adjacent pixels, and insert pixels of the difference color between the adjacent pixels.

[0044] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory;

[0045] The memory is used to store a computer program;

[0046] When the processor executes the program stored in the memory, the method described in any one of the present application is implemented.

[0047] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the methods described in any one of the present application are implemented.

[0048] In a fifth aspect, an embodiment of the present application further provides a computer program product containing instructions, which when running on a computer, causes the computer to execute the methods described in any one of the present application.

[0049] Beneficial effects of the embodiments of the present application:

[0050] The image generation method, device, and electronic device based on a digital twin rendering engine provided by the embodiments of the present application obtain the target distance of each object to be displayed in a three-dimensional scene from a rendering camera; for each object to be displayed, determine the distance interval to which the target distance of the object to be displayed belongs within a plurality of preset distance intervals as the target distance interval of the object to be displayed; for each object to be displayed, determine the object model of the detail level corresponding to the target distance interval of the object to be displayed as the target object model of the object to be displayed; where different distance intervals correspond to different detail levels, and for any distance interval, the longer the distance represented by the distance interval, the lower the detail level corresponding to the distance interval; for any detail level, the lower the detail level, the lower the resolution of the object model at the detail level; based on the respective target object models of each object to be displayed, generate an image to be displayed. If the object to be displayed is in a relatively far distance interval, the resolution of the target object model of the object to be displayed is relatively low, and if the object to be displayed is in a relatively near distance interval, the resolution of the target object model of the object to be displayed is relatively high, that is, the object to be displayed farther from the viewer has a lower resolution and a more single texture, and the object to be displayed closer to the viewer has a higher resolution and a richer texture, so that the situation of rendering multiple different colors in one pixel can be reduced in the generated image, and the moiré phenomenon in the image can be reduced; and the image after eliminating moiré generated in the embodiments of the present application is a lossless image and will not become blurred after processing.

[0051] Of course, implementing any product or method of the present application does not necessarily require achieving all the above-mentioned advantages simultaneously. Description of the Drawings

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application, and those of ordinary skill in the art can also obtain other drawings based on these drawings.

[0053] Figure 1A schematic diagram of an image generation method based on a digital twin rendering engine according to an embodiment of the present application;

[0054] Figure 2 A schematic diagram of a possible implementation manner of step S104 in an embodiment of the present application;

[0055] Figure 3 A schematic diagram of an object model establishment method according to an embodiment of the present application;

[0056] Figure 4 Another schematic diagram of an object model establishment method according to an embodiment of the present application;

[0057] Figure 5 A schematic diagram of an image generation device based on a digital twin rendering engine according to an embodiment of the present application;

[0058] Figure 6 A schematic diagram of an electronic device according to an embodiment of the present application. Detailed implementation manners

[0059] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art based on the present application belong to the scope of protection of the present application.

[0060] To reduce the moiré phenomenon in an image, an embodiment of the present application provides an image generation method based on a digital twin rendering engine. Refer to Figure 1 , the method includes:

[0061] S101, obtaining the target distance of each object to be displayed in the three-dimensional scene from the rendering camera.

[0062] The image generation method based on a digital twin rendering engine in an embodiment of the present application can be implemented by an electronic device and can be applied to scenarios such as digital twin engines, three-dimensional scene modeling, and image data rendering. In one example, the above-mentioned electronic device can be a personal computer, a server, a smart phone, or a VR (Virtual Reality) device, etc.

[0063] Here, the three-dimensional scene is the three-dimensional scene for which image display is required. The rendering camera is a virtual camera, which can be understood as the viewing point for observing this three-dimensional scene, rather than a real camera device. For the sake of easy understanding, the following is a simple example. For instance, if the three-dimensional scene is a classroom three-dimensional scene and the user hopes to view the entire classroom from the exact center of the podium, then it is considered that the rendering camera is at the exact center of the podium, that is, the rendering camera is the user's viewing point. The image captured by the rendering camera is the image to be displayed, that is, the image seen by the user.

[0064] In one example, the target distance of the object to be displayed from the rendering camera can be obtained through a front-end perception algorithm. The front-end perception algorithm refers to the algorithm for the distance of an object in the three-dimensional scene of the digital twin engine from the rendering camera and the relative device display resolution size.

[0065] In a possible implementation manner, the obtaining of the target distances of the objects to be displayed in the three-dimensional scene from the rendering camera includes: obtaining the pose of the rendering camera in the three-dimensional scene; and determining, according to the pose of the rendering camera, the target distances of the objects to be displayed in the three-dimensional scene and the rendering camera from each of the objects to be displayed.

[0066] The pose includes two aspects: position and attitude. The attitude of the rendering camera represents the shooting direction of the rendering camera. In combination with the position of the rendering camera, it can be determined which objects in the three-dimensional scene the rendering camera can capture. The objects in the three-dimensional scene captured by the rendering camera are called objects to be displayed. According to the pose of the objects to be displayed in the three-dimensional scene, the target distance of the rendering camera from each object to be displayed can also be obtained.

[0067] The method for obtaining the pose of the rendering camera in the three-dimensional scene can refer to the prior art. In one example, taking a VR device as an example, the pose of the VR device in the world coordinate system can be obtained according to the gyroscope, positioner, etc. in the VR device, and in combination with the resolution of the display and the conversion relationship between the world coordinate system and the three-dimensional scene coordinate system, the pose of the rendering camera in the three-dimensional scene can thus be obtained.

[0068] S102. For each object to be displayed, determine the distance interval to which the target distance of the object to be displayed belongs within a plurality of preset distance intervals as the target distance interval of the object to be displayed.

[0069] The distance intervals are preset. The number of distance intervals and the distance span of each distance interval can be customized according to the actual situation. The distance spans of different distance intervals can be the same or different. In one example, taking three distance intervals as an example, distance interval 1 is from 0 meters to 10 meters, distance interval 2 is from 10 meters to 50 meters, and distance interval 3 is more than 50 meters. In this case, if the target distance of display object 1 is 5 meters, and 5 meters belongs to distance interval 1 from 0 meters to 10 meters, then distance interval 1 is the target distance interval of display object 1. If the target distance of display object 2 is 15 meters, and 15 meters belongs to distance interval 2 from 10 meters to 50 meters, then distance interval 2 is the target distance interval of display object 2.

[0070] S103. For each display object to be displayed, determine the object model of the detail level corresponding to the target distance interval of the display object to be displayed as the target object model of the display object to be displayed. Among them, different distance intervals correspond to different detail levels. For any distance interval, the longer the distance represented by the distance interval, the lower the detail level corresponding to the distance interval. For any detail level, the lower the detail level, the lower the resolution of the object model at the detail level.

[0071] Different distance intervals correspond to different detail levels, and the resolutions of the object models of the same object at different detail levels are different. The longer the distance represented by the distance interval, the lower the detail level corresponding to the distance interval, and the lower the detail level, the lower the resolution of the object model at the detail level. The resolutions of the object models at different detail levels can be set according to the resolution size of the display.

[0072] For the convenience of understanding, a simple example is given below. Taking three distance intervals as an example, distance interval 1 is from 0 meters to 10 meters, corresponding to detail level 3; distance interval 2 is from 10 meters to 50 meters, corresponding to detail level 2; distance interval 3 is more than 50 meters, corresponding to detail level 1. Then the object model of object 1 at detail level 3 can be represented by 1000 pixels, the object model of object 1 at detail level 2 can be represented by 100 pixels, and the object model of object 1 at detail level 1 can be represented by 10 pixels. It can be understood that the values here are only for example, and in the actual scenario, it can be customized according to the principles in this application.

[0073] In one example, the detail level in the embodiment of this application can specifically be LOD (Levels of Detail). LOD refers to determining the allocation of rendering resources in the displayed image according to the position and importance of the key nodes of the object in the display environment, reducing the accuracy and detail of unimportant objects, so as to obtain high-efficiency calculation results. In the embodiment of this application, each object is displayed according to different visible distances (i.e., the target distances of the display objects to be displayed).

[0074] Use the z-buffer to represent the target distance of the object to be displayed from the rendering camera. In the case of a cross-distance interval, the larger the z-buffer value, the lower the LOD level of the object to be displayed. The lower the LOD level, the lower the resolution of the object model of the same object. Automatically calculate and switch different LOD levels according to the change of the z-buffer value to obtain the object models corresponding to each object to be displayed under their respective LODs.

[0075] S104. Generate the image to be displayed based on the target object models of each of the objects to be displayed.

[0076] Render the target object models of each object to be displayed to obtain the image to be displayed. In one example, based on the target object models of each object to be displayed, the image texture data and the feature map of the 3D model data can be fused through a multi-scale algorithm to restore the image after moiré elimination.

[0077] According to the target distance (z-buffer) from the rendering camera of the digital twin rendering engine to the object to be displayed, judge the LOD level of the object to be displayed, and finally determine the display resolution of the object to be displayed on the display. The smaller the z-buffer value of the object to be displayed, the higher the LOD level of its engine resolution. The larger the z-buffer value of the object to be displayed, the lower the LOD level shown by its engine resolution, thereby reducing the moiré flicker phenomenon.

[0078] In the embodiments of the present application, if the object to be displayed is in a relatively far distance interval, the resolution of the target object model of the object to be displayed is relatively low. If the object to be displayed is in a relatively near distance interval, the resolution of the target object model of the object to be displayed is relatively high. That is, the object to be displayed that is farther from the viewer has a lower resolution and a more single texture, and the object to be displayed that is closer to the viewer has a higher resolution and a richer texture. Thus, in the generated image, the situation of rendering multiple different colors in one pixel can be reduced, and the moiré phenomenon in the image can be reduced; and the image after moiré elimination generated in the embodiments of the present application is a lossless image and will not become blurred after processing.

[0079] In a possible implementation manner, refer to Figure 2 , the generating the image to be displayed based on the target object models of each of the objects to be displayed includes:

[0080] S1041. Obtain the poses of each of the objects to be displayed in the image to be displayed.

[0081] In one example, the poses of each of the objects to be displayed in the image to be displayed can be determined according to the pose of the rendering camera and the poses of each of the objects to be displayed in the 3D scene.

[0082] S1042. For each object to be displayed, based on the pose of the object to be displayed in the image to be displayed and the target distance of the object to be displayed, render the target object model of the object to be displayed to generate the pixel region of the object to be displayed in the image to be displayed.

[0083] For each object to be displayed, based on the target distance of the object to be displayed, the size of the target object model of the object to be displayed in the image to be displayed can be obtained. Combining with the pose of the object to be displayed in the image to be displayed, rendering the target object model of the object to be displayed can generate the pixel region of the object to be displayed in the image to be displayed.

[0084] In one example, object models at the same level of detail can also be divided into different levels of refinement. When the resolution of the display is fixed, assuming that the level of refinement ratio of the object to be displayed is 1 when the target distance D of the object to be displayed is 1, the level of refinement of the object to be displayed can be expressed as:

[0085] L = |1 / D|

[0086] L represents the level of refinement of the model. When D is less than 1, as D decreases, L approaches positive infinity, and the highest-definition object model under the corresponding LOD is applied; when D is greater than 1, as D increases, L approaches 0, and the object model with the lowest precision under the corresponding LOD is applied.

[0087] S1043. Generate the image to be displayed based on the pixel regions of the objects to be displayed in the image to be displayed.

[0088] In the embodiment of the present application, render the target object models of the objects to be displayed to obtain the pixel regions of the objects to be displayed in the image to be displayed, and further obtain the image to be displayed.

[0089] The following describes the process of establishing the object model. In one possible implementation, see Figure 3 , the method further includes:

[0090] S301. Obtain the image texture maps of the objects in the three-dimensional scene collected at different distances, the collection distances of the image texture maps of the objects, and the preset distance intervals.

[0091] In one example, the collected image texture maps can be preprocessed, and the digital twin rendering engine is used to process the image texture maps at different levels of detail. The size of the image texture map needs to be a power of 2. Among them, the reference table of powers of 2 has pictures with sizes of 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, and 2048.

[0092] In S302, for each image texture map, divide the image texture map into the image set corresponding to the distance interval to which the acquisition distance of the image texture map belongs.

[0093] For each image texture map, determine the detail level corresponding to the image texture map according to the distance from the rendering camera to the acquisition of the image texture map. Different detail levels correspond to different distance intervals. Divide the image texture map into the image set corresponding to the distance interval to which the acquisition distance of the image texture map belongs. The smaller the acquisition distance of the image texture map, the higher the detail level corresponding to the image texture map. The larger the acquisition distance of the image texture map, the lower the detail level corresponding to the image texture map, thereby reducing the moiré flickering phenomenon caused by the image texture map factor.

[0094] In S303, for each distance interval, establish an object model of each object at the detail level corresponding to the distance interval according to the image texture maps of the objects in the image set corresponding to the distance interval.

[0095] In one example, the objects at different detail levels can be modeled by combining the image texture map data, the three-dimensional model data of the object, and the multi-resolution parameter data respectively.

[0096] In a possible implementation manner, the step of establishing an object model of each object at the detail level corresponding to the distance interval according to the image texture maps of the objects in the image set corresponding to the distance interval includes:

[0097] Step 1, for each distance interval, select multiple batches of sample data from the image texture maps of the objects in the distance interval, and use the sample data of each batch to respectively determine multiple primary object models of each object at the detail level corresponding to the distance interval.

[0098] Step 2, for each object, respectively perform model mixing on the primary object models of the object at the detail level corresponding to the same distance interval to obtain the object model of the object at the detail level corresponding to each distance interval.

[0099] In one example, a multi-data model mixing algorithm can be used to implement model mixing. The multi-data model mixing algorithm refers to mixing and fusing calculation of data source models in multiple dimensions, which can achieve accurate analysis and prediction capabilities. The multi-data model mixing algorithm mixes multiple weak models of different types into a strong model. For each distance interval, automatic sampling of multiple batches of sample data is performed from the image texture maps of each object in the distance interval. Each batch of sampled data contains multiple sample data, with a total of N batches. N primary object models are trained using the sample data of N batches respectively, and then the results of the N primary object models of the same object are fused and mixed to obtain the object model corresponding to the object at the corresponding level of detail in the distance interval.

[0100] The model mixing algorithm can adopt the voting method and the mean method respectively. Among them, the voting method means that if it is a classification model, each model will give a category prediction result. Through voting, a new prediction result is fused according to the principle of the minority obeying the majority. The mean method means that if it is a regression model, the prediction results given by each model are numerical. At this time, we can obtain the final fusion result by calculating the mean of the prediction results of all sub-models. There is no mutual connection between the primary object models in the model mixing algorithm, which is a parallel fusion method. It can process N primary object models in parallel at the same time, which can greatly improve the algorithm execution efficiency; its process can be as Figure 4 shown, where the LOD data can include 3D model data of the object in addition to the image texture map.

[0101] In the embodiment of the present application, the data source used to establish the object model is simple and convenient to obtain, the implementation cost is low, and the model mixing method is used to obtain the object model. The algorithm is simple and does not require a large amount of deep learning training; it can be applied to digital twin projects where data cannot be collected by drones, which can greatly reduce the implementation cost.

[0102] In order to further reduce the phenomenon of moiré flickering, in one possible implementation, the method further includes:

[0103] Step A, for the moiré region in the image texture map with moiré, calculate the color difference between adjacent pixels in the moiré region.

[0104] Step B, for adjacent pixels with a color difference greater than the preset difference threshold, calculate the difference color according to the colors of the two pixels in the adjacent pixels, and insert pixels of the difference color between the adjacent pixels.

[0105] The preset difference threshold can be custom-set according to the actual situation, and can be an experimental value or an empirical value. In one example, the colors of the two pixels in the adjacent pixels can be weighted and averaged to obtain the difference color.

[0106] When moiré patterns appear in the image texture map at a certain LOD level, the phenomenon of moiré flicker can be reduced by means of the auxiliary method of filling in the difference value. The difference value filling refers to when the color difference between adjacent pixels is too large, such as when the adjacent pixels are black and white respectively, an interpolation color is added as an intermediate value between black and white, so as to further reduce the moiré flicker phenomenon. In addition to adding the interpolation color, it is also possible to remove the model structures of two adjacent pixels with large color differences, or adjust the pixels to the model structures with similar colors, etc.

[0107] In the embodiments of the present application, by filling in the color difference value, the color drop between adjacent pixels can be reduced, thereby further reducing the moiré flicker phenomenon.

[0108] The embodiments of the present application provide an image generation device based on a digital twin rendering engine. Refer to Figure 5 and the device includes:

[0109] A distance acquisition module 501, configured to acquire the target distance of each object to be displayed in the three-dimensional scene from the rendering camera;

[0110] An interval determination module 502, configured to, for each object to be displayed, determine the distance interval to which the target distance of the object to be displayed belongs within a plurality of preset distance intervals as the target distance interval of the object to be displayed;

[0111] A model determination module 503, configured to, for each object to be displayed, determine the object model of the detail level corresponding to the target distance interval of the object to be displayed as the target object model of the object to be displayed; wherein, different distance intervals correspond to different detail levels, for any distance interval, the longer the distance represented by the distance interval, the lower the detail level corresponding to the distance interval; for any detail level, the lower the detail level, the lower the resolution of the object model at the detail level;

[0112] An image generation module 504, configured to generate an image to be displayed based on the target object models of the respective objects to be displayed.

[0113] In a possible implementation manner, the image generation module is specifically configured to:

[0114] Acquire the poses of the respective objects to be displayed in the image to be displayed;

[0115] For each object to be displayed, render the target object model of the object to be displayed according to the pose of the object to be displayed in the image to be displayed and the target distance of the object to be displayed, and generate the pixel area of the object to be displayed in the image to be displayed;

[0116] Generate an image to be displayed based on the pixel regions of the objects to be displayed in the image to be displayed.

[0117] In a possible implementation manner, the apparatus further includes:

[0118] A texture map acquisition module, configured to acquire the image texture maps of the objects in the three-dimensional scene collected at different distances, the acquisition distances of the image texture maps of the objects, and preset distance intervals.

[0119] A texture map division module, configured to divide each image texture map into an image set corresponding to the distance interval to which the acquisition distance of the image texture map belongs.

[0120] A model establishment module, configured to establish an object model of each object at the corresponding level of detail for each distance interval according to the image texture maps of the objects in the image set corresponding to the distance interval.

[0121] In a possible implementation manner, the model establishment module is specifically configured to:

[0122] For each distance interval, select multiple batches of sample data from the image texture maps of the objects in the distance interval, and use the sample data of each batch to respectively determine multiple primary object models of each object at the corresponding level of detail in the distance interval.

[0123] For each object, respectively perform model mixing on the primary object models of the object at the corresponding level of detail in the same distance interval to obtain the object model of the object at the corresponding level of detail in each distance interval.

[0124] In a possible implementation manner, the apparatus further includes a color compensation module, configured to:

[0125] For the moiré region in the image texture map with moiré, calculate the color difference between adjacent pixels in the moiré region.

[0126] For adjacent pixels with a color difference greater than a preset difference threshold, calculate a difference color according to the colors of the two pixels in the adjacent pixels, and insert pixels of the difference color between the adjacent pixels.

[0127] An embodiment of the present application further provides an electronic device, including: a processor and a memory;

[0128] The above-mentioned memory is used to store a computer program.

[0129] When the above-mentioned processor executes the computer program stored in the above-mentioned memory, it implements the image generation method based on the digital twin rendering engine described in any one of the present application.

[0130] Optionally, refer to Figure 6 , the electronic device according to the embodiment of the present application further includes a communication interface 602 and a communication bus 604. Among them, the processor 601, the communication interface 602, and the memory 603 complete communication with each other through the communication bus 604.

[0131] The communication bus mentioned in the above electronic device may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity, only a thick line is used in the figure to represent it, but it does not mean that there is only one bus or one type of bus.

[0132] The communication interface is used for communication between the above electronic device and other devices.

[0133] The memory may include a RAM (Random Access Memory), or may also include an NVM (Non-Volatile Memory), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0134] The above-mentioned processor may be a general-purpose processor, including a CPU (Central Processing Unit), an NP (Network Processor), etc.; it may also be a DSP (Digital Signal Processing), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0135] The embodiment of the present application also provides a computer-readable storage medium. The computer program stored in the above computer-readable storage medium, when executed by the processor, implements any one of the image generation methods based on the digital twin rendering engine in the present application.

[0136] In another embodiment provided by the present application, there is also provided a computer program product containing instructions, which, when running on a computer, causes the computer to execute any of the image generation methods based on the digital twin rendering engine described in the present application.

[0137] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that the computer can access or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state disk (SSD)).

[0138] It should be noted that in this document, the technical features in each alternative solution can be combined to form a solution as long as they are not contradictory, and these solutions are all within the scope disclosed in the present application. Relative terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including", or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article, or device including a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article, or device including the element.

[0139] Each embodiment in this specification is described in a related manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the device, electronic device, computer program product, and storage medium, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, reference can be made to the partial description of the method embodiments.

[0140] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are all included in the protection scope of the present application.

Claims

1. An image generation method based on a digital twin rendering engine, characterized in that, the method includes: Obtaining the target distance of each object to be displayed in the three-dimensional scene from the rendering camera; For each object to be displayed, determine the distance interval to which the target distance of the object to be displayed belongs within a plurality of preset distance intervals as the target distance interval of the object to be displayed; For each object to be displayed, determine the object model of the detail level corresponding to the target distance interval of the object to be displayed as the target object model of the object to be displayed; wherein, different distance intervals correspond to different detail levels, and for any distance interval, the longer the distance represented by the distance interval, the lower the detail level corresponding to the distance interval; for any detail level, the lower the detail level, the lower the resolution of the object model at the detail level; Generating an image to be displayed based on the target object models of the respective objects to be displayed; The method further includes: for each of the distance intervals, selecting multiple batches of sample data from the image texture maps of the objects in the distance interval, and using the sample data of each batch to respectively determine multiple primary object models of each object at the detail level corresponding to the distance interval; For each of the objects, respectively perform model mixing on the primary object models of the object at the detail level corresponding to the same distance interval to obtain the object model of the object at the detail level corresponding to each of the distance intervals, wherein the detail level is used to determine the allocation of rendering resources in the image to be displayed according to the position and importance of the key nodes of the object in the display environment; The generating an image to be displayed based on the target object models of the respective objects to be displayed includes: based on the target object models of the respective objects to be displayed, restoring the image after eliminating moiré by fusing the image texture data and the three-dimensional model data feature map through a multi-scale algorithm.

2. The method according to claim 1, characterized in that, the generating an image to be displayed based on the target object models of the respective objects to be displayed includes: Obtaining the poses of the respective objects to be displayed in the image to be displayed; For each object to be displayed, rendering the target object model of the object to be displayed according to the pose of the object to be displayed in the image to be displayed and the target distance of the object to be displayed, and generating a pixel area of the object to be displayed in the image to be displayed; Generating an image to be displayed based on the pixel areas of the respective objects to be displayed in the image to be displayed.

3. The method according to claim 1, characterized in that, the method further includes: Obtaining the image texture maps of the respective objects in the three-dimensional scene collected at different distances, the collection distances of the image texture maps of the respective objects, and the preset distance intervals; For each image texture map, dividing the image texture map into the image set corresponding to the distance interval to which the collection distance of the image texture map belongs; For each distance interval, based on the image texture maps of each object in the image set corresponding to this distance interval, establish the object models of each object at the level of detail corresponding to this distance interval.

4. The method according to claim 3, wherein, the method further includes: For the moiré region in the image texture map with moiré, calculate the color difference between adjacent pixels in the moiré region; For adjacent pixels with a color difference greater than a preset difference threshold, calculate a difference color based on the colors of the two pixels in the adjacent pixels, and insert pixels of the difference color between the adjacent pixels.

5. An image generation device based on a digital twin rendering engine, wherein, the device includes: a distance acquisition module, configured to acquire the target distance of each object to be displayed in the three-dimensional scene from the rendering camera; an interval determination module, configured to, for each object to be displayed, determine the distance interval to which the target distance of the object to be displayed belongs in a plurality of preset distance intervals as the target distance interval of the object to be displayed; a model determination module, configured to, for each object to be displayed, determine the object model at the level of detail corresponding to the target distance interval of the object to be displayed as the target object model of the object to be displayed; wherein, different distance intervals correspond to different levels of detail, for any distance interval, the longer the distance represented by this distance interval, the lower the level of detail corresponding to this distance interval; for any level of detail, the lower the level of detail, the lower the resolution of the object model at this level of detail; an image generation module, configured to generate an image to be displayed based on the target object models of each object to be displayed: based on the target object models of each object to be displayed, restore the image after eliminating moiré by fusing image texture data and three-dimensional model data feature maps through a multi-scale algorithm; a model establishment module, configured to, for each of the distance intervals, select multiple batches of sample data from the image texture maps of the objects in this distance interval, and use the sample data of each batch to respectively determine multiple primary object models of each object at the level of detail corresponding to this distance interval; For each of the objects, respectively perform model mixing on the primary object models of the object at the level of detail corresponding to the same distance interval to obtain the object model of the object at the level of detail corresponding to each distance interval, wherein the level of detail is used to determine the allocation of rendering resources in the image to be displayed according to the position and importance of the key nodes of the object in the display environment.

6. The device according to claim 5, wherein, the image generation module is specifically configured to: acquire the poses of each object to be displayed in the image to be displayed; For each object to be displayed, render the target object model of the object to be displayed according to the pose of the object to be displayed in the image to be displayed and the target distance of the object to be displayed, and generate a pixel area of the object to be displayed in the image to be displayed; Generate the image to be displayed based on the pixel areas of each object to be displayed in the image to be displayed.

7. The device according to claim 5, wherein, the device further comprises: a texture map acquisition module, configured to acquire the image texture maps of each object in the three-dimensional scene collected at different distances, the acquisition distances of the image texture maps of each object, and preset distance intervals; a texture map division module, configured to divide each image texture map into the image set corresponding to the distance interval to which the acquisition distance of the image texture map belongs; the model establishment module, configured to establish the object models of each object at the corresponding level of detail for each distance interval according to the image texture maps of each object in the image set corresponding to the distance interval.

8. The device according to claim 7, wherein, the device further comprises a color compensation module, configured to: calculate the color difference between adjacent pixels in the moiré pattern area of the image texture map with moiré pattern; for adjacent pixels with a color difference greater than a preset difference threshold, calculate a difference color according to the colors of the two pixels in the adjacent pixels, and insert pixels of the difference color between the adjacent pixels.

9. An electronic device, wherein, it comprises a processor and a memory; the memory is used for storing a computer program; the processor is configured to implement the method according to any one of claims 1-4 when executing the program stored on the memory.

10. A computer-readable storage medium, wherein, the computer-readable storage medium stores a computer program, and the computer program implements the method according to any one of claims 1-4 when executed by a processor.

Images