Real scene rendering method and device based on digital twinning and related equipment

Abstract

The disclosure provides a real scene rendering method and device based on digital twinning, and related equipment, specifically relates to image processing, digital twinning, virtual reality technology and other artificial intelligence fields, and can be applied in smart city, city governance and public security emergency scene. The scheme is: constructing an initial three-dimensional model; constructing a spherical panorama in a set spherical region according to a target panorama obtained by a camera shooting a plurality of target objects; determining a target scaling ratio according to height information of the camera from the horizontal ground, a set field of view radius of the camera and a radius of the set spherical region; mapping the initial three-dimensional model into the set spherical region according to the target scaling ratio to obtain a target three-dimensional model; and rendering the target three-dimensional model by using the spherical panorama. Thus, the data amount processed by the machine can be reduced, the machine resource occupation is reduced, the method can be applied in an electronic device with lower performance, and the applicability of the method is improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to the fields of image processing, digital twins, and virtual reality technology, and can be applied to smart cities, urban governance, and public security emergency scenarios. In particular, it relates to a real-scene rendering method, device, and related equipment based on digital twins. Background Technology

[0002] With the booming development of image technology and the digital economy, 3D model rendering is widely used in industries such as urban construction, film, games, and public security emergency response. Currently, rendering 3D models using images can create realistic virtual environments, meeting various demonstration or training needs and enabling users to have an immersive experience. Summary of the Invention

[0003] This disclosure provides a method, apparatus, and related equipment for real-world rendering based on digital twins.

[0004] According to one aspect of this disclosure, a real-scene rendering method based on digital twins is provided, comprising: constructing an initial three-dimensional model corresponding to multiple target objects based on three-dimensional spatial information of multiple target objects; acquiring a target panoramic image captured by a camera of the multiple target objects, and constructing a spherical panoramic image within a set spherical region based on the target panoramic image; determining a target scaling ratio based on the height information of the camera above the horizontal ground when it takes the picture, a set field of view radius of the camera, and the radius of the set spherical region; mapping the initial three-dimensional model onto the set spherical region according to the target scaling ratio to obtain a target three-dimensional model; and rendering the target three-dimensional model using the spherical panoramic image.

[0005] According to another aspect of this disclosure, a digital twin-based real-scene rendering device is provided, comprising: a first construction module, configured to construct an initial 3D model corresponding to the multiple target objects based on the 3D spatial information of multiple target objects; a second construction module, configured to acquire a target panoramic image captured by a camera of the multiple target objects, and construct a spherical panoramic image within a set spherical region based on the target panoramic image; a determination module, configured to determine a target scaling ratio based on the height information of the camera above the horizontal ground when it takes the picture, a set field of view radius of the camera, and the radius of the set spherical region; a mapping module, configured to map the initial 3D model to the set spherical region according to the target scaling ratio to obtain a target 3D model; and a rendering module, configured to render the target 3D model using the spherical panoramic image.

[0006] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect of this disclosure.

[0007] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method described in the first aspect of this disclosure.

[0008] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described in the first aspect of this disclosure.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0011] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure;

[0012] Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure;

[0013] Figure 3 This is a schematic diagram according to the third embodiment of the present disclosure;

[0014] Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure;

[0015] Figure 5 This is a schematic diagram of a defined spherical region according to an embodiment of the present disclosure;

[0016] Figure 6 This is a schematic diagram according to the fifth embodiment of the present disclosure;

[0017] Figure 7 This is a schematic diagram according to the sixth embodiment of the present disclosure;

[0018] Figure 8 This is a schematic diagram of the rendered target 3D model according to an embodiment of the present disclosure;

[0019] Figure 9 This is a schematic diagram according to the seventh embodiment of the present disclosure;

[0020] Figure 10 This is a block diagram of an electronic device used to implement the digital twin-based real-scene rendering method of the embodiments of this disclosure. Detailed Implementation

[0021] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0022] In related technologies, 3D models of cities, movies, games, and public security emergency situations are created using engines or 3D modeling tools based on information such as these. Then, real-world photographs are captured, processed, and textures are extracted for the model. These textures are then added to the model's surface during the modeling or rendering stage. However, the information acquisition and processing costs are high, and rendering the model requires a large number of real-world images, placing high demands on machine resources and making it unusable on low-performance electronic devices.

[0023] To address the aforementioned issues, this disclosure proposes a method, apparatus, and related equipment for real-scene rendering based on digital twins.

[0024] The following description, with reference to the accompanying drawings, outlines a method, apparatus, and related equipment for rendering real-world scenes based on digital twins, according to embodiments of this disclosure.

[0025] Figure 1 This is a schematic diagram according to the first embodiment of the present disclosure. It should be noted that the embodiments of the present disclosure are illustrated by way of the rendering method being configured in a digital twin-based real-scene rendering device. The rendering device can be applied to any electronic device so that the electronic device can perform digital twin-based real-scene rendering functions.

[0026] Among them, electronic devices can be any device with computing capabilities, such as personal computers (PCs), mobile terminals, etc. Mobile terminals can be hardware devices with various operating systems, touch screens and / or displays, such as mobile phones, tablets, personal digital assistants, wearable devices, etc.

[0027] like Figure 1 As shown, this digital twin-based real-scene rendering method may include the following steps:

[0028] Step 101: Based on the three-dimensional spatial information of multiple target objects, construct an initial three-dimensional model corresponding to the multiple target objects.

[0029] In this embodiment of the disclosure, an engine or 3D modeling tool can be used to construct a model based on the three-dimensional spatial information of multiple target objects to obtain an initial three-dimensional model. The target objects can be urban buildings, game characters and scenes, movie characters and scenes, public security emergency scenes, etc. The initial three-dimensional model can be a transparent three-dimensional model. The three-dimensional spatial information can include the size information, position information, and angle information of each target object.

[0030] Step 102: Obtain the target panoramic image captured by the camera of multiple target objects, and construct a spherical panoramic image within a set spherical region based on the target panoramic image.

[0031] Furthermore, a camera can be used to take 360-degree panoramic photos of multiple target objects. Based on the panoramic images of the multiple target objects taken in the panoramic photos, a target panoramic image is determined. Then, the target panoramic image is mapped onto the spherical surface of a set spherical region to obtain a spherical panoramic image. It should be noted that the center of the set spherical region can be the location of the camera.

[0032] Step 103: Determine the target scaling ratio based on the camera's height above the horizontal ground, the camera's set field of view radius, and the radius of the set spherical region.

[0033] Furthermore, in order to reduce the size of the initial three-dimensional model, the initial three-dimensional model can be scaled up and the scaled initial three-dimensional model can be mapped onto a set spherical region. Therefore, in this embodiment of the disclosure, the scaling ratio of the initial three-dimensional model can be determined.

[0034] As one possible implementation of this disclosure, the target scaling ratio can be determined based on the height information of the camera above the horizontal ground when taking pictures, the camera's set field of view radius, and the radius of the set spherical region.

[0035] Step 104: Based on the target scaling ratio, map the initial 3D model onto the set spherical region to obtain the target 3D model.

[0036] Furthermore, based on the target scaling ratio, the initial 3D model is scaled and then translated into a set spherical region to obtain the target 3D model. It should be noted that each target object in the target 3D model is aligned with the corresponding target object in the spherical panoramic image.

[0037] Step 105: Render the target 3D model using a spherical panoramic image.

[0038] Then, a spherical panoramic image is used to render the target 3D model to obtain the rendered 3D model.

[0039] Therefore, when operating on the spherical panoramic image, it is possible to operate on the target 3D model, which can meet the user's personalized needs.

[0040] In summary, this method constructs initial 3D models corresponding to multiple target objects based on their 3D spatial information; acquires panoramic images of the targets captured by a camera, and constructs a spherical panoramic image within a defined spherical region based on these panoramic images; determines the target scaling ratio based on the camera's height above the horizontal ground, the camera's defined field of view radius, and the radius of the defined spherical region; maps the initial 3D model onto the defined spherical region according to the target scaling ratio to obtain the target 3D model; and renders the target 3D model using the spherical panoramic image. Thus, by mapping the initial 3D model onto the defined spherical region according to the target scaling ratio, a smaller target 3D model can be obtained. Rendering this smaller target 3D model using a single spherical panoramic image reduces the amount of data processed by the machine, thereby reducing machine resource consumption and making it applicable to lower-performance electronic devices, thus improving the method's applicability.

[0041] To clearly illustrate how the above embodiments map the initial 3D model onto a set spherical region according to the target scaling ratio to obtain the target 3D model, this disclosure proposes another real-scene rendering method based on digital twins.

[0042] Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure.

[0043] like Figure 2 As shown, the rendering method may include the following steps:

[0044] Step 201: Based on the three-dimensional spatial information of multiple target objects, construct an initial three-dimensional model corresponding to the multiple target objects.

[0045] Step 202: Obtain the target panoramic image captured by the camera of multiple target objects, and construct a spherical panoramic image within a set spherical region based on the target panoramic image.

[0046] Step 203: Determine the target scaling ratio based on the camera's height above the horizontal ground, the camera's set field of view radius, and the radius of the set spherical region.

[0047] Step 204: Scale the initial 3D model according to the target scaling ratio to obtain the scaled initial 3D model.

[0048] In this embodiment of the disclosure, in order to reduce the size of the model, a target scaling ratio is used to scale the initial 3D model, resulting in a scaled initial 3D model. For example, a target scaling ratio can be used to scale the size information of the initial 3D model.

[0049] Step 205: Align each target object in the scaled initial 3D model with each target object in the spherical panoramic image to obtain the target 3D model.

[0050] Furthermore, in order to import the scaled initial 3D model into the spherical region, the scaled initial 3D model can be translated into the spherical region.

[0051] To avoid deviations between target objects in the target 3D model and target objects in the spherical panoramic image, which could affect subsequent rendering effects, as a possible implementation of this disclosure, the target objects of the scaled initial 3D model can be aligned with the target objects in the spherical panoramic image to obtain the target 3D model.

[0052] Step 206: Render the target 3D model using a spherical panoramic image.

[0053] It should be noted that the execution process of steps 201 to 203 and step 206 can be implemented in any of the embodiments of this disclosure. This disclosure does not limit this and will not elaborate further.

[0054] In summary, by scaling the initial 3D model according to the target scaling ratio, a scaled initial 3D model is obtained. Then, each target object in the scaled initial 3D model is aligned with each target object in the spherical panoramic image to obtain the target 3D model. Thus, by aligning each target object in the scaled initial 3D model with each target object in the spherical panoramic image, the initial 3D model can be mapped to a set spherical region. Furthermore, the size of the target 3D model within the set spherical region can be reduced, decreasing the amount of data processed by the machine. Simultaneously, the rendering effect of the target 3D model can be further improved.

[0055] To clearly illustrate how the above embodiments map the initial 3D model onto a set spherical region according to the target scaling ratio to obtain the target 3D model, this disclosure proposes another real-scene rendering method based on digital twins.

[0056] Figure 3 This is a schematic diagram according to the third embodiment of the present disclosure.

[0057] like Figure 3 As shown, the rendering method may include the following steps:

[0058] Step 301: Based on the three-dimensional spatial information of multiple target objects, construct an initial three-dimensional model corresponding to the multiple target objects.

[0059] Step 302: Obtain the target panoramic image captured by the camera of multiple target objects, and construct a spherical panoramic image within a set spherical region based on the target panoramic image.

[0060] Step 303: Determine the target scaling ratio based on the camera's height above the horizontal ground, the camera's set field of view radius, and the radius of the set spherical region.

[0061] Step 304: Scale the initial 3D model according to the target scaling ratio to obtain the scaled initial 3D model.

[0062] Step 305: Based on the mapping relationship between the image coordinate system and the world coordinate system, transform each pixel in the spherical panoramic image to the world coordinate system to obtain the pose information of each point of each target object in the spherical panoramic image in the world coordinate system.

[0063] In this embodiment of the disclosure, in order to align each target object of the scaled initial 3D model with each target object in the spherical panoramic image and further improve the model rendering effect, each pixel of the spherical panoramic image can be transformed to the world coordinate system, and the pose information of each point of each target object in the scaled initial 3D model can be aligned with the pose information of each point in the world coordinate system.

[0064] As one possible implementation of this disclosure, each pixel in the spherical panoramic image can be transformed to the world coordinate system according to the mapping relationship between the image coordinate system and the world coordinate system, thereby obtaining the pose information of each point of the target object in the spherical panoramic image in the world coordinate system.

[0065] Step 306: Align the pose information of each point of each target object in the scaled initial 3D model with the pose information of each point in the world coordinate system to obtain the target 3D model.

[0066] Then, by aligning the pose information of each point of each target object in the scaled initial 3D model with the pose information of each point in the world coordinate system, the target 3D model can be obtained.

[0067] Step 307: Render the target 3D model using a spherical panoramic image.

[0068] It should be noted that the execution process of steps 301 to 304 and step 307 can be implemented in any of the embodiments of this disclosure. This disclosure does not limit this and will not elaborate further.

[0069] In summary, by transforming each pixel in the spherical panoramic image to the world coordinate system based on the mapping relationship between the image coordinate system and the world coordinate system, the pose information of each point of each target object in the spherical panoramic image in the world coordinate system is obtained. The pose information of each point of each target object in the scaled initial 3D model is then aligned with the pose information of each point in the world coordinate system to obtain the target 3D model. Therefore, by transforming each pixel in the spherical panoramic image to the world coordinate system and aligning the pose information of each point of each target object in the scaled initial 3D model with the pose information of each point in the world coordinate system, the initial 3D model can be mapped to a set spherical region. This reduces the size of the target 3D model within the set spherical region, decreases the amount of data processed by the machine, and further improves the rendering effect of the target 3D model.

[0070] To clearly illustrate how the above embodiments determine the target scaling ratio based on the camera's height above the horizontal ground, the camera's set field of view radius, and the radius of the set spherical region, this disclosure proposes another real-scene rendering method based on digital twins.

[0071] Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure.

[0072] like Figure 4 As shown, the rendering method may include the following steps:

[0073] Step 401: Based on the three-dimensional spatial information of multiple target objects, construct an initial three-dimensional model corresponding to the multiple target objects.

[0074] Step 402: Obtain target panoramic images captured by the camera on multiple target objects, and construct a spherical panoramic image within a set spherical region based on the target panoramic images.

[0075] Step 403: Determine the target scaling parameters based on the height information of the camera above the horizontal ground when taking the picture and the camera's set field of view radius.

[0076] In this embodiment of the disclosure, the target scaling parameters can be calculated based on the height information of the camera above the horizontal ground when it takes a picture, and the set field of view radius of the camera.

[0077] Step 404: The ratio of the radius of the set spherical region to the target scaling parameter is used as the target scaling ratio.

[0078] Furthermore, the radius of the set spherical region is compared with the target scaling parameter, and the ratio of the radius of the set spherical region to the target scaling parameter is used as the target scaling ratio.

[0079] Step 405: Based on the target scaling ratio, map the initial 3D model onto the set spherical region to obtain the target 3D model.

[0080] For example, such as Figure 5 As shown, taking the height information of the camera above the horizontal ground as H, the camera's set field of view radius as D, and the radius of the spherical region as r as an example, where the camera can be a perspective camera, the target scaling ratio k can be expressed by the following formula:

[0081]

[0082] Then, based on the target scaling ratio k, the initial 3D model is scaled to obtain a scaled initial 3D model. This scaled 3D model can then be translated to align each target object in the scaled initial 3D model with the target objects in the spherical panoramic image, thus obtaining the target 3D model. The position of the camera in the world coordinate system is defined as P. camera Taking the initial 3D model (X,Y,H) and the point positions P(x,y,z) in the initial 3D model as an example, to transform them into the point positions P(x′,y′,z′) in the target 3D model, the specific formula can be expressed as follows:

[0083] x′=kx+(1-k)X;

[0084] y′=ky+(1-k)Y;

[0085] z′=kz+(1-k)H;

[0086] In the initial 3D model, the position P(x,y,z) of a point is generated based on the point's actual position in the world coordinate system.

[0087] Step 406: Render the target 3D model using a spherical panoramic image.

[0088] It should be noted that the execution processes of steps 401 to 402 and step 406 can be implemented in any of the embodiments of this disclosure. This disclosure does not limit these processes and will not elaborate further.

[0089] In summary, the target scaling parameters are determined based on the camera's height above the horizontal ground and the camera's set field of view radius. The ratio of the radius of the set spherical region to the target scaling parameters is used as the target scaling ratio. Thus, the target scaling ratio can be obtained using the camera's height above the horizontal ground, the camera's set field of view radius, and the radius of the set spherical region.

[0090] To clearly illustrate how the above embodiments obtain initial panoramic images corresponding to multiple target objects and construct spherical panoramic images within a set spherical region based on the initial panoramic images, this disclosure proposes another real-scene rendering method based on digital twins.

[0091] Figure 6 This is a schematic diagram according to the fifth embodiment of the present disclosure.

[0092] like Figure 6 As shown, the rendering method may include the following steps:

[0093] Step 601: Based on the three-dimensional spatial information of multiple target objects, construct an initial three-dimensional model corresponding to the multiple target objects.

[0094] Step 602: Perform panoramic photography on each target object to obtain an initial panoramic image of each target object.

[0095] In this embodiment of the disclosure, by taking a 306-degree panoramic view of the target object, an initial panoramic image of each target object can be obtained, wherein the number of initial panoramic images corresponding to each target object can be one or more.

[0096] Step 603: Stitch together the initial panoramic images of each target object to obtain a target panoramic image corresponding to multiple target objects.

[0097] Furthermore, based on the pose information of each target object, the initial panoramic images of each target object can be stitched together to obtain a target panoramic image corresponding to multiple target objects.

[0098] Step 604: Map the target panoramic image onto the spherical surface of the designated spherical region to obtain a spherical panoramic image.

[0099] Furthermore, based on the area of ​​the inner surface of the set spherical region, the target panoramic image is mapped to obtain a spherical panoramic image, so that the area of ​​the spherical panoramic image is consistent with the area of ​​the inner surface of the set spherical region.

[0100] Step 605: Determine the target scaling ratio based on the camera's height above the horizontal ground, the camera's set field of view radius, and the radius of the set spherical region.

[0101] Step 606: Map the initial 3D model onto the set spherical region according to the target scaling ratio to obtain the target 3D model.

[0102] Step 607: Render the target 3D model using a spherical panoramic image.

[0103] It should be noted that the execution process of steps 601, 605 to 607 can be implemented in any of the embodiments of this disclosure. This disclosure does not limit this and will not elaborate further.

[0104] In summary, by taking panoramic photos of each target object, an initial panoramic image of each target object is obtained. These initial panoramic images are then stitched together to obtain a target panoramic image corresponding to multiple target objects. Thus, by stitching and mapping the initial panoramic images of each target object, a spherical panoramic image can be obtained. Therefore, rendering a small-scale 3D target model using a single spherical panoramic image can reduce the amount of data processed by the machine, thereby reducing machine resource consumption. This method can be applied to low-performance electronic devices, improving its applicability.

[0105] To clearly illustrate how a spherical panoramic image is used to render a target 3D model, this disclosure proposes an alternative real-world rendering method based on digital twins.

[0106] Figure 7 This is a schematic diagram according to the sixth embodiment of the present disclosure.

[0107] like Figure 7 As shown, the rendering method may include the following steps:

[0108] Step 701: Based on the three-dimensional spatial information of multiple target objects, construct an initial three-dimensional model corresponding to the multiple target objects.

[0109] Step 702: Obtain the target panoramic image captured by the camera of multiple target objects, and construct a spherical panoramic image within a set spherical region based on the target panoramic image.

[0110] Step 703: Determine the target scaling ratio based on the camera's height above the horizontal ground, the camera's set field of view radius, and the radius of the set spherical region.

[0111] Step 704: Map the initial 3D model onto the set spherical region according to the target scaling ratio to obtain the target 3D model.

[0112] Step 705: Obtain the coordinate information of the vertices of each target object in the target 3D model.

[0113] In this embodiment of the disclosure, the coordinate information of the vertices of each target object in the target 3D model can be generated based on the real coordinate information of each vertex of each target object in the world coordinate system.

[0114] Step 706: Construct multiple triangular faces based on the coordinate information of the vertices of each target object.

[0115] Step 707: Use a spherical panoramic image to render multiple triangular faces.

[0116] Furthermore, multiple triangular faces can be constructed based on the coordinate information of the vertices of each target object, and the pose information of each point of each target object in the spherical panoramic image in the world coordinate system can be used to render the multiple triangular faces.

[0117] Furthermore, in response to the trigger operation of the spherical panoramic image, the location information of the trigger point of the spherical panoramic image is determined; the location information of the trigger point is marked with color to obtain the color-marked location information; the color-marked location information is displayed. Thus, the spherical panoramic image can be used to operate on the target 3D model, satisfying the user's personalized needs.

[0118] In summary, by obtaining the coordinate information of the vertices of each target object in the target 3D model and using a spherical panoramic image to render multiple triangular faces, it is possible to render the target 3D model using a single spherical panoramic image. This reduces the amount of data processed by the machine, thereby reducing machine resource consumption and making it applicable to low-performance electronic devices, thus improving the applicability of the method.

[0119] To clearly illustrate the above embodiments, examples are given below.

[0120] For example, such as Figure 8 As shown, the real-scene rendering method based on digital twins in this disclosure embodiment may include the following steps:

[0121] 1. Construct a transparent 3D model;

[0122] 2. Construct a spherical panoramic image;

[0123] 3. Based on the height information of the camera above the horizontal ground when shooting, the camera's set field of view radius, and the radius of the set spherical region, determine the target scaling ratio, and scale the 3D model according to the target scaling ratio to obtain a scaled transparent 3D model, and translate the scaled transparent 3D model into the spherical region.

[0124] 4. A spherical panoramic view can be used to render the scaled-down transparent 3D model, allowing users to manipulate the scaled-down transparent 3D model while operating on the spherical panoramic view.

[0125] The digital twin-based real-scene rendering method disclosed in this embodiment constructs an initial 3D model corresponding to multiple target objects based on their 3D spatial information; acquires a panoramic image of the target objects captured by a camera, and constructs a spherical panoramic image within a set spherical region based on the panoramic image; determines the target scaling ratio based on the camera's height above the horizontal ground, the camera's set field of view radius, and the radius of the set spherical region; maps the initial 3D model onto the set spherical region according to the target scaling ratio to obtain the target 3D model; and renders the target 3D model using the spherical panoramic image. Thus, by mapping the initial 3D model onto the set spherical region according to the target scaling ratio, a smaller target 3D model can be obtained. Rendering a smaller target 3D model using a single spherical panoramic image reduces the amount of data processed by the machine, thereby reducing machine resource consumption and making the method applicable to lower-performance electronic devices, thus improving its applicability.

[0126] To achieve the above embodiments, this disclosure proposes a real-scene rendering device based on digital twins.

[0127] Figure 9 This is a schematic diagram according to the seventh embodiment of this disclosure. (See diagram below.) Figure 9 As shown, the digital twin-based real-scene rendering device 900 includes: a first construction module 910, a second construction module 920, a determination module 930, a mapping module 940, and a rendering module 950.

[0128] The system comprises the following modules: a first construction module 910, used to construct an initial 3D model corresponding to multiple target objects based on their 3D spatial information; a second construction module 920, used to acquire a panoramic image of the target objects captured by a camera, and construct a spherical panoramic image within a set spherical region based on the panoramic image; a determination module 930, used to determine the target scaling ratio based on the camera's height above the horizontal ground, the camera's set field of view radius, and the radius of the set spherical region; a mapping module 940, used to map the initial 3D model onto the set spherical region according to the target scaling ratio to obtain the target 3D model; and a rendering module 950, used to render the target 3D model using the spherical panoramic image.

[0129] As one possible implementation of this disclosure, the mapping module 940 is configured to: scale the initial 3D model according to the target scaling ratio to obtain a scaled initial 3D model; and align each target object of the scaled initial 3D model with each target object in the spherical panoramic image to obtain a target 3D model.

[0130] As one possible implementation of this disclosure, the mapping module 940 is further configured to: transform each pixel in the spherical panoramic image to the world coordinate system according to the mapping relationship between the image coordinate system and the world coordinate system, so as to obtain the pose information of each point of each target object in the spherical panoramic image in the world coordinate system; and align the pose information of each point of each target object in the scaled initial three-dimensional model with the pose information of each point in the world coordinate system to obtain the target three-dimensional model.

[0131] As one possible implementation of this disclosure, the determining module 930 is configured to: determine the target scaling parameter based on the height information of the camera above the horizontal ground when shooting and the camera's set field of view radius; and use the ratio of the radius of the set spherical region to the target scaling parameter as the target scaling ratio.

[0132] As one possible implementation of this disclosure, the second construction module 920 is used to: perform panoramic photography on each target object to obtain an initial panoramic image of each target object; stitch together the initial panoramic images of each target object to obtain a target panoramic image corresponding to multiple target objects; and map the target panoramic images onto the spherical surface of a set spherical region to obtain a spherical panoramic image.

[0133] As one possible implementation of this disclosure, the rendering module 950 is used to: obtain the coordinate information of the vertices of each target object in the target 3D model; construct multiple triangular faces based on the coordinate information of the vertices of each target object; and render the multiple triangular faces using a spherical panoramic image.

[0134] As one possible implementation of this disclosure, the digital twin-based real-scene rendering device 900 further includes: an acquisition module, a marking module, and a display module.

[0135] The module includes an acquisition module, which responds to a trigger operation on the spherical panoramic image and determines the location information of the trigger point; a marking module, which marks the location information of the trigger point with color to obtain the color-marked location information; and a display module, which displays the color-marked location information.

[0136] The digital twin-based real-scene rendering device of this disclosure constructs an initial 3D model corresponding to multiple target objects based on their 3D spatial information; acquires a panoramic image of the target objects captured by a camera, and constructs a spherical panoramic image within a set spherical region based on the panoramic image; determines the target scaling ratio based on the camera's height above the horizontal ground, the camera's set field of view radius, and the radius of the set spherical region; maps the initial 3D model onto the set spherical region according to the target scaling ratio to obtain the target 3D model; and renders the target 3D model using the spherical panoramic image. Thus, by mapping the initial 3D model onto the set spherical region according to the target scaling ratio, a smaller target 3D model can be obtained. Rendering a smaller target 3D model using a single spherical panoramic image reduces the amount of data processed by the machine, thereby reducing machine resource consumption and making it applicable to lower-performance electronic devices, thus improving the applicability of the method.

[0137] To implement the above embodiments, this disclosure also proposes an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute the digital twin-based real-scene rendering method of the above embodiments.

[0138] To implement the above embodiments, this disclosure also proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the digital twin-based real-scene rendering method described in the above embodiments.

[0139] To implement the above embodiments, this disclosure also proposes a computer program product, including a computer program that, when executed by a processor, implements the real-scene rendering method based on digital twins described in the above embodiments.

[0140] It should be noted that the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solution disclosed herein are all carried out with the consent of the user, and all comply with the provisions of relevant laws and regulations, and do not violate public order and good morals.

[0141] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0142] Figure 10A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0143] like Figure 10 As shown, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1002 or a computer program loaded from storage unit 1008 into random access memory (RAM) 1003. The RAM 1003 may also store various programs and data required for the operation of device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0144] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0145] The computing unit 1001 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as rendering methods. For example, in some embodiments, the rendering method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the rendering method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform a digital twin-based real-world rendering method by any other suitable means (e.g., by means of firmware).

[0146] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0147] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0148] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0149] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0150] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0151] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0152] It's important to note that artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0153] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0154] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A real-scene rendering method based on digital twins, comprising: Based on the three-dimensional spatial information of multiple target objects, construct an initial three-dimensional model corresponding to the multiple target objects; Acquire target panoramic images captured by a camera on multiple target objects, and construct a spherical panoramic image within a defined spherical region based on the target panoramic images; The target scaling ratio is determined based on the camera's height above the horizontal ground when it takes the picture, the camera's set field of view radius, and the radius of the set spherical region. Based on the target scaling ratio, the initial 3D model is mapped onto the set spherical region to obtain the target 3D model; The target 3D model is rendered using the spherical panoramic image; The step of mapping the initial 3D model onto the set spherical region according to the target scaling ratio to obtain the target 3D model includes: The initial 3D model is scaled according to the target scaling ratio to obtain a scaled initial 3D model. Align each target object in the scaled initial 3D model with each target object in the spherical panoramic image to obtain the target 3D model; The step of aligning each target object of the scaled initial 3D model with each target object in the spherical panoramic image to obtain the target 3D model includes: Based on the mapping relationship between the image coordinate system and the world coordinate system, each pixel in the spherical panoramic image is transformed to the world coordinate system to obtain the pose information of each point of each target object in the spherical panoramic image in the world coordinate system. Align the pose information of each point of each target object in the scaled initial 3D model with the pose information of each point in the world coordinate system to obtain the target 3D model.

2. The method according to claim 1, wherein, The step of determining the target scaling ratio based on the camera's height above the horizontal ground during shooting, the camera's set field of view radius, and the radius of the set spherical region includes: The target scaling parameters are determined based on the height information of the camera above the horizontal ground when it takes a picture and the camera's set field of view radius; The ratio of the radius of the defined spherical region to the target scaling parameter is used as the target scaling ratio.

3. The method according to claim 1, wherein, The step of obtaining an initial panoramic image corresponding to the plurality of target objects, and constructing a spherical panoramic image within a set spherical region based on the initial panoramic image, includes: A panoramic view of each target object is taken to obtain an initial panoramic image of each target object; The initial panoramic images of each target object are stitched together to obtain multiple target panoramic images corresponding to the target objects; The target panoramic image is mapped onto the spherical surface of the designated spherical region to obtain the spherical panoramic image.

4. The method according to claim 1, wherein, The process of rendering the target 3D model using the spherical panoramic image includes: Obtain the coordinate information of the vertices of each target object in the target 3D model; Based on the coordinate information of the vertices of each target object, construct multiple triangular faces; The spherical panoramic image is used to render the multiple triangular faces.

5. The method according to claims 1-4, wherein, The method further includes: In response to a triggering operation on the spherical panoramic image, the location information of the trigger point of the spherical panoramic image is determined; The location information of the trigger point is color-marked to obtain the color-marked location information; The position information following the color marking is displayed.

6. A reality rendering device based on digital twins, comprising: The first construction module is used to construct an initial three-dimensional model corresponding to the multiple target objects based on the three-dimensional spatial information of the multiple target objects; The second construction module is used to acquire target panoramic images captured by the camera on multiple target objects, and to construct a spherical panoramic image within a set spherical region based on the target panoramic images. The determination module is used to determine the target scaling ratio based on the height information of the camera above the horizontal ground when it takes a picture, the set field of view radius of the camera, and the radius of the set spherical region; A mapping module is used to map the initial 3D model onto the set spherical region according to the target scaling ratio to obtain the target 3D model; A rendering module is used to render the target 3D model using the spherical panoramic image; The mapping module is used for: The initial 3D model is scaled according to the target scaling ratio to obtain a scaled initial 3D model. Align each target object in the scaled initial 3D model with each target object in the spherical panoramic image to obtain the target 3D model; The mapping module is also used for: Based on the mapping relationship between the image coordinate system and the world coordinate system, each pixel in the spherical panoramic image is transformed to the world coordinate system to obtain the pose information of each point of each target object in the spherical panoramic image in the world coordinate system. Align the pose information of each point of each target object in the scaled initial 3D model with the pose information of each point in the world coordinate system to obtain the target 3D model.

7. The apparatus according to claim 6, wherein, The determining module is used for: The target scaling parameters are determined based on the height information of the camera above the horizontal ground when it takes a picture and the camera's set field of view radius; The ratio of the radius of the defined spherical region to the target scaling parameter is used as the target scaling ratio.

8. The apparatus according to claim 6, wherein, The second building module is used for: A panoramic view of each target object is taken to obtain an initial panoramic image of each target object; The initial panoramic images of each target object are stitched together to obtain multiple target panoramic images corresponding to the target objects; The target panoramic image is mapped onto the spherical surface of the designated spherical region to obtain the spherical panoramic image.

9. The apparatus according to claim 6, wherein, The rendering module is used for: Obtain the coordinate information of the vertices of each target object in the target 3D model; Based on the coordinate information of the vertices of each target object, construct multiple triangular faces; The spherical panoramic image is used to render the multiple triangular faces.

10. The apparatus according to claims 6-9, wherein, The device further includes: The acquisition module is used to determine the location information of the trigger point of the spherical panoramic image in response to the trigger operation of the spherical panoramic image; A marking module is used to color-mark the location information of the trigger point to obtain the color-marked location information; The display module is used to display the position information after the color marking.

11. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-5.

13. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-5.

Images