3D Scene Generation Method, Device, Electronic Device, and Storage Medium

Abstract

The present disclosure relates to a three-dimensional scene generation method, apparatus, electronic device, and storage medium. The method includes: obtaining a planar structure diagram of the physical scene and feature description data of each entity object in the physical scene according to the scan data of the physical scene, where the planar structure diagram represents the planar structure of the physical scene based on the two-dimensional structure of the entity object; matching at least one model component for the entity object in an asset library according to the two-dimensional structure of the entity object and the feature description data of the entity object; and building a three-dimensional model of the entity object at the position calibrated by the corresponding two-dimensional structure through the model components matched for the entity object to generate a virtual scene corresponding to the physical scene. According to the embodiments of the present disclosure, the accuracy of the generated virtual scene can be improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of computer scene reconstruction, and more particularly, to a three-dimensional scene generation method, apparatus, and storage medium. Background Art

[0002] Scene reconstruction refers to performing three-dimensional modeling on an entity scene through scene scan data to obtain a three-dimensional virtual scene that is substantially the same as the entity scene. In scene reconstruction, affected by factors such as the completeness of scene scanning and the accuracy of reconstruction algorithms, the reconstruction accuracy of the entity scene is low and cannot meet the requirements of applications such as virtual shooting. Summary of the Invention

[0003] In view of this, the present disclosure provides a three-dimensional scene generation method, apparatus, electronic device, and storage medium, which can improve the reconstruction accuracy of the entity scene.

[0004] According to one aspect of the present disclosure, there is provided a three-dimensional scene generation method, including:

[0005] Based on the scan data of the entity scene, obtain a planar structure diagram of the entity scene and feature description data of each entity object in the entity scene, where the planar structure diagram represents the planar structure of the entity scene based on the two-dimensional structure of the entity object;

[0006] Based on the two-dimensional structure of the entity object and the feature description data of the entity object, match at least one model component for the entity object in an asset library;

[0007] Generate a three-dimensional model of the entity object at the position calibrated by the corresponding two-dimensional structure through the model components matched for the entity object, and generate a virtual scene corresponding to the entity scene.

[0008] In a possible implementation manner, the matching at least one model component for the entity object in the asset library includes:

[0009] Match at least one model component for the entity object in the asset group to which the entity object belongs in the asset library, where the asset library includes a plurality of asset groups divided according to object categories.

[0010] In a possible implementation manner, after obtaining the planar structure diagram of the entity scene and before matching at least one model component for the entity object in the asset library, the method further includes:

[0011] Determine a regularized shape adapted to the initial shape according to the initial shape of the two-dimensional structure in the planar structure diagram;

[0012] Regularize the corresponding two-dimensional structure according to the regularized shape to update the two-dimensional structure.

[0013] In a possible implementation manner, matching at least one model component for the entity object in the asset library according to the two-dimensional structure of the entity object and the feature description data of the entity object includes:

[0014] Match an initial model component with a feature similarity that meets the matching condition for the entity object in the asset library according to the two-dimensional structure of the entity object and the feature description data of the entity object;

[0015] Adjust the parameter values of the initial model component according to the feature description data of the entity object to obtain a model component that matches the entity object.

[0016] In a possible implementation manner, obtaining the planar structure diagram of the entity scene and the feature description data of each entity object in the entity scene according to the scan data of the entity scene includes:

[0017] Perform point cloud reconstruction on the entity scene according to the scan data of the entity scene to obtain the three-dimensional point cloud of the entity scene;

[0018] Segment the three-dimensional point cloud of the entity scene into multiple single-point clouds, where one single-point cloud corresponds to one entity object in the entity scene;

[0019] Obtain the feature description data of the corresponding entity object and the planar structure diagram of the entity scene according to each segmented single-point cloud.

[0020] In a possible implementation manner, the scan data includes multiple frames of images collected by aerial photography of the entity scene and the camera parameter data corresponding to each image.

[0021] In a possible implementation manner, the feature description data of the entity object includes at least one of attribute description data, dimension description data, and orientation description data.

[0022] According to another aspect of the present disclosure, a three-dimensional scene generation device is provided, including:

[0023] A data processing module for obtaining the planar structure diagram of the entity scene and the feature description data of each entity object in the entity scene according to the scan data of the entity scene, where the planar structure diagram represents the planar structure of the entity scene based on the two-dimensional structure of the entity object;

[0024] A component matching module, configured to match at least one model component for the entity object in an asset library according to the two-dimensional structure of the entity object and the feature description data of the entity object;

[0025] A scene generation module, configured to build a three-dimensional model of the entity object at the position calibrated by the corresponding two-dimensional structure through the model components matched for the entity object, and generate a virtual scene corresponding to the entity scene.

[0026] According to another aspect of the present disclosure, there is also provided an electronic device, including:

[0027] A processor;

[0028] A memory for storing instructions executable by the processor;

[0029] Wherein, the processor is configured to implement the above method when executing the instructions stored in the memory.

[0030] According to another aspect of the present disclosure, there is also provided a non-volatile computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above method is implemented.

[0031] According to the embodiments of the present disclosure, by analyzing the scan data obtained by scanning the entity scene, the planar structure diagram of the entity scene and the feature description data of each entity object in the entity scene can be obtained. According to the two-dimensional structure representing the entity object in the planar structure diagram and the feature description data of the entity object, at least one model component can be matched for the entity object from a preset asset library, and then, through the matched model components, a three-dimensional model of the entity object is built at the position calibrated by the corresponding two-dimensional structure, completing the reconstruction of the entity scene and generating a virtual scene corresponding to the entity scene. In this embodiment, since the model components in the asset library have high precision, therefore, the virtual scene obtained by reconstructing the entity scene by matching the model components will also have high precision, and can achieve a high degree of restoration of the entity scene, meeting the requirements of applications such as virtual shooting.

[0032] Through the following detailed description of the exemplary embodiments of the present invention with reference to the accompanying drawings, other features and advantages of the present disclosure will become clear. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The drawings incorporated in the specification and constituting a part of the specification illustrate embodiments of the present invention and, together with the description, are used to explain the principles of the present invention.

[0034] Figure 1 A schematic diagram showing an application scenario according to some embodiments of the present disclosure;

[0035] Figure 2 Shows a schematic flowchart of a three - dimensional scene generation method according to some embodiments of the present disclosure;

[0036] Figure 3 Shows a schematic diagram of the implementation process of a three - dimensional scene generation method according to some embodiments of the present disclosure;

[0037] Figure 4 Shows a schematic diagram of the operation of model component matching according to some embodiments of the present disclosure;

[0038] Figure 5 Shows a block diagram of the structure of a three - dimensional scene generation device according to some embodiments of the present disclosure;

[0039] Figure 6 Shows a block diagram of the structure of an electronic device according to some embodiments of the present disclosure;

[0040] Figure 7 Shows a block diagram of the structure of an electronic device according to some other embodiments of the present disclosure. Detailed implementation manners

[0041] Now, various exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be noted that: Unless otherwise specifically stated, the relative arrangements of components and steps, numerical expressions, and values set forth in these embodiments do not limit the scope of the present invention.

[0042] The following description of at least one exemplary embodiment is merely illustrative in nature and in no way limits the present invention or its application or use.

[0043] Techniques, methods, and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, the techniques, methods, and devices should be regarded as part of the specification.

[0044] In all the examples shown and discussed here, any specific values should be construed as merely exemplary and not as limitations. Therefore, other examples of the exemplary embodiments may have different values.

[0045] It should be noted that: Similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0046] The present disclosure relates to a scene reconstruction solution for reconstructing an entity scene. Here, scene reconstruction refers to the process of using computer technology to restore the entity scene based on the scan data obtained by scanning the entity scene, and obtaining a three-dimensional virtual scene corresponding to the entity scene. Among them, the scan data can be image data obtained by aerial photographing the entity scene, or a data set of spatial points obtained by laser scanning the entity scene, etc.

[0047] As Figure 1 shown, in an application scenario, a flying device 1000 such as a drone can carry a camera, a laser device, etc. to scan the entity scene to obtain scan data of the entity scene; then, the scan data is input into a three-dimensional scene generation device 2000, and the three-dimensional scene generation device 2000 performs three-dimensional reconstruction on the entity scene based on the scan data of the entity scene to generate a virtual scene corresponding to the entity scene. This virtual scene can be instantiated and applied by an instantiation device 3000, for example. For example, the generated virtual scene can be used for virtual shooting, animation production, game production, etc.

[0048] As Figure 1 shown, scene reconstruction and virtual scene instantiation can be respectively implemented by the three-dimensional scene generation device 2000 and the instantiation device 3000. Among them, the three-dimensional scene generation device 2000 can be any electronic device with the computing power to support the reconstruction algorithm. Such an electronic device is, for example, a PC, a laptop, a workstation, a server, etc.; the instantiation device 3000 can be any electronic device that can meet the application requirements. Such an electronic device is, for example, a PC, a laptop, a virtual reality device, etc. Those skilled in the art can understand that scene reconstruction and virtual scene instantiation can also be implemented by the same device, which is not limited herein.

[0049] The above-mentioned instantiation applications such as virtual shooting have relatively high precision requirements for the virtual scenes used. However, in the related art, the precision of scene reconstruction is affected by many factors, including the reconstruction algorithm, the scanning integrity of the entity scene, the texture quality, the categories of entity objects in the scene, etc. For example, local missing of the scan data will result in lower reconstruction precision; for another example, the reconstruction algorithm cannot handle vegetation objects, and in the case where the entity scene contains vegetation objects, it will also result in lower reconstruction precision, etc. Therefore, the three-dimensional virtual scene reconstructed based on the related art cannot meet the requirements of applications such as virtual shooting, animation production, and game production. It is necessary to perform a large number of repair operations on the generated virtual scene manually in the later stage and then perform instantiation application, which increases the production cost and production cycle.

[0050] To solve the technical problem of low scene reconstruction accuracy, the embodiments of the present disclosure provide a new technical solution for scene reconstruction. By using the model components matched for the entity objects in the entity scene from the asset library, a three-dimensional model of the entity object is built in the virtual space, thereby generating a virtual scene corresponding to the entity scene. In the embodiments of the present disclosure, since the model components in the asset library have high accuracy, and by building the three-dimensional model of the entity object through matching the model components, various factors affecting the reconstruction accuracy can also be shielded. Therefore, the solution of the embodiments of the present disclosure can effectively improve the accuracy of the generated virtual scene, thereby meeting the requirements of applications such as virtual shooting, animation production, and game production.

[0051] Figure 2 FIG. shows a schematic flowchart of a three-dimensional scene generation method according to some embodiments of the present disclosure. This method can be implemented, for example, by Figure 1 the three-dimensional scene generation device 2000 shown in FIG. As Figure 2 shown, this method includes steps S210 to S230:

[0052] Step S210, according to the scan data of the entity scene, obtain the planar structure diagram of the entity scene and the feature description data of each entity object in the entity scene.

[0053] The entity scene in this embodiment is also a real scene that actually exists in the real world, which can be any real scene that needs to be restored through scene reconstruction. For example, it can be a school, a commercial area, a residential area, or a scenic area, etc., which is not limited herein.

[0054] In this embodiment, the scan data of the entity scene can be a dataset of spatial points obtained by lidar scanning the entity scene, or an image set obtained by camera scanning the entity scene, and can also include the above dataset of spatial points and image set, etc. In the case where the scan data includes an image set, more feature information of the entity scene can be obtained according to this scan data, which is beneficial to perform higher-precision scene reconstruction. For example, in the case where the scan data includes an image set, in addition to feature information such as category, position, size, and orientation, other feature information such as color and texture can also be obtained through image recognition.

[0055] In this embodiment, the planar structure diagram of the entity scene represents the planar structure of the entity scene based on the two-dimensional structure of the entity object. The planar structure diagram includes the planar structure diagram of the entity scene with respect to the ground plane. The two-dimensional structure here can represent the two-dimensional main structure of the corresponding entity object through lines, contour graphics, etc. That is, the entity object can be mapped onto the ground plane to obtain corresponding two-dimensional structures such as lines and contour graphics, and these two-dimensional structures represent the two-dimensional main structure of the entity object. Based on these two-dimensional structures, the planar structure of the entity scene is represented. For example, in the planar structure diagram of the entity scene, the center line of the road can be used as the two-dimensional structure of the road to calibrate the road; for another example, in the planar structure diagram, the contour graphic of the building foundation can be used as the two-dimensional structure of the building to calibrate the building; for yet another example, in the planar structure diagram, the contour graphic of the water area can be used as the two-dimensional structure of the water area to calibrate the water area, and so on.

[0056] The planar structure diagram of the entity scene can perform an equal-proportion mapping of the entity scene, and is used to describe the main shapes, dimensions of each entity object, and the relative positions of each entity object in the entity scene, etc.

[0057] The feature description data of the entity object can include at least one of the attribute description data of the entity object, the dimension description data of the entity object, and the direction description data of the entity object. Based on the scan data of the entity scene, various features of the entity object are extracted to obtain relatively rich feature description data of the entity object, which is beneficial to improving the efficiency and accuracy of model component matching and model construction.

[0058] The attribute description data is the description data about the own attributes of the entity object. The attribute description data can at least include the description about the category of the entity object. For example, the category of an entity object is a building, a road, a tree, a flower, a water area, a wall, a corridor or a stone, etc. For entity objects such as trees and flowers, their categories can further include subdivisions such as families, genera or species, which are not limited here.

[0059] According to the category of the entity object, its attribute description data can further include at least one of the descriptions about style, color, surface texture, material, etc.; moreover, it can also describe each component part of the entity object separately according to the modeling needs, which is not limited here. For example, for entity objects such as buildings, their attribute description data can also include style, color, material, number of floors, etc. For another example, for entity objects such as water areas, their attribute description data can also include color, etc.

[0060] The size description data may include the sizes of an entity object at different parts and / or in different directions. For an entity object, the size description data required for scene reconstruction based on the method of this embodiment may be defined according to the category of the entity object. For example, for an entity object of a building, its size description data may include the length, width, height, etc. of the building, and may further include the floor height of subdivisions, etc. For another example, for an entity object of a tree, its size description data may include the outer diameter of the tree trunk, the height of the tree trunk, the height of the tree crown, the outer diameters of the tree crown on multiple horizontal planes, etc. For yet another example, for an entity object of a road, its size description data may include the length of the road, the width of the road, etc.

[0061] The direction description data may include the orientation or the extending direction of an entity object. For an irregular shape, the direction description data may further include the corner orientations of the entity object, etc. For an entity object, the direction description data required for scene reconstruction by implementing the method of this embodiment may be defined according to at least one of the category of the entity object and the main shape of the entity object in the planar structure diagram. For example, the direction description data of a building describes its orientation as facing south with north behind. For another example, the direction description data of a road describes its extending direction as east-west. For yet another example, the direction description data of a river includes its extending direction, and the corner orientations of each corner of the river, etc., which are not specifically defined herein.

[0062] As Figure 3 described above, taking N entity objects scanned into the entity scene as an example, where N is an integer greater than or equal to 1, in step S210, based on the scan data of the entity scene, a planar structure diagram of the entity scene, and the feature description data of entity object 1, the feature description data of entity object 2, up to the feature description data of entity object N can be obtained. Here, as described above, according to the category to which the entity object belongs, the feature description data of different entity objects may involve the same features or may involve different features, which are not limited herein.

[0063] In this embodiment, the two-dimensional structure body is associated with the feature description data based on the corresponding entity object.

[0064] In order to quickly and accurately obtain the planar structure diagram of the entity scene and the feature description data of each entity object in the entity scene, point cloud reconstruction can be performed on the entity scene based on the scan data of the entity scene, and based on the three-dimensional point cloud of the entity scene, the planar diagram of the entity scene and the feature description data of each entity object can be obtained. Correspondingly, in some embodiments, the step S210 of obtaining the planar structure diagram of the entity scene and the feature description data of each entity object in the entity scene according to the scan data of the entity scene may include the following steps S2101 to step S2103:

[0065] Step S2101: Perform point cloud reconstruction on the entity scene according to the scan data of the entity scene to obtain the three-dimensional point cloud of the entity scene.

[0066] The three-dimensional point cloud of the entity scene can be understood as a set of spatial points composed of a large number of sampling points on the surface of the entity scene. By scanning the entity scene with at least one scanning device such as a camera or a lidar, corresponding scan data can be obtained. Based on such scan data, point cloud reconstruction can be performed on the entity scene, and then the three-dimensional point cloud describing the entity scene can be obtained. In the three-dimensional point cloud, each spatial point is represented by an array, and the array includes at least the three-dimensional coordinates of the corresponding sampling point. The array can further include at least one other dimension of data such as the reflectivity, intensity value, and color of the corresponding sampling point.

[0067] In some examples, the scan data of the entity scene can include multiple frames of images collected by aerial photographing the entity scene. Here, the images are depth images with depth information, as well as the camera parameter data corresponding to each image.

[0068] The camera parameter data includes the internal parameter values of the camera internal parameters and the external parameter values of the camera external parameters. The function of the camera internal parameters is to determine the projection relationship of the camera from the three-dimensional space to the two-dimensional image. There are a total of 6 parameters (f, κ, Sx, Sy, Cx, Cy). Among them, f is the focal length; κ represents the radial distortion amount; Sx and Sy are the scaling factors; Cx and Cy are the principal points of the image, that is, the intersection points of the plane passing through the lens axis perpendicular to the imaging plane and the image plane. The function of the camera external parameters is to determine the relative position relationship between the camera coordinate system and the world coordinate system. There are a total of 6 parameters (α, β, γ, Tx, Ty, Tz). Among them, T=(Tx, Ty, Tz) is the translation vector, and R = R(α, β, γ) is the rotation matrix. The camera internal parameters generally do not change with the change of the camera pose (including the camera position and the camera attitude), but the camera external parameters will change dynamically with the change of the camera pose. According to the camera parameter data, the three-dimensional coordinates of the sampling points of the entity scene in the image can be determined, and it can also be used to determine the direction characteristics of the entity object to obtain its direction description data, etc.

[0069] Since the camera external parameters will change dynamically with the change of the camera pose, and the image set is dynamically collected by the flying device carrying the camera during movement. At different camera positions, the camera pose will change. Therefore, the external parameter values corresponding to each scene image in the image set may be different. This requires determining the corresponding external parameter values for each scene image in the image set. When determining the external parameter values of a scene image collected by the camera, it can be calculated based on the corresponding scene image, or it can be determined based on the pose data provided by the inertial sensor (Inertial Measurement Unit, IMU) carried by the flying device at the corresponding image acquisition moment. This is not limited here.

[0070] Step S2102: Segment the 3D point cloud of the entity scene into multiple single-body point clouds, where one single-body point cloud corresponds to one entity object in the entity scene.

[0071] In this step S2102, it is necessary to segment the 3D point cloud to divide the 3D point cloud into multiple single-body point clouds that correspond one-to-one to multiple entity objects in the entity scene, so as to be able to further extract the features of the corresponding entity objects based on the single-body point clouds.

[0072] In this step S2102, the 3D point cloud can be segmented based on semantic segmentation algorithms, edge detection-based segmentation algorithms, region-based segmentation algorithms (such as seed region algorithms, non-seed region algorithms, etc.), attribute-based clustering segmentation algorithms (such as Euclidean clustering, density clustering), geometric shape model-based segmentation algorithms (such as Random Sample Consensus), graph cut-based segmentation algorithms (such as Kruskal algorithm), etc. to obtain multiple single-body point clouds.

[0073] Step S2103: Obtain the feature description data of the corresponding entity object and the plane structure diagram of the entity scene according to each single-body point cloud.

[0074] After the 3D point cloud is segmented into each single-body point cloud, for each single-body point cloud, the category of the corresponding entity object can be identified to divide each single-body point cloud into categories such as buildings, roads, trees, flowers and plants, gates, walls, corridors, waters, etc. For each single-body point cloud, features such as its size, shape, color, and direction can also be extracted.

[0075] In some examples, the point cloud data of the single-body point cloud can be input into a feature extraction model to obtain the feature data of the single-body point cloud. For example, the category of the entity object corresponding to the single-body point cloud can be obtained, and other features such as the shape, size, and style of the corresponding entity object can also be obtained, which are not limited here. The feature extraction model can be trained with labeled point cloud samples.

[0076] In other examples, based on the array of spatial points in the single-body point cloud, features such as the size, shape, and color of the corresponding entity object can also be obtained.

[0077] In still other examples, based on the camera parameter data corresponding to the image, the direction feature of the entity object corresponding to the single-body point cloud can be obtained, etc.

[0078] In this way, according to the features of each single-body point cloud, the feature description data of the corresponding entity object can be obtained, and the plane structure diagram of the entity scene can be obtained.

[0079] When generating the planar structure diagram of the entity scene, based on the category of the entity object corresponding to the single-point cloud, two-dimensional structures such as the skeleton and contour of the entity object can be extracted from the single-point cloud to map the corresponding entity object. For example, the skeletons of entity objects such as roads and walls are extracted. Another example is to extract the foundation contour of entity objects such as buildings.

[0080] Step S220: According to the two-dimensional structure of the entity object and the feature description data of the entity object, match at least one model component for the entity object in the asset library.

[0081] In this embodiment, the asset library has model components with high precision, and these model components can be used to build the three-dimensional model of the entity object. The asset library can include model components that reflect the complete structure of the object, or can also include model components that reflect the local structure of the object. For example, the building assets include model components of various types of floors, rooftops, etc. Another example is that the road assets include model components of various road sections. Still another example is that the tree assets include tree models of different species and forms.

[0082] The asset library can include at least one of the following types of model components, that is, manually created model components, model components repaired after three-dimensional reconstruction, model components obtained from the virtual scene generated by the method of this embodiment, etc. These model components all have high precision.

[0083] In this embodiment, for each identified entity object, step S220 is executed to match at least one model component for building the entity object in the asset library according to the skeleton features or contour features indicated by the two-dimensional structure of the entity object, and according to the features such as the category, size, direction, color, etc. indicated by the feature description data of the entity object. For example, matching can be performed based on feature similarity to match the model component with the highest similarity or the similarity reaching the set threshold.

[0084] It can be understood that for entity objects with repeated structural units, the matched model components can include multiple identical model components. For example, for entity objects such as buildings, the structure of each floor is basically the same. Therefore, according to the number of floors, multiple floor components can be matched to build the three-dimensional model.

[0085] Still as Figure 3 shown, for the N identified entity objects, through step S220, the model components for matching entity object 1, the model components for matching entity object 2, and so on until the model components for matching entity object N can be obtained.

[0086] In some embodiments, to improve the efficiency of model matching, matching at least one model component for the entity object in the asset library in step S220 may include: matching at least one model component for the entity object in the asset group to which the entity object belongs in the asset library, where the asset library includes multiple asset groups divided according to object categories.

[0087] In these embodiments, referring to Figure 4 As shown, the asset library may include, for example, a building asset group, a natural ecological asset group, a road asset group, a building annex asset group, etc. Among them, natural ecology includes objects such as flowers, plants, trees, stones, water areas, etc. Building annexes may include objects such as walls, corridors, etc.

[0088] In these embodiments, the asset library may also include more refined asset groups. For example, the building asset group also includes a floor asset group and a roof asset group. Also, for example, the natural ecological asset group may further include a tree asset group, a water area asset group, a stone asset group, a flower and plant asset group, and so on.

[0089] In these embodiments, the category of the entity object may be determined according to the feature description data of the entity object, and the asset group to which the entity object belongs may be determined according to the category of the entity object. Then, in this asset group, a corresponding model component may be matched for it to reduce the matching calculation amount.

[0090] In some embodiments, multiple threads may also be created according to the classification of the asset groups, with one thread corresponding to one asset group. When matching model components, multiple threads perform the matching of model components for the corresponding entity objects in parallel to further improve the matching efficiency. For example, the first thread performs the matching of model components for entity objects of the building category, the second thread performs the matching of model components for entity objects of the natural ecological category, the third thread performs the matching of model components for entity objects of the road category, and the fourth thread performs the matching of model components for entity objects of the building annex category, etc.

[0091] In some embodiments, to improve the applicable range of the model components in the asset library, improve the matching speed, and at the same time reduce the pressure of preparing high-precision model components for the asset library, at least some parameter values of the model components in the asset library may be set to be adjustable. For example, the color, size, etc. of the model components may be adjusted so that the same model component can adapt to more entity objects through parameter adjustment. Correspondingly, step S220 of matching at least one model component for the entity object in the asset library according to the two-dimensional structure body of the entity object and the feature description data of the entity object may further include the following step S2201 and step S2202:

[0092] Step S2201: According to the two-dimensional structure of the entity object and the feature description data of the entity object, match the initial model components in the asset library whose feature similarity meets the matching conditions for the entity object.

[0093] The matching conditions can be determined according to the adjustable parameters of the model components, that is, this matching allows for differences in the adjustable parameters between the model components and the entity object.

[0094] Step S2202: According to the feature description data of the entity object, adjust the parameter values of the initial model components to obtain model components that match the entity object.

[0095] After matching the initial model components for an entity object, the corresponding parameter values of the initial model components can be adjusted according to the data such as size and color in the feature description data of the entity object, so as to obtain model components that match the entity object.

[0096] Step S230: Through the model components matched for the entity object, build the three-dimensional model of the entity object at the positions calibrated by the corresponding two-dimensional structure, and generate a virtual scene corresponding to the entity scene.

[0097] In this embodiment, for each recognized entity object, in step S230, as Figure 3 shown, through the matched model components, build the three-dimensional model of the entity object at the positions calibrated by the corresponding two-dimensional structure in the planar structure diagram. After completing the construction of the three-dimensional models of all entity objects, a three-dimensional virtual scene corresponding to the entity scene is generated.

[0098] Taking an entity object such as a building as an example, for a building object, the three-dimensional model of the building object can be built at the positions calibrated based on the planar structure diagram through the matched floor components and roof components.

[0099] In this embodiment, the model component matching operation in step S220 and the model building operation in step S230 can be carried out in parallel. After matching the corresponding model components for an entity object, the three-dimensional model of the entity object can be built, without having to wait until all entity object model component matching is completed before performing the model building operation. Similar to the matching of model components, the building of the three-dimensional model can also be carried out by the corresponding threads. For example, the first thread is responsible for the model component matching and three-dimensional model building of building entity objects, which will not be elaborated here.

[0100] According to the above steps S210 to S230, for the 3D model generation method according to this embodiment, since the model components in the asset library have high precision, and by matching the model components to build the 3D model of the entity object, it will be able to effectively shield the influence of factors such as scanning integrity, texture quality, and the category of entity objects in the scene on the reconstruction accuracy. Therefore, the generated virtual scene can meet the requirements of applications such as virtual shooting, animation production, and game production, and can greatly reduce the workload of post-adjustment, and even can be directly used without post-adjustment.

[0101] In some other embodiments, in order to further improve the accuracy of model component matching and 3D model building, after obtaining the planar structure diagram of the entity scene in the above step S210 and before matching at least one model component for the entity object in the asset library in step S220, the method may further include a step of regularizing the two-dimensional structure. Correspondingly, in these embodiments, the method may include the following steps:

[0102] Step S310, according to the scan data of the entity scene, obtain the planar structure diagram of the entity scene and the feature description data of each entity object in the entity scene.

[0103] Step S311, according to the initial shape of the two-dimensional structure in the planar structure diagram, determine a regularized shape adapted to the initial shape.

[0104] In this embodiment, the initial shape of the two-dimensional structure is, for example, a shape directly recognized based on the single-point cloud. The initial shape is affected by the scanning accuracy and the point cloud reconstruction accuracy, and there may be some errors relative to the actual shape of the entity object. By regularizing the initial shape, the errors can be reduced or eliminated, and it is also beneficial to improve the matching speed of the model components.

[0105] For various entity objects, various common regularized shapes can be set, and based on the similarity between the initial shape and the regularized shape, a regularized shape adapted to the initial shape can be determined.

[0106] Taking building entity objects as an example, the contour shapes of building foundations are mainly square, rectangular, L-shaped, T-shaped and other specifications. Therefore, for building entity objects, these specifications of regularized shapes can be set.

[0107] Step S312, regularize the corresponding two-dimensional structure according to the regularized shape to update the two-dimensional structure.

[0108] Still taking a building - type entity object as an example, if the initial shape of the contour graph of its two - dimensional structure is close to an L - shape, then the two - dimensional structure is regularized according to the regularized shape of the L - shape to update its two - dimensional structure, and further update the floor plan of the entity scene. Among them, the regularization process may include correcting the initial shape to a suitable regularized shape. For example, if the initial shape of a building - type entity object is close to a rectangle, the initial shape can be corrected to its circumscribed rectangle, that is, a rectangle that just covers the initial shape.

[0109] Step S320: According to the two - dimensional structure of the entity object and the feature description data of the entity object, match at least one model component for the entity object in the asset library.

[0110] Step S330: Through the model components matched for the entity object, build the three - dimensional model of the entity object at the position marked by the corresponding two - dimensional structure to generate a virtual scene corresponding to the entity scene.

[0111] The three - dimensional virtual scene generated by the method based on the embodiments of the present disclosure, as Figure 1 shown, can be used for virtual shooting, animation production, game production, etc. Taking virtual shooting as an example, the virtual shooting method may include the following steps S410 to S430:

[0112] Step S410: According to the scan data of the entity scene, obtain the floor plan of the entity scene and the feature description data of each entity object in the entity scene.

[0113] Step S420: According to the two - dimensional structure of the entity object and the feature description data of the entity object, match at least one model component for the entity object in the asset library.

[0114] Step S430: Through the model components matched for the entity object, build the three - dimensional model of the entity object at the position marked by the corresponding two - dimensional structure to generate a virtual scene corresponding to the entity scene.

[0115] Step S440: Shoot a work based on the generated virtual scene.

[0116] In some examples, a virtual camera can be created in the virtual scene, and parameters such as the position and angle of the virtual camera can be set to complete the shooting of the work in the generated virtual scene through the virtual camera.

[0117] In other examples, it is also possible to shoot a work by displaying the generated virtual scene on a large screen, etc., which will not be elaborated here.

[0118] The present disclosure also provides a three - dimensional scene generation device for implementing any of the above - mentioned method embodiments. Figure 5 The structural block diagram of the three - dimensional scene generation device according to some embodiments is shown. AsFigure 5 As shown in Figure 5 , the three-dimensional scene generation device 500 may include a data processing module 510, a component matching module 520, and a scene generation module 530.

[0119] The data processing module 510 is configured to obtain a planar structure diagram of the physical scene and feature description data of each physical object in the physical scene according to the scan data of the physical scene, wherein the planar structure diagram represents the planar structure of the physical scene based on the two-dimensional structure of the physical object.

[0120] The component matching module 520 is configured to match at least one model component for each identified physical object in the asset library according to the two-dimensional structure of the physical object and the feature description data of the physical object.

[0121] The scene generation module 530 is configured to build a three-dimensional model of each identified physical object at the position calibrated by the corresponding two-dimensional structure through the model components matched for the physical object, and generate a virtual scene of the corresponding physical scene.

[0122] In some embodiments, when the component matching module 520 matches at least one model component for a physical object in the asset library, it may be configured to: for the physical object to be matched, match at least one model component for the physical object in the asset group to which the physical object belongs in the asset library, wherein the asset library includes a plurality of asset groups divided according to object categories.

[0123] In some embodiments, after obtaining the planar structure diagram of the physical scene, the data processing module 510 may further be configured to: determine a regularized shape adapted to the initial shape according to the initial shape of the two-dimensional structure in the planar structure diagram; and perform regularization processing on the corresponding two-dimensional structure according to the regularized shape, so as to provide an updated planar structure diagram to the component matching module 520.

[0124] In some embodiments, when the component matching module 520 matches at least one model component for a physical object in the asset library according to the two-dimensional structure of the physical object and the feature description data of the physical object, it may be configured to: match an initial model component with a feature similarity meeting the matching condition for the physical object in the asset library according to the two-dimensional structure of the physical object and the feature description data of the physical object; and adjust the parameter values of the initial model component according to the feature description data of the physical object to obtain a model component matching the physical object.

[0125] In some embodiments, when the data processing module 510 obtains the planar structure diagram of the entity scene and the feature description data of each entity object in the entity scene based on the scan data of the entity scene, it can be used to: perform point cloud reconstruction on the entity scene based on the scan data of the entity scene to obtain the three-dimensional point cloud of the entity scene; segment the three-dimensional point cloud of the entity scene into multiple single-body point clouds, where one single-body point cloud corresponds to one entity object in the entity scene; and obtain the feature description data of the corresponding entity object and the planar structure diagram of the entity scene according to each single-body point cloud.

[0126] The present disclosure also provides an electronic device for implementing any of the above method embodiments. Figure 6 The structural block diagram of an electronic device 600 according to some embodiments is shown. The electronic device 600 may be a PC, a workstation, a laptop computer, a server, etc., which is not limited herein.

[0127] As Figure 6 shown, the electronic device 600 includes a processor 610 and a memory 620 for storing executable instructions of the processor 610. The processor 610 is configured to implement the three-dimensional scene generation method according to any embodiment of the present disclosure when executing the instructions stored in the memory 620.

[0128] The processor 610 is used to execute computer instructions, and the computer instructions can be written in instruction sets such as x86, Arm, RISC, MIPS, SSE, etc. The memory 620 includes, for example, ROM (read-only memory), RAM (random access memory), non-volatile memory such as a hard disk, etc., which is not limited herein.

[0129] Figure 7 The structural block diagram of an electronic device according to other embodiments is shown. As Figure 7 shown, in addition to the processor 710 and the memory 720, the electronic device 700 may further include a display device 730, an interface device 740, a communication device 750, an input device 760, etc.

[0130] The interface device 740 includes, for example, a USB interface, a bus interface, a network interface, etc. The communication device 750 can perform wired or wireless communication, for example. The communication device 750 may include at least one short-range communication module, for example, any module that performs short-range wireless communication based on short-range wireless communication protocols such as the Hilink protocol, WiFi (IEEE 802.11 protocol), Mesh, Bluetooth, ZigBee, Thread, Z-Wave, NFC, UWB, LiFi, etc. The communication device 750 may also include a remote communication module, for example, any module that performs WLAN, GPRS, 2G / 3G / 4G / 5G remote communication. The display device 730 can display the operation screen generated by the scene. The input device 760 may include a touch screen, a keyboard, a mouse, a microphone, a camera, etc., which are not limited herein.

[0131] The present invention may be a system, a method, and / or a computer program product. The computer program product may include a computer-readable storage medium having thereon computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0132] A computer-readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanically encoded device, such as a punched card or raised structures in grooves having instructions stored thereon, and any suitable combination of the foregoing. The computer-readable storage medium used herein is not construed as being an instantaneous signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., an optical pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0133] The computer-readable program instructions described herein can be downloaded to various computing / processing devices from a computer-readable storage medium or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0134] The computer program instructions for carrying out operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state-setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages - such as Smalltalk, C++, etc., and conventional procedural programming languages, such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network - including a local area network (LAN) or a wide area network (WAN) - or, alternatively, may be connected to an external computer (e.g., via an Internet service provider through the Internet). In some embodiments, by using the state information of the computer-readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer-readable program instructions to implement various aspects of the present invention.

[0135] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0136] These computer-readable program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine such that the instructions, when executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more boxes of the flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises a manufacture, the instructions being for implementing the various aspects of the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0137] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, such that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0138] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, a segment of a program, or a portion of an instruction, the module, segment of a program, or portion of an instruction containing one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the boxes may occur out of the order noted in the figures. For example, two consecutive boxes may in fact be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each box of the block diagrams and / or flowcharts, and combinations of boxes in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or acts, or by a combination of dedicated hardware and computer instructions. As is well known to those skilled in the art, implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0139] The embodiments of the present invention have been described above. The above description is exemplary and not exhaustive, and is also not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technological improvements in the market, or to enable other ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A three-dimensional scene generation method, characterized in that, Including: Based on the scan data of the entity scene, obtain the planar structure diagram of the entity scene and the feature description data of each entity object in the entity scene. Among them, the scan data includes a data set and / or an image set of spatial points. The planar structure diagram represents the planar structure of the entity scene based on the two-dimensional structure of the entity object. The planar structure diagram includes a planar structure diagram of the ground plane obtained by mapping the entity scene to the ground plane. The planar structure diagram performs an equal-scale mapping of the entity scene and is used to describe the main shape, size of each entity object, and the relative position of each entity object in the entity scene. The feature description data of the entity object includes attribute description data, size description data, and direction description data. The size description data includes the sizes of the entity object in different parts and / or different directions. The direction description data includes the orientation or extension direction of the entity object. Based on the two-dimensional structure of the entity object and the feature description data of the entity object, match at least one model component for the entity object in the asset library. Through the model components matched for the entity object, build a three-dimensional model of the entity object at the position calibrated by the corresponding two-dimensional structure, and generate a virtual scene corresponding to the entity scene.

2. The method according to claim 1, characterized in that, The step of matching at least one model component for the entity object in the asset library includes: In the asset group to which the entity object belongs in the asset library, match at least one model component for the entity object. Among them, the asset library includes multiple asset groups divided according to object categories.

3. The method according to claim 1, characterized in that, After obtaining the planar structure diagram of the entity scene and before matching at least one model component for the entity object in the asset library, the method further includes: According to the initial shape of the two-dimensional structure in the planar structure diagram, determine a regularized shape adapted to the initial shape. Perform regularized processing on the corresponding two-dimensional structure according to the regularized shape to update the corresponding two-dimensional structure.

4. The method according to claim 1, characterized in that, The step of matching at least one model component for the entity object in the asset library based on the two-dimensional structure of the entity object and the feature description data of the entity object includes: Based on the two-dimensional structure of the entity object and the feature description data of the entity object, match an initial model component in the asset library whose feature similarity meets the matching condition. According to the feature description data of the entity object, adjust the parameter values of the initial model component to obtain a model component matching the entity object.

5. The method according to claim 1, wherein Based on the scan data of the entity scene, obtaining the planar structure diagram of the entity scene and the feature description data of each entity object in the entity scene includes: Perform point cloud reconstruction on the entity scene according to the scan data of the entity scene to obtain the three-dimensional point cloud of the entity scene. Divide the three-dimensional point cloud of the entity scene into multiple single-point clouds, where one single-point cloud corresponds to one entity object in the entity scene. According to each single-point cloud obtained by segmentation, obtain the feature description data of the corresponding entity object and the planar structure diagram of the entity scene.

6. The method according to any one of claims 1 to 5, characterized in that, The scanned data includes multiple frames of images collected by aerial photographing the entity scene, and camera parameter data corresponding to each image.

7. A three-dimensional scene generation device, characterized in that, Comprising: A data processing module, configured to obtain a planar structure diagram of the entity scene and feature description data of each entity object in the entity scene according to the scanned data of the entity scene, wherein the scanned data includes a data set and / or an image set of spatial points, the planar structure diagram represents the planar structure of the entity scene based on a two-dimensional structure of the entity object, the planar structure diagram includes a planar structure diagram of the ground plane obtained by mapping the entity scene to the ground plane, the planar structure diagram performs an equal-scale mapping on the entity scene, and is used to describe the main shapes, sizes of each entity object, and the relative positions of each entity object in the entity scene, the feature description data of the entity object includes attribute description data, size description data, and direction description data, the size description data includes the sizes of the entity object in different parts and / or different directions, and the direction description data includes the orientation or extension direction of the entity object; A component matching module, configured to match at least one model component for the entity object in the asset library according to the two-dimensional structure of the entity object and the feature description data of the entity object; A scene generation module, configured to build a three-dimensional model of the entity object at the position calibrated by the corresponding two-dimensional structure through the model components matched for the entity object, and generate a virtual scene corresponding to the entity scene.

8. An electronic device, characterized in that, Comprising: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to implement the method according to any one of claims 1 to 6 when executing the instructions stored in the memory.

9. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that, The computer program instructions, when executed by the processor, implement the method according to any one of claims 1 to 6.

Images