Three-dimensional scene reconstruction method and apparatus, and computing device cluster

Abstract

A three-dimensional scene reconstruction method comprises the steps that a set of sequence images obtained by shooting a first scene from different visual angles or angles are obtained, and the first scene can provide visual information needed for constructing a three-dimensional model; on the basis of an image containing a moving entity in the sequence image, points related to the moving entity in a first three-dimensional point cloud are removed, a second three-dimensional point cloud is obtained, and the first three-dimensional point cloud is generated on the basis of the sequence image; generating a Mesh model based on the second three-dimensional point cloud; and based on the sequence image, performing texture mapping on the Mesh model to obtain a three-dimensional model related to the first scene. Therefore, in the three-dimensional scene construction process, since the points associated with the mobile entity in the three-dimensional point cloud are removed before the Mesh model is constructed, the geometric removal of the mobile entity can be realized, high-quality input is provided for the subsequent construction of the Mesh model, and a correct shielding relationship is provided for the subsequent texture mapping; therefore, the quality of the constructed three-dimensional model can be improved.

Description

Technical Field

[0001] The present application relates to the field of information technology (IT), and in particular to a three-dimensional scene reconstruction method, apparatus, and computing device cluster. Background Art

[0002] Realistic 3D is a digital space that provides a realistic, three-dimensional, and temporal representation of human production, living, and ecological spaces within a specific area. It is based on 3D imaging technology, enabling the perception, mapping, modeling, and management of the real world. It is integrated with information technologies such as the Internet and the Internet of Things to create a three-dimensional digital space. In recent years, the maturity of drone technology and advancements in modeling algorithms have significantly reduced the cost of acquiring image data. Consequently, city-level 3D scene reconstruction based on multi-view oblique imagery has become increasingly common.

[0003] Currently, accurately modeling moving objects in urban 3D scenes is difficult. This is especially true for vehicles, which are essential transportation tools and can be found everywhere. Most vehicles are constantly in motion, causing geometric and texture distortion in 3D models. This results in a cluttered and disorganized 3D reconstruction, impacting not only the model's visual quality but also various applications based on real-world 3D models. Therefore, minimizing the impact of moving objects on 3D models during the modeling process is a pressing technical challenge. Summary of the Invention

[0004] The present application provides a three-dimensional scene reconstruction method, apparatus, computing device cluster, computer storage medium and computer product, which can reduce the impact of moving objects on the three-dimensional model during the modeling process.

[0005] In a first aspect, the present application provides a three-dimensional scene reconstruction method, comprising: obtaining a set of sequential images obtained by photographing a first scene from different viewpoints or angles, wherein the first scene is used to provide visual information required to construct a three-dimensional model; based on images of mobile entities contained in the sequential images, removing points related to the mobile entities in the first three-dimensional point cloud to obtain a second three-dimensional point cloud, wherein the first three-dimensional point cloud is generated based on the sequential images; generating a mesh model based on the second three-dimensional point cloud; and performing texture mapping on the mesh model based on the sequential images to obtain a three-dimensional model related to the first scene. Exemplarily, the mobile entity refers to an object moving in space, such as pedestrians, cars, motorcycles, bicycles, electric vehicles, animals, etc.

[0006] In this way, during the three-dimensional scene construction process, since the points associated with the mobile entities in the three-dimensional point cloud have been removed before the Mesh model is constructed, the geometric removal of the mobile entities can be achieved, providing high-quality input for the subsequent construction of the Mesh model, and providing the correct occlusion relationship for the subsequent texture mapping, thereby improving the quality of the constructed three-dimensional model.

[0007] In one possible implementation, based on images of a mobile entity included in a sequence of images, points associated with the mobile entity in a first 3D point cloud are removed to obtain a second 3D point cloud, including: marking points in the first 3D point cloud as entity points or non-entity points based on the number of images of the first entity included in the visible images of the points, wherein entity points are points in the first 3D point cloud associated with the first entity, and the first entity is associated with the mobile entity; obtaining a first grid based on the points marked as non-entity points in the first 3D point cloud; identifying mobile entity points among the points marked as entity points based on projection results of the points marked as entity points in the first 3D point cloud onto the first grid, wherein mobile entity points are points associated with the mobile entity among the points marked as entity points; and removing the mobile entity points from the first 3D point cloud to obtain a second 3D point cloud. In this way, points associated with the mobile entity in the 3D point cloud are removed through grid projection.

[0008] In one possible implementation, in a first grid, if the number of existing projected points in the grid containing the projection point of the first point is greater than a first threshold, the first point is considered a moving entity point. The first point is any point marked as an entity point in the first three-dimensional point cloud. If the number of existing projected points in the grid containing the projection point of the first point is less than or equal to the first threshold, the first point is considered a stationary entity point. In this way, points marked as entity points can be further classified based on the number of existing projected points in the grid. For example, the first threshold can be, but is not limited to, 0.

[0009] In one possible implementation, texture mapping is performed on a mesh model based on a sequence of images, including: identifying solid faces in the mesh model based on the number of images of a first entity contained in the visible images of the mesh model triangular faces, wherein the solid faces are faces in the mesh model triangular faces associated with the first entity; identifying moving solid faces in the mesh model triangular faces based on the number of visible images of the mesh model triangular faces and the number of images of the first entity contained in the visible images of the mesh model triangular faces, wherein the moving solid faces are faces in the mesh model triangular faces associated with the moving entity; removing the images of the first entity contained in the visible images of the moving solid faces, and, based on the removal results, performing texture mapping on the mesh model. Thus, by classifying the triangular faces and filtering the visible images corresponding to the triangular faces classified as moving solid faces, no moving objects can be observed in the visible image set of any triangular face during the texture mapping process. This achieves the purpose of accurately removing moving objects, thereby avoiding the impact of moving objects on texture mapping, avoiding post-processing or manual modeling, improving model reconstruction efficiency, and reducing costs.

[0010] In one possible implementation, if the number of images of the first entity contained in the visible image of the triangular face is greater than a second threshold, the triangular face is considered a solid face; if the number of images of the first entity contained in the visible image of the triangular face is less than or equal to the second threshold, the triangular face is considered a non-solid face. This achieves classification of triangular faces.

[0011] In one possible implementation, if the first number equals the second number, the triangular face is a stationary entity face. The first number represents the number of visible images of the triangular face, while the second number represents the number of visible images of the triangular face that contain the first entity. If the first number exceeds the second number, the triangular face is a moving entity face. This allows for further classification of triangular faces.

[0012] In a possible implementation, the mobile entity includes one or more of the following: a pedestrian, a car, a motorcycle, a bicycle, an electric vehicle, or an animal.

[0013] In a possible implementation, the three-dimensional model is a three-dimensional map.

[0014] In a possible implementation, the first scene is a city scene.

[0015] In a second aspect, the present application provides a three-dimensional scene reconstruction device, comprising: an acquisition module and a processing module. The acquisition module is used to acquire a set of sequential images obtained by shooting a first scene from different viewing angles or angles, wherein the first scene is used to provide the visual information required to construct a three-dimensional model. The processing module is used to remove the points related to the mobile entity in the first three-dimensional point cloud based on the image of the mobile entity contained in the sequential images to obtain a second three-dimensional point cloud, wherein the first three-dimensional point cloud is generated based on the sequential images. The processing module is also used to generate a Mesh model based on the second three-dimensional point cloud. The processing module is also used to perform texture mapping on the Mesh model based on the sequential images to obtain a three-dimensional model related to the first scene.

[0016] In one possible implementation, when the processing module removes points related to the mobile entity in the first three-dimensional point cloud based on the images of the sequence containing the mobile entity to obtain the second three-dimensional point cloud, the processing module is specifically used to: mark the points in the first three-dimensional point cloud as entity points or non-entity points based on the number of images of the first entity included in the visible images of the points in the first three-dimensional point cloud, wherein the entity points are points related to the first entity in the first three-dimensional point cloud, and the first entity is related to the mobile entity; obtain a first grid based on the points marked as non-entity points in the first three-dimensional point cloud; identify mobile entity points among the entity points based on the projection results of the points marked as entity points in the first three-dimensional point cloud on the first grid, wherein the mobile entity points are points related to the mobile entity among the points marked as entity points; and remove the mobile entity points in the first three-dimensional point cloud to obtain the second three-dimensional point cloud.

[0017] In one possible implementation, in the first grid, when the number of existing projection points in the grid where the projection point of the first point is located is greater than a first threshold, the first point is a moving entity point, wherein the first point is any point marked as an entity point in the first three-dimensional point cloud; when the number of existing projection points in the grid where the projection point of the first point is located is less than or equal to the first threshold, the first point is a stationary entity point.

[0018] In one possible implementation, when the processing module performs texture mapping on the Mesh model based on the sequence images, it is specifically used to: identify the entity faces in the triangular faces on the Mesh model based on the number of images of the first entity contained in the visual images of the triangular faces on the Mesh model, wherein the entity faces are faces related to the first entity in the triangular faces on the Mesh model; identify the moving entity faces in the entity faces based on the number of images of the visual images of the entity faces on the Mesh model and the number of images of the first entity contained in the visual images of the entity faces, wherein the moving entity faces are faces related to the moving entity in the entity faces; remove the images of the first entity contained in the visual images of the moving entity faces, and, based on the removal results, perform texture mapping on the Mesh model.

[0019] In one possible implementation, when the number of images of the first entity included in the visible image of the triangular face is greater than a second threshold, the triangular face is a solid face; when the number of images of the first entity included in the visible image of the triangular face is less than or equal to the second threshold, the triangular face is a non-solid face.

[0020] In one possible implementation, when the first number is equal to the second number, the triangular face is a stationary entity face, the first number is the number of visible images of the triangular face, and the second number is the number of images of the first entity included in the visible images of the triangular face; when the first number is greater than the second number, the triangular face is a moving entity face.

[0021] In a possible implementation, the mobile entity includes one or more of the following: a pedestrian, a car, a motorcycle, a bicycle, an electric vehicle, or an animal.

[0022] In a possible implementation, the three-dimensional model is a three-dimensional map.

[0023] In a possible implementation, the first scene is a city scene.

[0024] In a third aspect, the present application provides a computing device cluster comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect.

[0025] In a fourth aspect, the present application provides a computer-readable storage medium comprising computer program instructions, which, when executed by a computing device cluster, causes the computing device cluster to execute the method described in the first aspect or any possible implementation of the first aspect, wherein the computing device cluster includes at least one computing device.

[0026] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed by a computing device cluster, enables the computing device cluster to execute the method described in the first aspect or any possible implementation of the first aspect, wherein the computing device cluster includes at least one computing device.

[0027] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a schematic diagram of the architecture of a three-dimensional scene reconstruction system provided in an embodiment of the present application;

[0029] Figure 2 1 is a schematic diagram of a point cloud classification process provided in an embodiment of the present application;

[0030] Figure 3 This is a schematic diagram of a client-server model provided in an embodiment of the present application;

[0031] Figure 4 This is a flow chart of a three-dimensional scene reconstruction method provided in an embodiment of the present application;

[0032] Figure 5 This is a schematic diagram of a process for removing points related to a moving entity in a first three-dimensional point cloud based on a sequence of images containing an image of a moving entity, provided by an embodiment of the present application;

[0033] Figure 6 This is a schematic diagram of a process for texture mapping a Mesh model based on a sequence of images provided in an embodiment of the present application;

[0034] Figure 7 Schematic diagram of the structure of a three-dimensional scene reconstruction device provided in an embodiment of the present application;

[0035] Figure 8 is a structural diagram of a computing device provided in an embodiment of the present application;

[0036] Figure 9 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0037] Figure 10 This is a schematic diagram of the structure of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The term "and / or" as used herein describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. The symbol " / " as used herein indicates that the related objects are in an "or" relationship, for example, A / B means either A or B.

[0039] The terms "first" and "second" in this specification and claims are used to distinguish different objects rather than to describe a specific order of objects. For example, "first response message" and "second response message" are used to distinguish different response messages rather than to describe a specific order of response messages.

[0040] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0041] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more, for example, multiple processing units means two or more processing units, etc.; multiple elements means two or more elements, etc.

[0042] Generally, in the process of making a three-dimensional model, vehicles in the image can be identified and images containing vehicles can be marked. After the three-dimensional model is generated based on the image, the visual image corresponding to each triangular face on the three-dimensional model (that is, the image that is not blocked and the triangular face can be observed) can be counted through the projection relationship between the triangular face and the image. Then, the triangular face is divided into: non-vehicles, moving vehicles and non-moving vehicles based on the visual images corresponding to the triangular face. Finally, the triangular face classified as a moving vehicle is removed from the three-dimensional model, and the boundaries of the holes in the three-dimensional model caused by the removal of the triangular face are filled. Although this method can reduce the impact of moving objects on the three-dimensional model to a certain extent, because the moving vehicle has been constructed in the modeling stage, there will be moving vehicles on the triangular faces of the three-dimensional model. When there is a moving vehicle on a certain triangular face of a three-dimensional model, in the process of counting the visual images corresponding to the triangular face through projection relationships, since the moving vehicle only exists in a part of the image corresponding to the triangular face, it is easy to mistakenly judge the image corresponding to the triangular face where there is no moving vehicle as an image where there is a vehicle and the vehicle is blocked, resulting in the inability to obtain a correct visual image, leading to incorrect recognition of the moving vehicle and the inability to accurately remove the moving vehicle.

[0043] In view of this, an embodiment of the present application provides a three-dimensional scene reconstruction system that can remove points associated with mobile vehicles in a three-dimensional point cloud before constructing a Mesh model, thereby achieving the geometric removal of the mobile vehicle, providing high-quality input for the subsequent construction of the Mesh model, and providing a correct occlusion relationship for subsequent texture mapping, thereby improving the quality of the constructed three-dimensional model. In addition, during the texture mapping stage, the mobile vehicles present in the visual image set of the triangular faces on the Mesh model can also be removed, so that during the texture mapping process, no mobile vehicles can be observed from the visual image set of any triangular face, thereby avoiding the impact of mobile vehicles on texture mapping, and thus avoiding post-processing or manual modeling, improving model reconstruction efficiency, and reducing costs.

[0044] For example, Figure 1 FIG. 1 shows a schematic diagram of the architecture of a three-dimensional scene reconstruction system provided by an embodiment of the present application. Figure 1 As shown, the three-dimensional scene reconstruction system 100 may include: an image acquisition module 110, an aerial triangulation (aerotriangulation) module 120, a point cloud generation module 130, a vehicle identification module 140, a three-dimensional point cloud optimization module 150, a Mesh construction module 160, a moving vehicle texture removal module 170 and a texture mapping module 180.

[0045] The image acquisition module 110 is primarily used to acquire a set of sequential images (i.e., a series of images arranged in a certain order) of a target scene captured from multiple different viewpoints or angles, with adjacent images overlapping (i.e., containing the same areas). The target scene provides the visual information necessary for constructing a 3D model, such as geometric shape, texture, and color. For example, these images may be, but are not limited to, satellite images or aerial images.

[0046] The aerial triangulation module 120 is primarily used to perform aerial triangulation on the images, determining the camera pose (e.g., position and orientation) corresponding to each image, as well as the 3D coordinates of certain feature points on the ground. The aerial triangulation results provide initial scene geometry information for subsequent 3D model construction. Processing the images with the aerial triangulation module 120 produces a relatively sparse 3D point cloud; the points in the 3D point cloud are derived from feature points detected in the images through aerial triangulation.

[0047] The point cloud generation module 130 is mainly used to generate a richer and more accurate 3D point cloud based on the 3D point cloud provided by the aerial triangulation module 120. Exemplarily, the point cloud generation module 130 may include: a dense matching module 131 and a point cloud fusion module 132.

[0048] The dense matching module 131 is primarily used to match pixels in the image using a dense matching algorithm to find correspondences between adjacent images. Furthermore, for the matched pixels, the dense matching module 131 uses the camera's intrinsic parameters and the 3D point cloud provided by the aerotriangulation module 120 as a reference to estimate the depth values (distance from the camera) of these matched pixels. Furthermore, the dense matching module 131 can also convert its estimated depth values into 3D coordinates to obtain the 3D point corresponding to each pixel in the image, thereby generating a relatively dense 3D point cloud.

[0049] The point cloud fusion module 132 is mainly used to fuse the sparse three-dimensional point cloud provided by the aerial triangulation module 120 and the dense three-dimensional point cloud provided by the dense matching module 131 to generate a richer and more accurate three-dimensional point cloud. Since the image is not processed to remove moving vehicles during the process of generating the three-dimensional point cloud by the point cloud generation module 130, the three-dimensional point cloud obtained at this time contains moving vehicles. In some embodiments, the point cloud fusion module 132 can also record which depth maps are fused to obtain each point in the three-dimensional point cloud during the point cloud fusion process, and then determine the visible image of each point in the three-dimensional point cloud based on the correspondence between the depth map and the image. For example, when a point is obtained through a depth map D, if the depth map D is obtained by processing the image P, the visible image of the point is image P.

[0050] The vehicle recognition module 140 is mainly used to identify vehicles in each image through algorithms such as neural networks, so as to divide the images into images containing vehicles and images not containing vehicles. In addition, when the vehicle recognition module 140 recognizes that a certain image contains a vehicle, it can also assign a label value to each pixel in the image that belongs to a vehicle, and the label value is used to indicate that the pixel belongs to a vehicle. For example, Figure 2 As shown in (A), a label value can be assigned to both pixel points 22 and 23 to indicate that both belong to the vehicle. Of course, another label value can also be assigned to the pixel points that do not belong to the vehicle to indicate that these pixel points do not belong to the vehicle. For example, continue to refer to Figure 2 (A) can assign a label value to pixel 21 indicating that it does not belong to a vehicle.

[0051] The 3D point cloud optimization module 150 is primarily used to utilize the recognition results of the vehicle identification module 140 to remove points belonging to moving vehicles from the 3D point cloud generated by the point cloud generation module 130, thereby obtaining a 3D point cloud that does not contain moving vehicles. Exemplarily, the 3D point cloud optimization module 150 may include a point cloud classification module 151 and a moving vehicle point cloud removal module 152.

[0052] Among them, the point cloud classification module 151 is mainly used to divide the points in the three-dimensional point cloud into vehicle points and non-vehicle points based on the recognition results of the vehicle recognition module 140 and the visual images of each point in the three-dimensional point cloud, so as to complete the classification of the points in the three-dimensional point cloud. The visual images of the points in the three-dimensional point cloud can be obtained from, but not limited to, the point cloud generation module 130. For any point (point cloud) in the three-dimensional point cloud, when there is a pixel point marked as a vehicle in any visual image of the point, the point can be marked as a vehicle point (or vehicle); when there is no pixel point marked as a vehicle in all the visual images of the point, the point can be marked as a non-vehicle point (or vehicle). For example, continue to refer to Figure 2(A), the visible image of point cloud A is P1, the visible image of point cloud B is P2, the visible images of point cloud C are P1 and P2, point cloud A is associated with pixel 21, point cloud B is associated with pixel 22, point cloud C is associated with pixel 23, and the coordinates of pixels 21 and 23 are the same. Figure 2 In (A), pixels 22 and 23 can be marked as vehicles, while pixel 21 will not be marked as a vehicle. Therefore, the result of dividing point clouds A, B and C can be as follows: Figure 2 As shown in (B), point cloud A is a non-vehicle point, and point clouds B and C are both vehicle points. Figure 2 In (A), since vehicle C3 does not appear in P1, but appears in P2, for the point cloud related only to P1, all points except the point clouds corresponding to the pixel points related to vehicles C1 and C2 will be classified as non-vehicle points; for the point cloud related only to P2, all points except the point clouds corresponding to the pixel points related to vehicles C1, C2 and C3 will be classified as non-vehicle points; for the point cloud related only to P1 and P2, all points except the point clouds corresponding to the pixel points related to vehicles C1, C2 and C3 will be classified as non-vehicle points.

[0053] The mobile vehicle point cloud removal module 152 is mainly used to identify points belonging to mobile vehicles in the three-dimensional point cloud based on the classification results of the point cloud classification module 151, and to remove points belonging to mobile vehicles in the three-dimensional point cloud to obtain a three-dimensional point cloud that does not contain mobile vehicles. Exemplarily, the mobile vehicle point cloud removal module 152 can project the points marked as non-vehicles in the three-dimensional point cloud onto the XY plane (ground) for grid division. The result of the grid division can be that each point is in a grid, and a grid can accommodate at most one point. Then, the mobile vehicle point cloud removal module 152 can project each point marked as a vehicle in the three-dimensional point cloud onto the grid. When the grid where the projection point of a point is located is empty, it is considered that the ground is missing and the point is marked as a stationary vehicle point; otherwise, it is marked as a moving vehicle point. For example, continue to refer to Figure 2 After completing the classification of all point clouds related to images P1 and P2, the point clouds marked as non-vehicle points can be projected onto the XY plane and meshed, ensuring that a point cloud is located in a grid and that a grid can accommodate at most one point, that is, the following can be obtained: Figure 2 The division result is shown in (C). Figure 2 In (C), the grid 24 is the projection point of the point cloud A. Since there are vehicle points in the classification results of all point clouds related to images P1 and P2, Figure 2 There will be grids (such as grid 25) that do not contain projection points in (C). Figure 2After the division result shown in (C) is obtained, each point cloud marked as a vehicle can be traversed and these point clouds can be projected onto Figure 2 In the grid shown in (C), when the projection point of a point cloud marked as a vehicle is located in an empty grid, the point cloud can be marked as a stationary vehicle point (i.e., a non-moving vehicle point). For example, the projection point of point cloud C is located in grid 25, but previously this grid 25 did not contain any projection points. Therefore, point cloud C can be marked as a stationary vehicle point. When the projection point of a point cloud marked as a vehicle is located in a non-empty grid (i.e., a grid with projection points), the point cloud can be marked as a moving vehicle point. For example, the projection point of point cloud B is located in grid 24, but previously this grid 24 already contained the projection point of point cloud A. Therefore, point cloud B can be marked as a moving vehicle point. In this way, we can get Figure 2 The classification result is shown in (D). After completing the classification of all point clouds in the three-dimensional point cloud, the mobile vehicle point cloud removal module 152 can remove the point clouds marked as mobile vehicle points from the three-dimensional point cloud. In some embodiments, when performing grid division, a grid can also accommodate multiple points. At this time, when the points marked as vehicle points are projected into the grid respectively, when the number of existing projection points in the grid where the projection point of a point is located is greater than a certain threshold, the point can be marked as a mobile vehicle point, otherwise it is marked as a stationary vehicle point.

[0054] Mesh Construction Module 160 primarily uses a 3D triangulated meshing method based on Delaunay tetrahedrons and graph cuts to process the 3D point cloud output from 3D Point Cloud Optimization Module 150, excluding any moving vehicles, to construct a mesh model. Because the point cloud containing moving vehicles is removed before constructing the Mseh model, the triangular faces of the constructed mesh model will not contain any moving vehicles. For example, each vertex of a triangular face can be associated with a point in the 3D point cloud.

[0055] The moving vehicle texture removal module 170 is primarily responsible for filtering out, from the images acquired by the image acquisition module 110, images used for each triangular facet on the mesh model during the texture mapping phase. These filtered images do not contain moving vehicles. Exemplarily, the moving vehicle texture removal module 170 may include a module 171 for counting visible images corresponding to triangular faces and containing vehicles, a triangular face labeling module 172, and a visible image optimization module 173.

[0056] The module 171 for counting the number of visible images corresponding to triangular faces and containing vehicles is primarily used to count the number of images marked as containing vehicles in the visible images corresponding to each triangular face, based on the recognition results of the vehicle recognition module 140 and the visible images corresponding to the triangular faces of the Mseh model. For example, the visible images of each triangular face on the Mesh model can be found from the images acquired by the image acquisition module 110 using, but not limited to, projection methods, Z-Buffer algorithms, or Ray casting algorithms. For example, the set of visible images of a triangular face is the set of images in which the vertices of the triangular face can be observed in the sequence of images.

[0057] The triangle face labeling module 172 is primarily used to classify triangle faces into vehicles or non-vehicles based on the statistics generated by the vehicle count module 171. For any triangle face, if the number of vehicle-containing images corresponding to the triangle face is greater than a first threshold, the triangle face may be labeled as a vehicle; if the number of vehicle-containing images corresponding to the triangle face is less than or equal to the first threshold, the triangle face may be labeled as a non-vehicle.

[0058] The visual image optimization module 173 is mainly used to classify the triangular faces marked as vehicles based on the number of visual images containing vehicles corresponding to the triangular faces, so as to divide these triangular faces into moving vehicles and stationary vehicles, and to remove the images containing vehicles from the visual images corresponding to the triangular faces marked as moving vehicles, that is, only retain the images that do not contain vehicles. For example, for any triangular face marked as a vehicle, when the number of visual images containing vehicles corresponding to the triangular face is equal to the number of visual images corresponding to the triangular face, the triangular face can be marked as a stationary vehicle; when the number of visual images containing vehicles corresponding to the triangular face is not equal to the number of visual images corresponding to the triangular face, the triangular face can be marked as a moving vehicle. For example, if there are 10 visual images corresponding to the triangle surface T, and if there are 8 visual images corresponding to it that contain vehicles, then the triangle surface T can be marked as a moving vehicle. At the same time, these 8 visual images containing vehicles can be removed later, and the remaining 2 visual images that do not contain vehicles can be used to perform texture mapping for the triangle surface T; if there are 10 visual images corresponding to it that contain vehicles, then the triangle surface T can be marked as a stationary vehicle, and at the same time, these 10 visual images can be used to perform texture mapping for the triangle surface T later.

[0059] The texture mapping module 180 is mainly used to perform texture mapping on the Mesh model output by the Mesh construction module 160 based on the visual image that does not contain the moving vehicle output by the moving vehicle texture removal module 170, so as to obtain a real-life three-dimensional model. For example, for any triangular face, a suitable image can be selected from the visual image that does not contain the vehicle corresponding to the triangular face based on the criteria such as the minimum angle between the visual image corresponding to the triangular face and the normal vector on the Mesh model, color consistency, and geometric consistency. The selected texture image is then mapped to the triangular face through texture mapping algorithms such as normal mapping, screen space reflection (SSR), displacement mapping, and triplanar mapping to obtain a real-life three-dimensional model.

[0060] In some embodiments, Figure 1 The three-dimensional scene reconstruction system 100 shown can be based on Figure 3 The 3D scene reconstruction system 100 can be implemented using the client-server model shown in FIG. At this point, the 3D scene reconstruction system 100 can include a client-side portion (hereinafter referred to as "client 310") executed on the user device 301, and a server-side portion (hereinafter referred to as "server 320") executed on the server system. The client 310 communicates with the server 320 via one or more networks 330. The client 310 provides client-side functions, such as receiving images of the target scene input by the user and presenting the constructed real-scene 3D model to the user, and communicates with the server 320, such as uploading the images input by the user to the server 320 and receiving the real-scene 3D model issued by the server 320. The server 320 provides server-side functions for one or more clients 310, such as computing and processing functions (such as constructing a real-scene 3D model based on images). Exemplarily, the client 310 can be, but is not limited to, a desktop application, a mobile application, a web application, or a web-based application. The server 320 can be, but is not limited to, a cloud server, a central server, an edge server, or a local server in a local data center. The network 330 may be a wired network or a wireless network.

[0061] The above is an introduction to the 3D scene reconstruction system provided in the embodiments of the present application. Next, based on the above content, a 3D scene reconstruction method provided in the embodiments of the present application is introduced. It is understood that this method is proposed based on the 3D scene reconstruction system described above, and part or all of the content of this method can be found in the description of the 3D scene reconstruction system above.

[0062] For example, Figure 4 The flowchart of a three-dimensional scene reconstruction method provided by an embodiment of the present application is shown. It can be understood that the method can be executed by any device, equipment, platform, or equipment cluster with computing and processing capabilities. For ease of description, the following is an example of server execution. Figure 4 As shown, the three-dimensional reconstruction method may include the following steps:

[0063] S401: Acquire a set of sequential images obtained by photographing a first scene from different viewing angles or perspectives, wherein the first scene is used to provide visual information required for constructing a three-dimensional model.

[0064] In this embodiment, a user can use a device such as a drone to capture a first scene from different perspectives or angles to obtain a sequence of images. The user can then upload these images to a server so that the server can access them. For example, the first scene can be used to provide visual information required to construct a three-dimensional model, such as geometric shapes and textures. For example, the first scene can be a city scene, for example.

[0065] S402 : Based on the images of the sequence of images including the images of the mobile entity, remove points related to the mobile entity from the first three-dimensional point cloud to obtain a second three-dimensional point cloud, wherein the first three-dimensional point cloud is generated based on the sequence of images.

[0066] In this embodiment, based on the images containing the mobile entities in the sequence of images, the server can remove the points related to the mobile entities in the first three-dimensional point cloud generated by the sequence of images to obtain a second three-dimensional point cloud. In this way, the second three-dimensional point cloud no longer contains the mobile entities. Exemplarily, the mobile entity refers to an object that moves in space, such as pedestrians, cars, motorcycles, bicycles, electric vehicles, animals, etc. Exemplarily, the sequence of images can be processed by the aforementioned aerial triangulation module 120 and the point cloud generation module 130 to generate the first three-dimensional point cloud. In addition, the first three-dimensional point cloud can be processed by the aforementioned three-dimensional point cloud optimization module 150 to obtain the second three-dimensional point cloud.

[0067] As a possible implementation, Figure 5As shown, the process of obtaining the second 3D point cloud may include the following steps: At S501, a set of visible images of points in the first 3D point cloud is obtained from a sequence of images, where the visible image set of a point is the set of images in which the point can be observed in the sequence of images. At S502, based on the number of images of the point in the visible image set that contain the first entity, the point is marked as an entity point or a non-entity point. The first entity may be an entity related to the mobile entity to be removed. For example, when a mobile vehicle is to be removed, the first entity is a vehicle. When the number of images of the first entity in the visible image set of a point is greater than zero, the point may be marked as an entity point; otherwise, it is marked as a non-entity point. Exemplarily, marking may be understood as identifying. It should be understood that the entity points and non-entity points refer to the same entity as the first entity. For example, when the first entity is a vehicle, entity points are vehicle points, and non-entity points are non-vehicle points. In other words, entity points are points in the first 3D point cloud that are related to the first entity, while non-entity points are points in the first 3D point cloud that are not related to the first entity. In S503, the points marked as non-entity points in the first three-dimensional point cloud are projected onto a first plane (e.g., an XY plane or the ground, etc.), and meshed to obtain a first mesh. In S504, the points marked as entity points in the first three-dimensional point cloud are projected onto the first mesh, and based on the projection results, the points marked as entity points are marked as moving entity points or stationary entity points. Here, a moving entity point is a point related to a mobile entity among the points marked as entity points, and a stationary entity point is a point not related to a mobile entity among the points marked as entity points. For example, when the first entity is a vehicle, a moving entity point is a point related to the mobile vehicle in the three-dimensional point cloud, and a stationary entity point is a point not related to the mobile vehicle in the three-dimensional point cloud. Exemplarily, if the number of existing projection points in the grid where the projection point of the first point resides is greater than a first threshold (e.g., 0), the first point is marked as a moving entity point, where the first point is any point marked as an entity point in the first three-dimensional point cloud. If the number of existing projection points in the grid where the projection point of the first point resides is less than or equal to the first threshold, the first point is marked as a stationary entity point. In S505, the points marked as moving entity points in the first three-dimensional point cloud are removed to obtain a second three-dimensional point cloud. In this way, points associated with moving objects in the three-dimensional point cloud are removed.

[0068] S403: Generate a mesh model based on the second three-dimensional point cloud. The mesh model can be generated by processing the second three-dimensional point cloud based on a three-dimensional triangulation method using Delaunay tetrahedron and graph cuts.

[0069] S404: Based on the sequence of images, texture mapping is performed on the Mesh model to obtain a three-dimensional model related to the first scene.

[0070] In this embodiment, after obtaining the mesh model, texture mapping can be performed on the mesh model based on the acquired sequence of images, thereby obtaining a three-dimensional model related to the first scene. Exemplarily, the three-dimensional model related to the first scene can be a virtual model that realistically reflects the first scene in the real world; it can include all objects, environments, and details in the first scene, and can be observed and interacted with from different perspectives and angles. Exemplarily, the three-dimensional model related to the first scene can be a three-dimensional map.

[0071] In this way, the construction of a 3D scene is achieved. Because the points associated with the moving entities in the 3D point cloud are removed before the Mesh model is constructed, the moving entities can be geometrically removed, providing high-quality input for the subsequent construction of the Mesh model and correct occlusion relationships for subsequent texture mapping, thereby improving the quality of the constructed 3D model.

[0072] In some embodiments, for S404, as Figure 6 As shown, the process of texture mapping a Mesh model based on a sequence of images can include the following steps:

[0073] S610: Based on the number of images of the first entity in the visible image set of the triangular facets on the mesh model, the triangular facets on the mesh model are classified to mark the triangular facets as non-solid, stationary, or moving. It should be understood that the entity involved in the non-solid, stationary, or moving entity facets is the same as the first entity. For example, when the first entity is a vehicle, the solid facets are vehicle faces, and the non-solid facets are non-vehicle faces. Exemplarily, S610 can be implemented through S611 to S617. In S611, a visible image set of the triangular facets on the mesh model is obtained from a sequence of images. The visible image set of the triangular facets is the set of images in the sequence of images in which the vertices of the triangular facets can be observed. In S612, it is determined whether the number of images of the first entity in the visible image set of the triangular facets is greater than a second threshold. If so, S614 to S617 can be executed; otherwise, S613 is executed. In S613, the triangular facets are marked as non-solid. Exemplarily, marking can be understood as identification. The non-solid face is a face in the triangular face on the Mesh model that is not related to the first entity. For example, when the first entity is a vehicle, and the visible image of a certain triangular face does not contain a vehicle, the triangular face can be marked as a non-solid face. In S614, the triangular face is marked as a solid face. The solid face is a face in the triangular face on the Mesh model that is related to the first entity. For example, when the first entity is a vehicle, and the visible image of a certain triangular face contains a vehicle, the triangular face can be marked as a non-solid face. In S615, when the triangular face is marked as a solid face, it can be determined whether the number of images contained in the visual image set of the triangular face is equal to the number of images containing the first entity in the visual image set of the triangular face. If so, S617 can be executed; otherwise, S616 can be executed. In S616, the triangular face is marked as a moving solid face. In S617, the triangular face is marked as a stationary solid face. The moving entity face is a face related to the moving entity in the triangular face marked as the entity face, and the stationary entity face is a face not related to the moving entity in the triangular face marked as the entity face.

[0074] S620. When a certain triangular face is marked as a moving entity face, it indicates that a moving object (i.e., a first entity, such as a vehicle, etc.) can be observed by combining all the images in the visual image set of the triangular face. In order to reduce the influence of the moving object on the texture mapping of the triangular face, the image containing the first entity in the visual image set of the triangular face can be removed. In this way, the moving object cannot be seen when all the images in the visual image set of the triangular face are combined.

[0075] S630: Texture mapping is performed on the mesh model based on the visible image set of the first triangular facet, the visible image set of the second triangular facet, and the visible image set of the third triangular facet that does not contain the first entity. The first triangular facet is a triangular facet marked as a non-solid facet, the second triangular facet is a triangular facet marked as a stationary solid facet, and the third triangular facet is a triangular facet marked as a moving solid facet. In other words, texture mapping can be performed on the mesh model based on the visible image set of the third triangular facet that does not contain the first entity.

[0076] Since no moving objects can be observed in the visible image set of any triangle face during the texture mapping process, the purpose of accurately removing moving objects is achieved, thereby avoiding the impact of moving objects on texture mapping, avoiding post-processing or manual modeling, improving model reconstruction efficiency, and reducing costs.

[0077] It should be understood that the order of execution of the steps in the above embodiments does not necessarily imply a specific order of execution. The order of execution of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In addition, the various embodiments and / or steps in the embodiments described above can be combined according to actual circumstances, and the combined solutions are still within the scope of protection of this application.

[0078] Next, based on the method in the above embodiment, the three-dimensional scene reconstruction device provided in the embodiment of the present application is introduced.

[0079] For example, Figure 7 FIG. 1 shows a schematic structural diagram of a three-dimensional scene reconstruction device provided in an embodiment of the present application. Figure 7 As shown, the 3D scene reconstruction device 700 includes: an acquisition module 701 and a processing module 702. The acquisition module 701 is used to acquire a set of sequential images of a first scene captured from different viewing angles or angles, wherein the first scene is used to provide visual information required for constructing a 3D model. The processing module 702 is used to remove points related to the moving entities in a first 3D point cloud based on the images of the moving entities contained in the sequential images, to obtain a second 3D point cloud, wherein the first 3D point cloud is generated based on the sequential images; and, based on the second 3D point cloud, to generate a mesh model; and, based on the sequential images, to perform texture mapping on the mesh model to obtain a 3D model related to the first scene.

[0080] In some embodiments, when removing points related to the mobile entity in the first three-dimensional point cloud based on the images of the sequence containing the mobile entity to obtain the second three-dimensional point cloud, the processing module 702 is specifically used to: mark the points in the first three-dimensional point cloud as entity points or non-entity points based on the number of images of the first entity in the visible images of the points in the first three-dimensional point cloud, wherein the entity points are points related to the first entity in the first three-dimensional point cloud, and the first entity is related to the mobile entity; obtain a first grid based on the points marked as non-entity points in the first three-dimensional point cloud; identify mobile entity points among the entity points based on the projection results of the points marked as entity points in the first three-dimensional point cloud on the first grid, wherein the mobile entity points are points related to the mobile entity among the points marked as entity points; and remove the mobile entity points in the first three-dimensional point cloud to obtain the second three-dimensional point cloud.

[0081] In some embodiments, in the first grid, when the number of existing projection points in the grid where the projection point of the first point is located is greater than a first threshold, the first point is a moving entity point, wherein the first point is any point marked as an entity point in the first three-dimensional point cloud; when the number of existing projection points in the grid where the projection point of the first point is located is less than or equal to the first threshold, the first point is a stationary entity point.

[0082] In some embodiments, when the processing module 702 performs texture mapping on the Mesh model based on the sequence images, it is specifically used to: identify the solid faces in the triangular faces on the Mesh model based on the number of images of the first entity contained in the visual images of the triangular faces on the Mesh model, wherein the solid faces are faces related to the first entity in the triangular faces on the Mesh model; identify the moving solid faces in the solid faces based on the number of images of the visual images of the solid faces on the Mesh model and the number of images of the first entity contained in the visual images of the solid faces, wherein the moving solid faces are faces related to the moving entity in the solid faces; remove the images of the first entity contained in the visual images of the moving solid faces, and, based on the removal results, perform texture mapping on the Mesh model.

[0083] In some embodiments, when the number of images of the first entity included in the visible image of the triangular face is greater than a second threshold, the triangular face is a solid face; when the number of images of the first entity included in the visible image of the triangular face is less than or equal to the second threshold, the triangular face is a non-solid face.

[0084] In some embodiments, when the first number is equal to the second number, the triangular face is a stationary physical face, the first number is the number of visible images of the triangular face, and the second number is the number of images of the first entity included in the visible images of the triangular face; when the first number is greater than the second number, the triangular face is a moving physical face.

[0085] In some embodiments, the mobile entity includes one or more of the following: a pedestrian, a car, a motorcycle, a bicycle, an electric vehicle, or an animal.

[0086] In some embodiments, the three-dimensional model is a three-dimensional map.

[0087] In some embodiments, the first scene is a city scene.

[0088] In some embodiments, Figure 7 The acquisition module 701 and the processing module 702 shown in FIG can be implemented by software or hardware. For example, the implementation of the acquisition module 701 is described below using the acquisition module 701 as an example. Similarly, the implementation of the processing module 702 can refer to the implementation of the acquisition module 701.

[0089] As an example of a software functional unit, the acquisition module 701 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the acquisition module 701 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0090] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0091] As an example of a hardware functional unit, the acquisition module 701 may include at least one computing device, such as a server. Alternatively, the acquisition module 701 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0092] The multiple computing devices included in acquisition module 701 can be distributed in the same region or in different regions. The multiple computing devices included in acquisition module 701 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in acquisition module 701 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0093] It should be noted that, in other embodiments, the acquisition module 701 can be used to execute any step in the three-dimensional scene reconstruction method described in the above embodiment, and the processing module 702 can be used to execute any step in the three-dimensional scene reconstruction method described in the above embodiment. The steps that the acquisition module 701 and the processing module 702 are responsible for implementing can be specified as needed, and the acquisition module 701 and the processing module 702 respectively implement different steps in the three-dimensional scene reconstruction method described in the above embodiment to achieve Figure 7 The entire functions of the three-dimensional scene reconstruction device 700 are shown.

[0094] The present application also provides a computing device 800. Figure 8 As shown, computing device 800 includes a bus 802, a processor 804, a memory 806, and a communication interface 808. Processor 804, memory 806, and communication interface 808 communicate with each other via bus 802. Computing device 800 can be a server, such as a central server, an edge server, or a local server in a local data center, or an electronic device such as a desktop computer, a laptop computer, or a smartphone. It should be understood that this application does not limit the number of processors and memories in computing device 800.

[0095] The bus 802 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The bus 804 may include a path for transmitting information between various components of the computing device 800 (eg, memory 806, processor 804, communication interface 808).

[0096] The processor 804 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0097] The memory 806 may include volatile memory, such as random access memory (RAM). The processor 804 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0098] The memory 806 stores executable program codes, and the processor 804 executes the executable program codes to respectively implement the aforementioned Figure 7 The functions of the acquisition module 701 and the processing module 702 shown in FIG are implemented to realize the 3D scene reconstruction method described in the above embodiment. That is, the memory 806 stores instructions for executing the 3D scene reconstruction method described in the above embodiment.

[0099] Alternatively, the memory 806 stores executable codes, and the processor 804 executes the executable codes to respectively implement the aforementioned Figure 7 The functions of the 3D scene reconstruction apparatus 700 shown in FIG are implemented to realize the 3D scene reconstruction method described in the above embodiment. That is, the memory 806 stores instructions for executing the 3D scene reconstruction method described in the above embodiment.

[0100] The communication interface 803 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 800 and other devices or a communication network.

[0101] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be an electronic device such as a desktop computer, a laptop computer, or a smartphone.

[0102] like Figure 9 As shown, the computing device cluster includes at least one computing device 800. The memory 806 in one or more computing devices 800 in the computing device cluster may store the same instructions for executing the three-dimensional scene reconstruction method described in the above embodiment.

[0103] In some possible implementations, the memory 806 of one or more computing devices 800 in the computing device cluster may also store partial instructions for executing the 3D scene reconstruction method described in the above embodiments. In other words, the combination of one or more computing devices 800 can jointly execute the instructions for executing the 3D scene reconstruction method described in the above embodiments.

[0104] It should be noted that the memory 806 in different computing devices 800 in the computing device cluster can store different instructions, which are respectively used to execute the above Figure 7 The illustrated three-dimensional scene reconstruction apparatus 700 partially functions. That is, the instructions stored in the memory 806 of different computing devices 800 can implement the functions of one or more modules in the acquisition module 701 and the processing module 702.

[0105] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network, which may be a wide area network or a local area network. Figure 10 A possible implementation is shown. Figure 10 As shown, two computing devices 800A and 800B are connected via a network. Specifically, the connection to the network is achieved through a communication interface within each computing device. In this possible implementation, the memory 806 within computing device 800A stores instructions for executing the functions of acquisition module 701. Simultaneously, the memory 806 within computing device 800B stores instructions for executing the functions of processing module 702.

[0106] It should be understood that Figure 10The functionality of the computing device 800A shown in FIG. 8 may also be implemented by multiple computing devices 800. Similarly, the functionality of the computing device 800B may also be implemented by multiple computing devices 800.

[0107] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similarly referred to as Figure 9 and Figure 10 The connection mode of the computing device cluster is different in that the memory 806 of one or more computing devices 800 in the computing device cluster may store the same instructions for executing the method in the above embodiment.

[0108] In some possible implementations, the memory 806 of one or more computing devices 800 in the computing device cluster may also store some instructions for executing the aforementioned data processing method. In other words, the combination of one or more computing devices 800 can jointly execute instructions for executing the aforementioned three-dimensional scene reconstruction method.

[0109] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, including computer program instructions, which, when executed by a computing device cluster comprising at least one computing device, causes the computing device cluster to execute the method in the above embodiment. Exemplarily, the computer-readable storage medium can be any available medium that can be stored in the computing device cluster or a data storage device such as a data center comprising one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive), etc.

[0110] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product containing instructions. When the instructions are executed by a computing device cluster including at least one computing device, the computing device cluster executes the method in the above embodiment.

[0111] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0112] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0113] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0114] It will be understood that the various numerical numbers involved in the embodiments of the present application are merely distinctions for the convenience of description and are not intended to limit the scope of the embodiments of the present application.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A three-dimensional scene reconstruction method, characterized in that: The method comprises: Acquire a set of sequential images obtained by photographing a first scene from different perspectives, wherein the first scene is used to provide visual information required for constructing a three-dimensional model; Based on the image of the mobile entity included in the sequence of images, removing points related to the mobile entity from a first three-dimensional point cloud to obtain a second three-dimensional point cloud, wherein the first three-dimensional point cloud is generated based on the sequence of images; Generate a mesh model based on the second three-dimensional point cloud; Based on the sequence of images, texture mapping is performed on the Mesh model to obtain a three-dimensional model related to the first scene.

2. The method according to claim 1, characterized in that The step of removing points related to the mobile entity from the first three-dimensional point cloud based on the image of the mobile entity included in the sequence of images to obtain a second three-dimensional point cloud includes: Marking points in the first three-dimensional point cloud as entity points or non-entity points based on the number of images of the first entity included in the visible images of the points in the first three-dimensional point cloud, wherein the entity points are points in the first three-dimensional point cloud associated with the first entity, and the first entity is associated with the mobile entity; Obtaining a first mesh based on points marked as non-entity points in the first three-dimensional point cloud; Identifying, based on projection results of points marked as entity points in the first three-dimensional point cloud onto the first grid, mobile entity points among the marked entity points, wherein the mobile entity points are points among the entity points that are related to the mobile entity; The moving entity points in the first three-dimensional point cloud are removed to obtain the second three-dimensional point cloud.

3. The method according to claim 2, characterized in that In the first grid, when the number of existing projection points in the grid where the projection point of the first point is located is greater than a first threshold, the first point is a moving entity point, wherein the first point is any point marked as an entity point in the first three-dimensional point cloud.

4. The method according to any one of claims 1 to 3, characterized in that: The step of performing texture mapping on the Mesh model based on the sequence of images includes: Identifying entity faces in the triangular faces on the Mesh model based on the number of images of the first entity contained in the visible images of the triangular faces on the Mesh model, wherein the entity faces are faces in the triangular faces on the Mesh model that are related to the first entity; Identifying a mobile entity face in the solid face based on the number of images of the visible image of the solid face on the mesh model and the number of images of the first entity included in the visible image of the solid face, wherein the mobile entity face is a face in the solid face associated with the mobile entity; The image of the first entity included in the visible image of the moving entity surface is removed, and texture mapping is performed on the Mesh model based on the removal result.

5. The method according to claim 4, characterized in that When the number of images of the first entity included in the visible image of the triangular face is greater than a second threshold, the triangular face is a solid face.

6. The method according to claim 4 or 5, characterized in that When the first number is greater than the second number, the triangular face is a moving entity face, the first number is the number of visible images of the entity face, and the second number is the number of images of the first entity included in the visible image of the entity face.

7. The method according to any one of claims 1 to 6, characterized in that: The mobile entity includes one or more of the following: Pedestrians, cars, motorcycles, bicycles, electric scooters, or animals.

8. The method according to any one of claims 1 to 7, characterized in that: The three-dimensional model is a three-dimensional map.

9. The method according to any one of claims 1 to 8, characterized in that: The first scene is an urban scene.

10. A three-dimensional scene reconstruction device, characterized in that: include: an acquisition module, configured to acquire a set of sequential images obtained by photographing a first scene from different perspectives, wherein the first scene is used to provide visual information required for constructing a three-dimensional model; a processing module configured to remove points related to the mobile entity from a first three-dimensional point cloud based on images of the mobile entity included in the sequence of images, so as to obtain a second three-dimensional point cloud, wherein the first three-dimensional point cloud is generated based on the sequence of images; The processing module is further configured to generate a mesh model based on the second three-dimensional point cloud; The processing module is further configured to perform texture mapping on the Mesh model based on the sequence of images to obtain a three-dimensional model related to the first scene.

11. The device according to claim 10, characterized in that The processing module is specifically configured to remove points related to the mobile entity in the first three-dimensional point cloud based on the image of the mobile entity included in the sequence of images to obtain the second three-dimensional point cloud: Marking points in the first three-dimensional point cloud as entity points or non-entity points based on the number of images of the first entity included in the visible images of the points in the first three-dimensional point cloud, wherein the entity points are points in the first three-dimensional point cloud associated with the first entity, and the first entity is associated with the mobile entity; Obtaining a first mesh based on points marked as non-entity points in the first three-dimensional point cloud; Identifying, based on projection results of points marked as entity points in the first three-dimensional point cloud onto the first grid, mobile entity points among the marked entity points, wherein the mobile entity points are points among the entity points that are related to the mobile entity; The moving entity points in the first three-dimensional point cloud are removed to obtain the second three-dimensional point cloud.

12. The device according to claim 11, characterized in that In the first grid, when the number of existing projection points in the grid where the projection point of the first point is located is greater than a first threshold, the first point is a moving entity point, wherein the first point is any point marked as an entity point in the first three-dimensional point cloud.

13. The device according to any one of claims 10 to 12, characterized in that: When the processing module performs texture mapping on the Mesh model based on the sequence of images, the processing module is specifically used to: Identifying entity faces in the triangular faces on the Mesh model based on the number of images of the first entity contained in the visible images of the triangular faces on the Mesh model, wherein the entity faces are faces in the triangular faces on the Mesh model that are related to the first entity; Identifying a mobile entity face in the solid face based on the number of images of the visible image of the solid face on the mesh model and the number of images of the first entity included in the visible image of the solid face, wherein the mobile entity face is a face in the solid face associated with the mobile entity; The image of the first entity included in the visible image of the moving entity surface is removed, and texture mapping is performed on the Mesh model based on the removal result.

14. The device according to claim 13, characterized in that When the number of images of the first entity included in the visible image of the triangular face is greater than a second threshold, the triangular face is a solid face.

15. The device according to claim 13 or 14, characterized in that When the first number is greater than the second number, the triangular face is a moving entity face, the first number is the number of visible images of the triangular face, and the second number is the number of images of the first entity included in the visible image of the triangular face.

16. The device according to any one of claims 10 to 15, characterized in that: The mobile entity includes one or more of the following: Pedestrians, cars, motorcycles, bicycles, electric scooters, or animals.

17. The device according to any one of claims 10 to 16, characterized in that: The three-dimensional model is a three-dimensional map.

18. The device according to any one of claims 10 to 17, characterized in that: The first scene is an urban scene.

19. A computing device cluster, characterized in that: comprising at least one computing device, each computing device including a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in a memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that The method comprises computer program instructions, which, when executed by a computing device cluster, enable the computing device cluster to perform the method according to any one of claims 1 to 9, wherein the computing device cluster comprises at least one computing device.

21. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster is caused to perform the method according to any one of claims 1 to 9, wherein the computing device cluster includes at least one computing device.

Images