A method, system, device and storage medium for reconstructing real three-dimensional scene

Abstract

The present invention aims at the problem that model editing efficiency is low and relies on user intuition in the prior art, and discloses a real-life three-dimensional scene reconstruction method, system, device and storage medium, which belongs to the field of computer vision modeling technology. The method includes: collecting multi-angle 2D images of the target scene, converting the 2D images into a Gaussian splash model of the target scene; assigning a semantic label to each Gaussian ellipsoid in the Gaussian splash model to obtain a semantic Gaussian splash model; editing the semantic Gaussian splash model to reconstruct the three-dimensional target scene. The present invention can significantly improve the model editing accuracy and editing efficiency, and provide technical support for film and television special effects, game development, virtual reality, etc.

Description

Technical Field

[0001] The present invention relates to three-dimensional scene reconstruction technology, belonging to the field of computer vision modeling technology, and specifically to a real-life three-dimensional scene reconstruction method, system, device and storage medium. Background Art

[0002] Among the related technologies of Gaussian scene editing, interactive editing methods are still the mainstream. Such methods enable users to directly manipulate the model to achieve the desired visual effect by providing intuitive editing tools, mainly deletion functions. However, these methods generally have defects such as low editing efficiency, reliance on user intuition, and poor automation. For large-scale or highly complex scenes, interactive editing is very time-consuming. Each change requires manual adjustment by the user, and it is impossible to accurately select the Gaussian ellipsoid to be edited, which is particularly inefficient when dealing with large amounts of data or subtle adjustments. In addition, for situations where the same or similar editing operations need to be repeatedly applied, interactive methods are not flexible enough, and intelligent and automated means are insufficient. Summary of the invention

[0003] In view of the deficiencies in the prior art, the present invention provides a real-life 3D scene reconstruction method, system, device and storage medium, aiming to solve the accuracy and timeliness problems of 3D scene editing in the prior art. The present invention integrates advanced image semantic segmentation technology to optimize the post-editing process of the Gaussian splash model and comprehensively improve the quality and efficiency of model editing.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] A method for reconstructing a real three-dimensional scene comprises the following steps:

[0006] Collect multi-angle 2D images of the target scene and convert the 2D images into a Gaussian splash model of the target scene;

[0007] Assign a semantic label to each Gaussian ellipsoid in the Gaussian splash model to obtain a semantic Gaussian splash model;

[0008] Edit the semantic Gaussian splash model and reconstruct the three-dimensional target scene.

[0009] To optimize the above technical solutions, the specific measures taken also include:

[0010] Furthermore, the Gaussian splash model for converting the 2D image into the target scene is specifically:

[0011] After feature extraction, feature matching, sparse reconstruction and global optimization of all 2D images, the camera pose corresponding to each 2D image and the sparse point cloud of the target scene are obtained;

[0012] Based on the processed 2D image, camera pose and sparse point cloud, the Gaussian splash algorithm is used to generate a Gaussian splash model of the target scene.

[0013] Furthermore, the specific process of assigning a semantic label to each Gaussian ellipsoid in the Gaussian splash model to obtain a semantic Gaussian splash model includes:

[0014] Use the Segment Anything semantic segmentation model to extract semantic information from all 2D images, generate a 2D segmentation mask, and record the coordinates (x, y) of each pixel in the 2D segmentation mask and the semantic label L of the pixel;

[0015] Get the depth information d of each 2D image;

[0016] Convert the pixel coordinates (x, y) to normalized image plane coordinates , the formula is as follows:

[0017] ;

[0018] ;

[0019] in,( u 0, v 0) is the principal point coordinate of the image, f x and f y Camera x The normalized focal length and y Normalized focal length along the axis;

[0020] Based on the inverse perspective projection technology, the depth information d is used, combined with the normalized image plane coordinates and the camera internal and external parameters, to calculate the real coordinates (X, Y, Z) of the pixel points in the 2D segmentation mask in the world coordinate system, and convert the pixel points into 3D points. The formula is as follows:

[0021] ;

[0022] in, R is a 3*3 rotation matrix, t is a 3*1 translation vector;

[0023] Assign the semantic label L to the 3D point corresponding to the pixel in the world coordinate system;

[0024] Get all 3D points within a fixed radius of each Gaussian ellipsoid in the Gaussian splash model to form a set P;

[0025] The confidence of each Gaussian ellipsoid on all semantic categories is calculated according to the semantic weight calculation strategy. The formula is as follows:

[0026] ;

[0027] In the formula, Indicates that the Gaussian ellipsoid belongs to i The confidence of the semantic category marked by the semantic label; p is a 3D point in the set P, Representing 3D points p The distance from the Gaussian ellipsoid, Represents the 3D point p Is it the i semantic tags, O represents the transparency of the Gaussian ellipsoid, |P| represents the number of 3D points in the set P;

[0028] The semantic label with the highest confidence is assigned to the Gaussian ellipsoid; finally, a semantic Gaussian splash model is obtained.

[0029] Furthermore, the method for obtaining all 3D points within a fixed radius of each Gaussian ellipsoid in the Gaussian splash model is a radius nearest neighbor algorithm.

[0030] Furthermore, the specific process of editing the semantic Gaussian splash model and reconstructing the three-dimensional target scene is as follows:

[0031] Select the Gaussian ellipsoid to be edited, and automatically obtain all Gaussian ellipsoids with the same semantics according to the semantic label and highlight them;

[0032] For the area where the Gaussian ellipsoid to be edited is located, a 2D image with a more complete viewing angle of the area is re-collected in the target scene, and the 2D image is re-converted into a Gaussian splash model of the area; based on the nearest point iteration algorithm of the point cloud, the Gaussian splash model of the area is aligned to the position of the Gaussian ellipsoid to be edited and replaced;

[0033] A mesh model is artificially constructed and combined with semantic information to calculate the three-dimensional center coordinates and orientation of the Gaussian ellipsoid to be edited, so that the mesh model is automatically placed at the position of the Gaussian ellipsoid to be edited and replaces the Gaussian ellipsoid to be edited. Finally, a three-dimensional scene that integrates the mesh model and the Gaussian splash model is obtained.

[0034] The present invention also provides a real-life three-dimensional scene reconstruction system, comprising:

[0035] Image acquisition equipment, used to acquire multi-angle 2D images of the target scene;

[0036] Gaussian splash model building module, used to convert 2D images into Gaussian splash models of target scenes;

[0037] Gaussian ellipsoid semantic tracking module, used to assign semantic labels to each Gaussian ellipsoid in the Gaussian splash model to obtain a semantic Gaussian splash model;

[0038] The model editing module is used to edit the semantic Gaussian splash model and reconstruct the three-dimensional target scene.

[0039] The present invention also proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the real-life three-dimensional scene reconstruction method as described above is implemented.

[0040] The present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the real-life three-dimensional scene reconstruction method as described above.

[0041] The beneficial effects of the present invention are:

[0042] (1) The present invention proposes a model editing process for Gaussian splash scenes, which realizes the rapid recognition and precise positioning of specific elements in 3D Gaussian scenes by means of image semantic segmentation, Gaussian semantic tracking, and automated model processing, achieves high-precision selection and deep optimization of editing targets, and significantly improves the accuracy, precision, and efficiency of editing.

[0043] (2) The present invention proposes a Gaussian ellipsoid semantic extraction strategy that combines nearest neighbor with semantic weight distribution, which achieves accurate calculation of the semantic information of each Gaussian ellipsoid and improves the classification accuracy of the Gaussian ellipsoid.

[0044] (3) The present invention proposes an automated addition, deletion, modification and query method for semantic Gaussian models, which realizes the batch and automated replacement of inferior Gaussian models with high-quality Gaussian and grid models, improves the fidelity of the Gaussian model, and enhances the ability to process the details of the Gaussian model.

[0045] (4) The present invention can significantly improve editing accuracy: The present invention integrates image semantic segmentation to ensure that editing targets can be accurately identified and locked in complex 3D Gaussian scenes, greatly enhancing the precision and accuracy of editing.

[0046] (5) The present invention can significantly improve editing efficiency: through automatic recognition and intelligent processing of editing objects, it can effectively shorten the editing time of large-scale or complex scenes, reduce manual intervention, and achieve faster post-production.

[0047] (6) The present invention can promote technology integration and application expansion: The present invention combines advanced technologies such as image processing, computer vision and deep learning to optimize the existing 3D scene editing method, and provide technical support for applications in more fields in the future (such as film and television special effects, game development, virtual reality, etc.), thereby promoting the integration and development of related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flowchart of the real-life three-dimensional scene reconstruction method proposed by the present invention;

[0049] Figure 2 is a schematic diagram of the Gaussian splash model of the test scene;

[0050] Figure 3 It is a schematic diagram of semantic segmentation of the test image;

[0051] Figure 4 This is a schematic diagram of the semantic Gaussian splash model;

[0052] Figure 5 It is a schematic diagram of replacing the Gaussian ellipsoid with a fine mesh model. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] Example 1

[0055] The present invention proposes a method for reconstructing a real three-dimensional scene, the process is as follows: Figure 1 As shown, the following steps are included:

[0056] Step 1: Use a camera to collect multi-angle 2D images of the target scene, and convert the 2D images into a Gaussian splash model of the target scene; the Gaussian splash model is composed of many translucent Gaussian ellipsoids. In this embodiment, the target scene is a desk and office supplies.

[0057] All 2D images are imported into COLMAP software. After a series of steps including image feature extraction, feature matching, sparse reconstruction, and global optimization, the camera pose corresponding to each image and the sparse point cloud of the target scene are obtained.

[0058] Based on the processed 2D image, camera pose and sparse point cloud, the Gaussian splash algorithm is used to generate a Gaussian splash model of the office desk scene, such as Figure 2 As shown in Figure 2, the Gaussian splash model is composed of many semi-transparent Gaussian ellipsoids.

[0059] Step 2, assigning a semantic label to each Gaussian ellipsoid in the Gaussian splash model to obtain a semantic Gaussian splash model; including the following sub-steps:

[0060] Step 2.1: Use the Segment Anything semantic segmentation model to extract semantic information from all 2D images with calculated poses, generate 2D segmentation masks, and record the coordinates of each pixel ( x , y ) and its semantic label L; objects such as water cups, laptops, keyboards, and mice can be extracted from the test scene, such as Figure 3 shown.

[0061] Step 2.2, under the Gaussian scene rendering perspective, obtain the depth information d of each 2D image from the rasterization process.

[0062] Step 2.3, the coordinates of the pixel points ( x , y ) is transformed into normalized image plane coordinates , the formula is as follows:

[0063] ;

[0064] ;

[0065] in,( u 0, v 0) is the principal point coordinate of the image, f x and f y Camera x The normalized focal length and y Normalized focal length in the axis direction;

[0066] Step 2.4, based on the inverse perspective projection technology, using the depth information d, combined with the normalized image plane coordinates and the camera internal and external parameters, calculate the real coordinates (X, Y, Z) of the pixel points in the 2D segmentation mask in the world coordinate system, and convert the pixel points into 3D points. The formula is as follows:

[0067] ;

[0068] in, R is a 3*3 rotation matrix, t is a 3*1 translation vector;

[0069] Assign the semantic label L to the 3D point corresponding to the pixel in the world coordinate system;

[0070] Step 2.5, use the Radius Nearest Neighbor algorithm to obtain all 3D points within a fixed radius of each Gaussian ellipsoid in the Gaussian splash model to form a set P; calculate the confidence of each Gaussian ellipsoid in all semantic categories according to the semantic weight calculation strategy, the formula is as follows:

[0071] ;

[0072] In the formula, Indicates that the Gaussian ellipsoid belongs to i The confidence of the semantic category marked by the semantic label; p is a 3D point in the set P, Representing 3D points p The distance from the Gaussian ellipsoid, Represents the 3D point p Is it the i semantic tags, O represents the transparency of the Gaussian ellipsoid, |P| represents the number of 3D points in the set P;

[0073] The semantic label with the highest confidence is assigned to the Gaussian ellipsoid; finally, the semantic Gaussian splash model is obtained, such as Figure 4 shown.

[0074] Step 3, editing the semantic Gaussian splash model to reconstruct the three-dimensional target scene; including the following sub-steps:

[0075] Step 3.1, select the Gaussian ellipsoid to be edited, automatically obtain all Gaussian ellipsoids with the same semantics according to the semantic label, and highlight them;

[0076] Step 3.2, for the area where the Gaussian ellipsoid to be edited is located, a 2D image with a more complete perspective of the area is re-collected in the target scene, and the 2D image is converted into a Gaussian splash model of the area; based on the nearest point iteration algorithm of the point cloud, the Gaussian splash model of the area is aligned to the position of the Gaussian ellipsoid to be edited and replaced; thereby a high-fidelity Gaussian scene is obtained through post-editing rather than complete retraining; the purpose of this step is to model a certain semantic area separately and replace the original model without rebuilding the entire scene.

[0077] It is not enough to simply replace part of the Gaussian ellipsoid in the Gaussian splash model to be edited with a high-quality Gaussian splash model. In order to adapt to different scenes, a mesh model is needed to replace the Gaussian ellipsoid.

[0078] Step 3.3, manually construct a mesh model, combine semantic information, calculate the 3D center coordinates and orientation of the Gaussian ellipsoid to be edited (such as keyboard, notebook, etc.), so that the mesh model is automatically placed at the location of the Gaussian ellipsoid to be edited, and replaces the Gaussian ellipsoid to be edited, and finally obtains a high-fidelity 3D scene that integrates the mesh model and the Gaussian splash model, such as Figure 5 shown.

[0079] Mesh Model is a computer graphics method for representing three-dimensional objects. It consists of a series of vertices, edges, and faces, which together define the shape and structure of the object. Mesh model is the most commonly used representation in three-dimensional modeling and is widely used in computer-aided design (CAD), computer graphics, animation, game development, virtual reality (VR), and augmented reality (AR).

[0080] Example 2

[0081] The present invention proposes a real-life 3D scene reconstruction system corresponding to the method of embodiment 1, comprising:

[0082] Image acquisition equipment, used to acquire multi-angle 2D images of the target scene;

[0083] Gaussian splash model building module, used to convert 2D images into Gaussian splash models of target scenes;

[0084] Gaussian ellipsoid semantic tracking module, used to assign semantic labels to each Gaussian ellipsoid in the Gaussian splash model to obtain a semantic Gaussian splash model;

[0085] The model editing module is used to edit the semantic Gaussian splash model and reconstruct the three-dimensional target scene.

[0086] The implementation methods of each module and module function in the system are completely consistent with the steps of the method in Example 1, so they will not be repeated here.

[0087] Example 3

[0088] The present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a real-life three-dimensional scene reconstruction method as described in Example 1 is implemented.

[0089] Example 4

[0090] The present invention provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the real-life three-dimensional scene reconstruction method of embodiment 1.

[0091] In the embodiments disclosed in the present application, the computer storage medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. The computer storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specific examples of computer storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0092] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0093] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A method for reconstructing a real three-dimensional scene, characterized in that: The following steps are involved: Collect multi-angle 2D images of the target scene and convert the 2D images into a Gaussian splash model of the target scene; Assigning a semantic label to each Gaussian ellipsoid in the Gaussian splash model to obtain a semantic Gaussian splash model; the specific process of assigning a semantic label to each Gaussian ellipsoid in the Gaussian splash model to obtain a semantic Gaussian splash model includes: Use the Segment Anything semantic segmentation model to extract semantic information from all 2D images, generate a 2D segmentation mask, and record the coordinates (x, y) of each pixel in the 2D segmentation mask and the semantic label L of the pixel; Get the depth information d of each 2D image; Convert the pixel coordinates (x, y) to normalized image plane coordinates , the formula is as follows: ； ； Among them, (u0,v0) is the coordinate of the principal point of the image, f x and f y They are the normalized focal length of the camera in the x-axis direction and the normalized focal length in the y-axis direction respectively; Based on the inverse perspective projection technology, the depth information d is used, combined with the normalized image plane coordinates and the camera internal and external parameters, to calculate the real coordinates (X, Y, Z) of the pixel points in the 2D segmentation mask in the world coordinate system, and convert the pixel points into 3D points. The formula is as follows: ； Among them, R is a 3*3 rotation matrix, and t is a 3*1 translation vector; Assign the semantic label L to the 3D point corresponding to the pixel in the world coordinate system; Obtain all 3D points within a fixed radius of each Gaussian ellipsoid in the Gaussian splash model to form a set P; the method for obtaining all 3D points within a fixed radius of each Gaussian ellipsoid in the Gaussian splash model is a radius nearest neighbor algorithm; The confidence of each Gaussian ellipsoid on all semantic categories is calculated according to the semantic weight calculation strategy. The formula is as follows: ； In the formula, represents the confidence that the Gaussian ellipsoid belongs to the semantic category marked by the i-th semantic label; p is a 3D point in the set P, represents the distance between the 3D point p and the Gaussian ellipsoid, Indicates whether the 3D point p belongs to the i-th semantic label, O Indicates the transparency of the Gaussian ellipsoid, | P | represents a collection P The number of 3D points in ; The semantic label with the highest confidence is assigned to the Gaussian ellipsoid; finally, a semantic Gaussian splash model is obtained; Edit the semantic Gaussian splash model and reconstruct the three-dimensional target scene.

2. The method for reconstructing a real three-dimensional scene according to claim 1, characterized in that: The Gaussian splash model for converting a 2D image into a target scene is specifically: After feature extraction, feature matching, sparse reconstruction and global optimization of all 2D images, the camera pose corresponding to each 2D image and the sparse point cloud of the target scene are obtained; Based on the processed 2D image, camera pose and sparse point cloud, the Gaussian splash algorithm is used to generate a Gaussian splash model of the target scene.

3. The method for reconstructing a real three-dimensional scene according to claim 1, characterized in that: The specific process of editing the semantic Gaussian splash model and reconstructing the three-dimensional target scene is as follows: Select the Gaussian ellipsoid to be edited, and automatically obtain all Gaussian ellipsoids with the same semantics according to the semantic label and highlight them; For the area where the Gaussian ellipsoid to be edited is located, a 2D image with a more complete viewing angle of the area is re-collected in the target scene, and the 2D image is converted into a Gaussian splash model of the area; based on the nearest point iteration algorithm of the point cloud, the Gaussian splash model of the area is aligned to the position of the Gaussian ellipsoid to be edited and replaced; A mesh model is artificially constructed, and the three-dimensional center coordinates and orientation of the Gaussian ellipsoid to be edited are calculated in combination with semantic information, so that the mesh model is automatically placed at the position of the Gaussian ellipsoid to be edited, replacing the Gaussian ellipsoid to be edited, and finally a three-dimensional scene that integrates the mesh model and the Gaussian splash model is obtained.

4. A real-life three-dimensional scene reconstruction system, characterized in that: include: Image acquisition equipment, used to acquire multi-angle 2D images of the target scene; Gaussian splash model building module, used to convert 2D images into Gaussian splash models of target scenes; The Gaussian ellipsoid semantic tracking module is used to assign a semantic label to each Gaussian ellipsoid in the Gaussian splash model to obtain a semantic Gaussian splash model. The specific process of assigning a semantic label to each Gaussian ellipsoid in the Gaussian splash model to obtain a semantic Gaussian splash model includes: Use the Segment Anything semantic segmentation model to extract semantic information from all 2D images, generate a 2D segmentation mask, and record the coordinates (x, y) of each pixel in the 2D segmentation mask and the semantic label L of the pixel; Get the depth information d of each 2D image; Convert the pixel coordinates (x, y) to normalized image plane coordinates , the formula is as follows: ； ； Among them, (u0,v0) is the coordinate of the principal point of the image, f x and f y They are the normalized focal length of the camera in the x-axis direction and the normalized focal length in the y-axis direction respectively; Based on the inverse perspective projection technology, the depth information d is used, combined with the normalized image plane coordinates and the camera internal and external parameters, to calculate the real coordinates (X, Y, Z) of the pixel points in the 2D segmentation mask in the world coordinate system, and convert the pixel points into 3D points. The formula is as follows: ； Among them, R is a 3*3 rotation matrix, and t is a 3*1 translation vector; Assign the semantic label L to the 3D point corresponding to the pixel in the world coordinate system; Obtain all 3D points within a fixed radius of each Gaussian ellipsoid in the Gaussian splash model to form a set P; the method for obtaining all 3D points within a fixed radius of each Gaussian ellipsoid in the Gaussian splash model is a radius nearest neighbor algorithm; The confidence of each Gaussian ellipsoid on all semantic categories is calculated according to the semantic weight calculation strategy. The formula is as follows: ； In the formula, represents the confidence that the Gaussian ellipsoid belongs to the semantic category marked by the i-th semantic label; p is a 3D point in the set P, represents the distance between the 3D point p and the Gaussian ellipsoid, Indicates whether the 3D point p belongs to the i-th semantic label, O Indicates the transparency of the Gaussian ellipsoid, | P | represents a collection P The number of 3D points in ; The semantic label with the highest confidence is assigned to the Gaussian ellipsoid; finally, a semantic Gaussian splash model is obtained; The model editing module is used to edit the semantic Gaussian splash model and reconstruct the three-dimensional target scene.

5. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for reconstructing a real three-dimensional scene as described in any one of claims 1 to 3 is implemented.

6. A computer-readable storage medium storing a computer program, characterized in that: The computer program enables a computer to execute the real-life three-dimensional scene reconstruction method as described in any one of claims 1 to 3.

Images