Three-dimensional scene reconstruction method and device

Abstract

The invention discloses a three-dimensional scene reconstruction method and device, and belongs to the field of computer vision and image processing. The method comprises: determining spatial geometric information of a shooting scene according to M two-dimensional images, the M two-dimensional images being images of the shooting scene at different shooting angles, the spatial geometric information at least comprising: at least one target spatial point and at least one spatial line segment, M being an integer greater than 1; and according to the spatial geometric information, a three-dimensional Gaussian ellipsoid set is generated, and the three-dimensional Gaussian ellipsoid set comprises at least one three-dimensional Gaussian ellipsoid used for representing three-dimensional scene information of the shooting scene.

Description

Technical Field

[0001] The present application belongs to the field of computer vision and image processing, and specifically relates to a three-dimensional scene reconstruction method and device. Background Art

[0002] With the development of 3D reconstruction technology, the demand for accurate reconstruction of scenes is growing.

[0003] In the related art, the scene is usually reconstructed using the position of spatial points. Since a large number of spatial points need to be used for fitting when reconstructing the scene, data redundancy occurs. Therefore, the current scene reconstruction method has the problem of data redundancy in initialization, especially when reconstructing complex scenes, a large number of three-dimensional Gaussian ellipsoids need to be generated. This results in a large amount of data in the three-dimensional scene reconstruction, resulting in low reconstruction efficiency. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a three-dimensional scene reconstruction method and device thereof, which can reduce the amount of data in the scene reconstruction process, thereby improving the efficiency of three-dimensional scene reconstruction.

[0005] In a first aspect, an embodiment of the present application provides a three-dimensional scene reconstruction method, the method comprising: determining spatial geometric information of a shooting scene based on M two-dimensional images, the M two-dimensional images being images of the shooting scene at different shooting angles, the spatial geometric information comprising at least: at least one target space point and at least one space line segment, M being an integer greater than 1; generating a three-dimensional Gaussian ellipsoid set based on the spatial geometric information, the three-dimensional Gaussian ellipsoid set comprising at least one three-dimensional Gaussian ellipsoid, for characterizing the three-dimensional scene information of the shooting scene.

[0006] In a second aspect, an embodiment of the present application provides a three-dimensional scene reconstruction device, which includes: a processing module; the processing module is used to determine the spatial geometric information of a shooting scene based on M two-dimensional images, the M two-dimensional images are images of the shooting scene at different shooting angles, and the spatial geometric information includes at least: at least one target space point and at least one space line segment, M is an integer greater than 1; the processing module is also used to generate a three-dimensional Gaussian ellipsoid set based on the spatial geometric information, the three-dimensional Gaussian ellipsoid set includes at least one three-dimensional Gaussian ellipsoid, and the three-dimensional Gaussian ellipsoid set is used to characterize the three-dimensional scene information of the shooting scene.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0010] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0011] In an embodiment of the present application, the electronic device determines the spatial geometric information of the shooting scene based on M two-dimensional images, wherein the M two-dimensional images are images of the shooting scene at different shooting angles, and the spatial geometric information at least includes: at least one target space point and at least one space line segment, and M is an integer greater than 1; based on the spatial geometric information, a three-dimensional Gaussian ellipsoid set is generated, wherein the three-dimensional Gaussian ellipsoid set includes at least one three-dimensional Gaussian ellipsoid for characterizing the three-dimensional scene information of the shooting scene. Through this method, the electronic device can generate a three-dimensional Gaussian ellipsoid set in combination with the space points and the space line segments, and realize the use of a smaller number of three-dimensional Gaussian ellipsoids to characterize the scene space of a specific area, that is, directly generating a three-dimensional Gaussian ellipsoid from the space line segments, thereby reducing the amount of data in the three-dimensional scene reconstruction, and thus improving the reconstruction efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a flow chart of a three-dimensional scene reconstruction method provided in an embodiment of the present application;

[0013] Figure 2 It is a schematic diagram of an actual photo of a scene, a spatial line segment, and a sparse point cloud provided in an embodiment of the present application;

[0014] Figure 3 It is a schematic diagram of an actual photo of a scene, a depth map, and a pseudo-color display of the depth map provided in an embodiment of the present application;

[0015] Figure 4 This is a schematic diagram of initializing a Gaussian ellipsoid based on a space line segment provided in an embodiment of the present application;

[0016] Figure 5 It is a schematic diagram of the relationship between the plane straight line width and the space straight line width provided in the embodiment of the present application;

[0017] Figure 6 This is a schematic diagram of the center point of a grid in a grid space of an SDF with a resolution of 7x7x7 provided in an embodiment of the present application;

[0018] Figure 7 It is a schematic diagram of space segmentation in a grid space of an SDF with a resolution of 7x7x7 provided in an embodiment of the present application;

[0019] Figure 8 is a schematic diagram of calculating the distance from a grid point to an object surface provided in an embodiment of the present application;

[0020] Fig. 9 It is a schematic diagram of a space cube, a surface and a normal vector represented by a distance field SDF provided in an embodiment of the present application;

[0021] Fig.10 is a schematic diagram of spatial information inside a location where a grid is recorded provided in an embodiment of the present application;

[0022] Fig.11 is a schematic diagram of a grid and a space segment provided in an embodiment of the present application;

[0023] Fig.12 This is a schematic diagram of judging the coplanarity of two straight lines in space provided by an embodiment of the present application;

[0024] Fig.13 A schematic diagram of the structure of a three-dimensional scene reconstruction device provided in an embodiment of the present application;

[0025] Fig.14 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0026] Fig.15 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0028] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0029] The terms "at least one (item)", "at least one of" and the like in the specification and claims of the present application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".

[0030] The three-dimensional scene reconstruction method provided in the embodiment of the present application can be applied to a scenario in which a group of photos containing the same shooting scene are three-dimensionally reconstructed by an electronic device.

[0031] When a group of photographs of a scene including a bookshelf, potted plants on the bookshelf, a background wall and a floor are reconstructed in three dimensions by an electronic device, the SFM (Structure From Motion, SFM) technology is first used to obtain the sparse point cloud expression of the scene, the camera position and the camera attitude. Next, the camera position and the camera attitude are fixed, a spatial coordinate system is established, and the multi-view stereo vision technology (Multi-View Stereo, MVS) is used to obtain the dense point cloud expression of the corresponding shooting scene of the group of photos based on the previously obtained sparse point cloud expression. Then, the spatial geometric information of the shooting scene is determined by the dense point cloud, such as the three-dimensional structure of the bookshelf can be extracted into spatial segments, and the potted plants on the bookshelf can be extracted into spatial points. Based on the spatial points and spatial segments, a set of three-dimensional Gaussian ellipsoids can be generated to characterize the three-dimensional scene information in the above-mentioned shooting scene. In this way, electronic devices can use a smaller number of three-dimensional Gaussian ellipsoids to represent the scene space of a specific area, such as using spatial line segments to generate a few flattened three-dimensional Gaussian ellipsoids to fit the outline and shelf structure of a bookshelf, without the need to generate a high-density set of three-dimensional Gaussian ellipsoids based on a dense point cloud, thereby reducing the number of three-dimensional Gaussian ellipsoids, optimizing the amount of data in three-dimensional scene reconstruction, and thereby improving reconstruction efficiency.

[0032] The execution subject of the 3D scene reconstruction method provided in the embodiment of the present application may be a 3D scene reconstruction device, and the 3D scene reconstruction device may be an electronic device, or a functional module or functional entity in an electronic device. The following takes the execution of the 3D scene reconstruction method by an electronic device as an example to illustrate the 3D scene reconstruction method provided in the embodiment of the present application.

[0033] The following is a detailed description of the three-dimensional scene reconstruction provided by the embodiments of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0034] Figure 1 A schematic diagram of a process for reconstructing a three-dimensional scene provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the video recording method may include the following steps 201 to 202:

[0035] Step 201: The electronic device determines spatial geometric information of a shooting scene according to M two-dimensional images.

[0036] The M two-dimensional images are images of the shooting scene at different shooting angles, the spatial geometric information includes at least: at least one target space point and at least one space line segment, and M is an integer greater than 1.

[0037] In some embodiments of the present application, the M two-dimensional images may be photos of the same scene taken at different angles using the same camera, or may be photos of the same scene taken at different angles using different cameras.

[0038] In some embodiments of the present application, the shooting scene may be an indoor scene or an outdoor scene containing a plurality of objects with complex geometric structures.

[0039] For example, a group of scenes including bookshelves, potted plants on the bookshelves, background walls, and floors are photographed at different angles using the same camera. For example, the overall structure of the bookshelf is photographed from the front to capture the outline of the bookshelf and the distribution of the shelves, as well as the approximate position of the potted plants on the bookshelf. The depth information of the bookshelf and the specific thickness of the shelves are obtained from the side, while showing the height of the potted plants and the texture details of the background wall.

[0040] As another example, a group of photographs of a scene including a bookshelf, potted plants on the bookshelf, a background wall, and a floor are taken at different angles using different cameras. For example, a high-resolution camera is used to shoot from a 45-degree angle to obtain a clear image of the items placed on each layer of the bookshelf, including the arrangement of books and the morphological characteristics of potted plants. A wide-angle camera is used to shoot from the top downward to obtain a macroscopic perspective of the entire scene, including the pattern of the floor and the layout of the entire room.

[0041] In some embodiments of the present application, the above-mentioned dense point cloud can be obtained by firstly subjecting the above-mentioned M two-dimensional images to the SFM technology to obtain a sparse point cloud expression of the scene, the camera position, and the camera posture, and then, according to the position and posture of the camera, establishing a spatial coordinate system, and using multi-view stereo vision MVS technology to restore the spatial position and depth information of each object in the scene relative to the camera according to the sparse point cloud expression, thereby obtaining a dense point cloud expression of the captured scene.

[0042] Specifically, the electronic device can use the 3D reconstruction technology (SFM) to reconstruct the camera lens parameters, the spatial position and posture of the camera, and the sparse structure of the scene for a set of photos, such as Figure 2 As shown in Figure 2, the feature points and feature lines extracted from the photo surface by 3D reconstruction technology can restore the sparse point cloud and spatial line segments in the scene.

[0043] Furthermore, after obtaining the lens parameters of the camera, the electronic device dedistorts the photo based on the lens parameters to obtain a photo without lens distortion; after obtaining the camera position and posture parameters, the electronic device can use the multi-view stereo vision technology MVS to reconstruct the depth map of each photo and the dense point cloud expression of the scene, such as Figure 3 As shown, the value of each pixel in the depth map of the photo is the distance from the observed object to the center of the lens.

[0044] The following is an exemplary description of the process of the electronic device reconstructing the depth map of each photo and the dense point cloud expression of the scene in combination with the formula.

[0045] Assume that a 3D point X is X = [X, Y, Z, 1]. Through the projection equation of the camera, as shown in the following formula 1, the corresponding 2D point x on the image plane can be obtained as x = [x, y, 1].

[0046]

[0047] Where K is the camera's intrinsic parameter matrix, as shown in Formula 2:

[0048]

[0049] Figure 2 (A) in the figure represents an actual photo of the scene. Figure 2 (B) in the figure is the sparse point cloud of the scene reconstructed by SFM; Figure 2 (C) in the figure is the spatial line segment reconstructed from the scene; Figure 2 (D) in the figure is a sparse representation of space, where red points are sparse point clouds and blue points are spatial line segments.

[0050] Figure 3 (A) in the figure indicates the actual photo of the scene; Figure 3(B) in the figure is the depth map of the example photo calculated by MVS, and the dark color represents the far distance; Figure 3 (C) in the figure is a pseudo-color display of the depth map, where dark colors represent far distances.

[0051] In some embodiments of the present application, the above-mentioned spatial points may be key feature points in the scene, such as corner points, edge points or significant texture points of an object.

[0052] For example, corner points of buildings or leaf edges of potted plants are used to determine spatial points of the three-dimensional contours of objects in the scene.

[0053] In some embodiments of the present application, the above-mentioned spatial line segment may be a straight line segment connecting two spatial points in a scene, which is used to represent the edge of an object or the connection relationship between objects.

[0054] For example, the long straight lines existing in the edge of the bookshelf outline, the edge lines of each layer in the bookshelf, the intersection lines of the wall and the floor or ceiling, or the gaps between the floors, etc., have clear line characteristics.

[0055] Step 202: The electronic device generates a three-dimensional Gaussian ellipsoid set according to the spatial geometric information.

[0056] The three-dimensional Gaussian ellipsoid set includes at least one three-dimensional Gaussian ellipsoid, and the three-dimensional Gaussian ellipsoid set is used to represent the three-dimensional scene information of the shooting scene.

[0057] In some embodiments of the present application, the three-dimensional Gaussian ellipsoid is a geometric entity with specific parameters, including the center coordinates of the ellipsoid, the dimensions along each axis (semi-axis length), and a rotation matrix describing its spatial direction.

[0058] In some embodiments of the present application, the three-dimensional Gaussian ellipsoid set is a group of multiple ellipsoids that can fully cover and accurately describe complex scenes, and each ellipsoid corresponds to a specific area or feature in the scene and is used to characterize the shooting scene.

[0059] In some embodiments of the present application, the above-mentioned three-dimensional scene information is detailed geometric and spatial data of the actual shooting scene represented by these Gaussian ellipsoid sets, including but not limited to the shape, size, relative position of objects in the actual shooting scene and the global structure of the shooting scene.

[0060] For example, suppose that a shooting scene contained in a set of photos needs to be restored, and the shooting scene includes a bookshelf, potted plants on the bookshelf, a background wall, and a floor. First, based on this set of photos, a dense point cloud of the shooting scene is reconstructed by using the SFM technology and the MVS technology. Then, all target space points and space segments in the shooting scene are determined based on the dense point cloud of the shooting scene, such as determining the outline of the bookshelf, the trunk of the plant in the potted plant, the edge of the wall of the background wall, and the gap in the floor as space segments; and determining other complex set-shaped objects in the shooting scene as target space points. Finally, a three-dimensional Gaussian ellipsoid set is generated based on the space segments and the target space points. Since a slender three-dimensional Gaussian ellipsoid is generated by using the space segments, a linear object, such as the outline of a bookshelf, is fitted; for a plane area, a flat three-dimensional Gaussian ellipsoid is generated to fit the plane area, so that there is no need to generate a high-density three-dimensional Gaussian ellipsoid set based on the dense point cloud, which reduces the number of three-dimensional Gaussian ellipsoids, optimizes the amount of data in the three-dimensional scene reconstruction, and thus improves the reconstruction efficiency.

[0061] In some embodiments of the present application, the above step 201 may include the following steps 201a and 201b:

[0062] Step 201a: The electronic device performs line segment detection on the M two-dimensional images to determine the spatial line segments in the shooting scene.

[0063] In some embodiments of the present application, the above-mentioned line segment detection can be performed by performing line segment detection and matching on a two-dimensional image during the SFM technology stage, and then generating spatial three-dimensional line segments based on line segment pairs in the two-dimensional image with a reasonable spatial structure.

[0064] Exemplarily, for the joints of floor tiles or the edges of carpets, image line segment detection is used to perform line segment detection and matching on multiple two-dimensional photos corresponding to the scene, and then a descriptor of the image line segment is generated based on the grayscale value distribution of the image. Then, the similarity of the images is determined by the vector geometric distance of the descriptor. The electronic device determines the line segment matching pairs with high similarity as a group of potential matching pairs, and each group of potential matching pairs corresponds to a potential spatial line segment. For example, for the floor, through the image line segment detection technology, the electronic device determines the joints of the floor as a group of potential matching pairs, corresponding to a potential spatial line segment.

[0065] Then, all the space segments are determined by searching all the line segments. For example, the edge of each layer of the bookshelf will be identified as a space segment, the trunk of the potted plant will be identified as a space segment, the decorative lines on the background wall will be identified as a separate space segment, and the seams on the floor will be identified as space segments.

[0066] Step 201b: The electronic device determines at least one target space point based on the target feature points in the dense point cloud of the shooting scene.

[0067] Among them, the target feature points are feature points in the dense point cloud except for the feature points corresponding to the space line segments.

[0068] In some embodiments of the present application, the above-mentioned feature points may be intersection points of edges of objects.

[0069] Exemplarily, the feature points may be points at the corners of a building or the corners of furniture in the shooting scene.

[0070] In some embodiments of the present application, the above-mentioned feature points may be edge points, that is, points on the edge of an object.

[0071] Exemplarily, the feature points may be points on a frame of a window or an edge of a table in the shooting scene.

[0072] In some embodiments of the present application, the above-mentioned feature points may be interest points, ie, points in the scene having unique texture or color changes.

[0073] For example, such as the center of a logo or pattern.

[0074] In some embodiments of the present application, the electronic device may determine other feature points in the dense point cloud except for the feature points corresponding to the space line segments as target feature points.

[0075] Exemplarily, in combination with the above example, when the electronic device reconstructs a photographed scene including a bookshelf, a potted plant on the bookshelf, a background wall and a floor, for points in the photographed scene that are not fitted as spatial line segments, for example, for a bookshelf, the target feature points may be feature points corresponding to certain irregular ornaments on the bookshelf; for a potted plant, the target feature points may be feature points corresponding to the edges of leaves or flowers with complex shapes; for the background wall, the target feature points may be feature points corresponding to the depicted content on the painting; and for the floor, the target feature points may be feature points corresponding to the complex patterns on the floor.

[0076] In this way, by extracting the spatial line segments containing the line segment feature areas, the number of spatial points in these areas can be effectively reduced, thereby solving the problem of data redundancy and improving the reconstruction efficiency.

[0077] In some embodiments of the present application, the above step 202 may include the following steps 202a and 202b:

[0078] Step 202a: The electronic device determines target parameters of at least one three-dimensional Gaussian ellipsoid based on the spatial geometric information.

[0079] The target parameters may include at least one of the following: the center position of the three-dimensional Gaussian ellipsoid, the scale of the three-dimensional Gaussian ellipsoid, and the rotation direction of the three-dimensional Gaussian ellipsoid.

[0080] Step 202b: The electronic device generates at least one three-dimensional Gaussian ellipsoid according to the spatial geometric information.

[0081] In some embodiments of the present application, the center position of the three-dimensional Gaussian ellipsoid may be the coordinates of the ellipsoid in three-dimensional space, which may be represented as (x, y, z) in a spatial rectangular coordinate system. The center position determines the "position" of the ellipsoid in space, i.e., the geometric center of the ellipsoid.

[0082] In some embodiments of the present application, the scale of the three-dimensional Gaussian ellipsoid may be the "size" or "radius" of the three-dimensional Gaussian ellipsoid along its three main axes. For example, in a standard ellipsoid, these scales may be represented by semi-axis lengths s1, s2, s3, corresponding to the semi-axis of the ellipsoid along the X-axis, Y-axis, and Z-axis in the spatial rectangular coordinate system, respectively. The above scale parameters determine the shape and size of the ellipsoid, wherein the scale of each axis may be different, so that the ellipsoid is stretched or compressed in different directions to form an elliptical shape.

[0083] In some embodiments of the present application, the rotation direction of the three-dimensional Gaussian ellipsoid may be the orientation direction of the ellipsoid relative to a spatial rectangular coordinate system.

[0084] It should be noted that since the three-dimensional Gaussian ellipsoid is elliptical, its three main axes are not necessarily aligned with the axes of the coordinate system. The rotation direction is described by the rotation matrix R, which contains the rotation information of the ellipsoid relative to the coordinate system. The rotation matrix ensures that the ellipsoid can be correctly aligned with other geometric shapes in space.

[0085] For example, as mentioned above Figure 2 As shown, electronic equipment Figure 2 The scene in (A) is reconstructed, and the target parameters of the three-dimensional Gaussian ellipsoid are determined according to the reconstructed spatial points and spatial line segments.

[0086] Specifically, the parameters of a three-dimensional Gaussian ellipsoid include the coordinates of the center point of the ellipsoid, the dimensions of the three axes of the ellipsoid xyz, and the rotation direction of the ellipsoid. In the phase plane projection formula of the Gaussian ellipsoid, the phase difference matrix needs to be composed of the ellipsoid size matrix and the rotation matrix, as shown in Formula 3.

[0087] S is a 3x3 diagonal matrix, as shown in Formula 4, with the diagonal dimensions of the three axes xyz. The R matrix is ​​the rotation matrix of the Gaussian ellipsoid, which has three unknowns. A Gaussian ellipsoid has the center point coordinate X, the dimensions of the three axes S, and a spatial rotation R, for a total of 9 unknowns.

[0088] ∑=RSS T R T (3)

[0089]

[0090] It should be noted that the above formula (3) is a multiplication matrix composed of the ellipsoid size matrix and the rotation matrix, and the above formula (4) is a 3x3 diagonal matrix.

[0091] Specifically, the center position of the ellipsoid is estimated by the center points of the two endpoints. The center position of the ellipsoid can be calculated by formula 5. Figure 4 As shown in (A) in .

[0092]

[0093] Among them, P1 and P2 represent the two endpoints of the space segment respectively, and Pc represents the midpoint of the space segment, that is, the center point of the three-dimensional Gaussian ellipsoid.

[0094] Specifically, the rotation matrix R is given an initial value by the direction of the line segment. In the XYZ three-axis coordinate system, find the coordinate axis with the smallest angle with the direction of the line segment, and record the angle as θ. Figure 4 As shown in (B) in FIG. 1 , the electronic device determines the smallest absolute value among [θx, θy, θz] as θ. Then, the electronic device can calculate the normal vector of the coordinate axis and the line segment direction, which can be expressed as n = [n x , n y , n z ],like Figure 4 Finally, the electronic device can obtain the rotation matrix R based on the Rodrigues rotation matrix formula (6) using the above parameters.

[0095] R=I+(sinθ)N+(1-cosθ) (6)

[0096]

[0097] Among them, N is a skew-symmetric matrix, which is associated with the unit vector on the rotation axis, and is a skew-symmetric matrix, which is associated with the unit vector on the rotation axis; I is the unit matrix; n x is the component of vector n in the x-axis direction, n y is the component of vector n in the y-axis direction, n z is the component of vector n in the z-axis direction.

[0098] Specifically, the three dimension information of the diagonal matrix S is initialized by the length of the line segment. The value of the longest axis is equal to the length of the line segment. The coordinate axis corresponding to the longest axis is the corresponding spatial axis of the rotation angle θ calculated in the previous step. That is, if the angle between the line segment and the X axis is the smallest, then Sx is the maximum axis length, and the line segment length is assigned to it. The other two short axes use the same initial value W, and the value is given by the width of the line segment on the photo and the projection distance, see formula (7), as Figure 5 As shown in formula (7), the distances from the camera center point to the plane line and the space line are d and D respectively. The widths of the plane line and the space line are w and W respectively.

[0099]

[0100] In some embodiments of the present application, the electronic device may initialize a Gaussian kernel according to spatial geometric information to generate at least one three-dimensional Gaussian ellipsoid.

[0101] Illustratively, in the above example, after processing in step 202a, the electronic device initializes the Gaussian kernel through space segments and space points to generate at least one three-dimensional Gaussian ellipsoid.

[0102] In some examples, for a three-dimensional Gaussian ellipsoid at a spatial line segment, the electronic device may generate a corresponding three-dimensional Gaussian ellipsoid according to the center position of the three-dimensional Gaussian ellipsoid, the scale of the three-dimensional Gaussian ellipsoid, and the rotation direction of the three-dimensional Gaussian ellipsoid.

[0103] In other examples, for the three-dimensional Gaussian ellipsoid at the target space point, its rotation matrix R is initialized by the normal vector n of the plane recorded by the grid. The calculation process is the same as the line segment initialization method. Find the corresponding coordinate axis and the normal vector (θ, n), and use formula (6) to calculate the rotation matrix. And initialize the scale s corresponding to θ to 0. In the area where the plane is detected in space, a flat ellipsoid is generated, and the direction quantity is the same as the plane normal vector, and the remaining dimensions are uniformly given a threshold T that is small enough. small , as shown in formula (8).

[0104]

[0105] It should be noted that the initialization process of the remaining spatial points to the spatial ellipsoid can refer to the traditional method, for example, the R matrix is ​​set to the unit matrix, and the three S values ​​are all set to T small , as shown in formula (9).

[0106]

[0107] In this way, by directly using spatial line segments to initialize the Gaussian kernel, not only can the linear structures in the scene, such as the edges of bookshelves and the intersection of the floor and ceiling, be represented more accurately, but the number of three-dimensional Gaussian ellipsoids generated can also be reduced, solving the data redundancy problem caused by the need for a large number of points to approximate the line segments in traditional point cloud initialization methods. This optimizes the amount of data in three-dimensional scene reconstruction and improves reconstruction efficiency.

[0108] In some embodiments of the present application, the target parameter includes the center position of the three-dimensional Gaussian ellipsoid; the above step 202a may include the following steps 202a1 and 201a2:

[0109] Step 202a1: the electronic device obtains endpoint information of at least one spatial line segment.

[0110] Step 202a2: The electronic device determines the center position of at least one three-dimensional Gaussian ellipsoid based on the endpoint information.

[0111] In some embodiments of the present application, the endpoint information of the above-mentioned space line segment may be information such as the coordinate position, connection relationship, etc. of the endpoint of the space line segment after processing in step 201a.

[0112] Exemplarily, the precise coordinates of an endpoint in three-dimensional space are expressed as (x, y, z).

[0113] Exemplarily, the electronic device may determine the center position parameters of the three-dimensional Gaussian ellipsoid according to the endpoint information of the reconstructed space line segment.

[0114] It should be noted that the process of determining the center position of the three-dimensional Gaussian ellipsoid can refer to the relevant description of the above embodiment, which will not be repeated here.

[0115] In this way, the initialization center coordinates of the three-dimensional Gaussian ellipsoid can be determined by the spatial line segment, which solves the data redundancy problem caused by the need for a large number of points as the corresponding three-dimensional Gaussian ellipsoid center coordinates in the prior art, thereby improving the reconstruction efficiency.

[0116] In some embodiments of the present application, the target parameter includes the size of the three-dimensional Gaussian ellipsoid; the above step 202a may include the following steps 202a3 and 202a4:

[0117] Step 202a3: The electronic device obtains the length of at least one spatial line segment.

[0118] Step 202a4: Determine the length of at least one space line segment as the length of the longest axis of at least one three-dimensional Gaussian ellipsoid.

[0119] In some embodiments of the present application, the length of the above-mentioned space line segment may be the length between two endpoints of the space line segment.

[0120] For example, assuming that the coordinates of the two endpoints a and b of a space line segment in the space rectangular coordinate system are a(0,0,0) and b(3,4,0), respectively, the length of the space line segment can be calculated by formula (10) to be 5. Formula (10) is as follows.

[0121]

[0122] In some embodiments of the present application, the longest axis of the three-dimensional Gaussian ellipsoid may be the longest axis among the three main axes of the three-dimensional Gaussian ellipsoid.

[0123] Exemplarily, as in the example of step 202a, the three size information of the diagonal S matrix is ​​initialized by the length of the line segment, wherein the value of the longest axis is equal to the length of the line segment.

[0124] For example, assume that for a space line segment, the coordinates of its two endpoints a and b are a(0,0,0) and b(3,4,0) respectively. According to formula 10, the electronic device can calculate that the length of the line segment ab is 5 unit lengths. Then, the length of this space line segment, that is, 5 unit lengths, is set as the length of the longest axis of the three-dimensional Gaussian ellipsoid corresponding to the space line segment.

[0125] In this way, by determining the length of the spatial line segment as the longest axis length of its corresponding three-dimensional Gaussian ellipsoid, the surface of objects with obvious line segment features in the shooting scene, such as the edge of a bookshelf, the intersection of the floor and the ceiling, etc., can be simply and efficiently represented, thereby optimizing the amount of data in the three-dimensional scene reconstruction and improving the reconstruction efficiency.

[0126] In some embodiments of the present application, the target parameter includes the rotation direction of the three-dimensional Gaussian ellipsoid; the above step 202a may include the following steps 202a5 and 201a6:

[0127] Step 202a5: The electronic device determines angle information corresponding to at least one spatial line segment.

[0128] The angle information includes: a first angle between at least one space line segment and the X-axis, a second angle between at least one space line segment and the Y-axis, and a third angle between at least one space line segment and the Z-axis.

[0129] Step 202a6: Determine the normal vector of the space line segment.

[0130] Step 202a7: Determine the rotation matrix of the three-dimensional Gaussian ellipsoid based on the angle information and the normal vector.

[0131] The rotation matrix is ​​used to characterize the rotation direction of the three-dimensional Gaussian ellipsoid.

[0132] Exemplarily, the electronic device may determine the rotation matrix R using the Rodrigues formula according to the direction of the space segment.

[0133] In some embodiments of the present application, the normal vector of the above-mentioned space line segment may be a vector perpendicular to the plane where the space line segment is located.

[0134] For example, if the coordinates of the two endpoints A and B of the space segment AB in the space rectangular coordinate system are A(1,2,3) and B(4,6,7) respectively, then the direction vector of the line segment AB can be calculated as (3,4,4). Then, a vector that is not collinear with the line segment AB is selected to calculate the normal vector. Usually, a unit vector on the coordinate axis is selected, such as the unit vector (0,1,0) along the Y axis. By calculating the cross product of these two vectors, a normal vector (-4,0,3) for the line segment AB can be obtained. For another example, suppose that the vector of the line segment AB is represented by n, and the vector n has a corresponding skew-symmetric matrix N, as shown in formula (6). The vectors in each row of N are vectors perpendicular to the vector n.

[0135] In some embodiments of the present application, the rotation matrix of the three-dimensional Gaussian ellipsoid can be a 3x3 orthogonal matrix, which can be used to describe the rotation angle of the three-dimensional Gaussian ellipsoid around the coordinate axis. The rotation matrix can transform a vector from one direction to another without changing its length (i.e., keeping the size of the vector unchanged). Its main purpose is to determine the spatial orientation of the three-dimensional Gaussian ellipsoid.

[0136] It should be noted that the process of determining the rotation matrix R can be referred to in the description of the above embodiment and will not be repeated here.

[0137] In this way, by determining the rotation direction of the corresponding three-dimensional Gaussian ellipsoid according to the angle information and normal vector of the spatial line segment, it can be ensured that the three-dimensional Gaussian ellipsoid is aligned with the line segment or object in the actual shooting scene, so that a small number of three-dimensional Gaussian ellipsoids can also represent complex geometric shapes, and fewer data points can be used to describe the scene, thereby simplifying the amount of data in the three-dimensional scene reconstruction and improving the reconstruction efficiency.

[0138] In some embodiments of the present application, after the above step 202, the 3D scene reconstruction method provided by the embodiment of the present application may further include the following steps 203 and 204:

[0139] Step 203: The electronic device obtains spatial feature information of the shooting scene.

[0140] Step 204: The electronic device performs a target operation on the three-dimensional Gaussian ellipsoid set according to the spatial feature information.

[0141] In some embodiments of the present application, the above-mentioned spatial feature information includes at least one of the following:

[0142] Normal vector information of the plane in the shooting scene;

[0143] Information about spatial line segments in the captured scene;

[0144] Among them, the above-mentioned spatial line segments include long line segments and short line segments. The long line segments are spatial line segments in the above-mentioned shooting scene whose length is greater than or equal to the length threshold, and the short line segments are spatial line segments in the above-mentioned shooting scene whose length is less than the length threshold.

[0145] In some embodiments of the present application, the above-mentioned spatial feature information may be information extracted from a geometric space corresponding to a shooting scene and used to optimize parameters such as the shape, position, scale or rotation direction of a three-dimensional Gaussian ellipsoid.

[0146] In some embodiments of the present application, the normal vector information of the above-mentioned plane may be information of normal vectors corresponding to relatively flat planes extracted from the surfaces of all objects in the shooting scene.

[0147] In some embodiments of the present application, the electronic device may establish a distance field corresponding to the shooting scene, and determine a plane in the shooting scene according to the distance field.

[0148] For example, the electronic device may set a spatial grid with a specific resolution and a size covering the scene at the reconstructed scene, such as Figure 6 and Figure 7 shown. Figure 6 Shown is the center point of the space grid, Figure 7 A plane that divides space and is perpendicular to the XYZ axes.

[0149] It should be noted that Figure 6 Represents a grid space of SDF with a resolution of 7x7x7 and the center point of the grid; Figure 7 Represents a grid space of SDF with a resolution of 7x7x7.

[0150] Furthermore, each point in the grid is projected onto the nearest photos where it can be observed. Here, being observable means that, using Formula 1, the projection point of the grid point onto the photo falls within the effective field of view.

[0151] Figure 8 The distance from the object surface to the camera center and the distance from the grid point to the camera center are shown. Figure 8As shown, the distance from the grid point to the object surface is the difference between the spatial distance and the depth map value of the projection point. j To the camera center c i The geometric distance is denoted as discp and is calculated by formula 11.

[0152]

[0153] Specifically, project the spatial point P(x, y, z) to the photo coordinates (x, y). If the distance from the grid point to the camera projection center is less than the depth map value of the projection point, then the grid point is outside the object; conversely, if the distance is greater than the depth map value of the projection point, then the point is inside the object. In formula 12, (x, y) is the grid point p j The projected point coordinates on the photo and depth map, d i (x, y) is the depth value of the projection point in the depth map. The distance from the grid point to the surface is the average of the distances from the grid point to the plane obtained by the nearest cameras that can observe it, as shown in Formula 13.

[0154]

[0155] It should be noted that the distance recorded by the grid point outside the object is a positive value; on the contrary, if the grid point is located inside the object, the opposite number of the distance recorded is a negative distance representing the inside of the object. The embodiment of the present application is to estimate the surface position of the scene object, and the very deep inside the object can be ignored. The threshold for the negative value of the grid point is set to the size of two grids. That is, if a grid point is deep inside the object and exceeds the set threshold, it will not be recorded in the distance.

[0156] Furthermore, by calculating the distance value from each grid point to the surface of the object, a distance field (SDF) of the space where the reconstructed scene is located is established, and then a zero level set surface extraction algorithm based on SDF, such as marching cube, is used to extract the surface of the figure in the scene, that is, the zero level set surface, and the normal vector is calculated for the surface.

[0157] It should be noted that the zero level set surface, that is, the set of surfaces with a distance to the object of 0, is the object surface. The meshes with detected surfaces are marked as meshes with surfaces. Fig. 9 Represents the space cube, surface and normal vectors represented by the distance field SDF. Fig. 9 (A) in the figure represents a grid point of an SDF. The color of the grid point represents the value, red for positive values ​​and blue for negative values. Fig. 9 (B) in the figure represents the object surface extracted from the SDF, that is, the surface with zero distance. Here, a cube is taken as an example. Fig. 9In (C), the blue surface is the zero-distance surface, and the red line segment is the normal vector on the surface.

[0158] For example, the gradient of each sampling point of the distance field can be calculated by difference calculation using formula (14). The gradient direction is the direction in which the value increases most rapidly, which is similar to the normal vector of the object surface. For the gradient at non-sampling points, the normal vector of the triangle patch can be used instead. Fig. 9 (C) is the zero-distance surface and its surface normal vector;

[0159]

[0160] Furthermore, the electronic device uses the spatial line segments, spatial surfaces, and normal vectors extracted in the previous steps to infer the spatial plane information. The obtained information is marked in the spatial grid. In the spatial grid, each cube will record the spatial information inside its location, such as Fig.10 As shown in the figure. This spatial information includes whether a plane exists, if a plane exists, record the plane normal vector, whether a long line segment exists, whether a short line segment exists, etc. This information is used to initialize and constrain the shape, position and size of the Gaussian ellipsoid when it is reconstructed using a three-dimensional Gaussian ellipsoid in the future.

[0161] In some embodiments of the present application, the length threshold may be an optimal threshold found by a histogram classification method, namely, the Otsu method, which may classify all spatial line segments into two categories: long line segments and short line segments.

[0162] For example, according to a histogram of the lengths of all line segments in the scene, a suitable threshold can be found through a histogram classification method to classify the line segments into two categories: long line segments and short line segments.

[0163] Specifically, the electronic device uses the long line segments to find the spatial plane in the SDF space established in the previous step by utilizing the strong correlation between the distribution of the long line segments and the plane.

[0164] First, when extracting plane information, the electronic device selects two line segments from the long line segment and determines whether their four vertices are coplanar. Fig.11 As shown, line segment L1 and line segment L2 are coplanar, while L1 and line segment L4 are not coplanar. For the two coplanar line segments, calculate the spatial plane P_L1L2 and its normal vector. And access the grid points of the area covered by the spatial quadrilateral formed by these two lines L1 and Ll2 in the grid calculated in the previous step. The judgment of whether two line segments are coplanar is given by formulas (15) to (18), as follows: Fig.12 shown.

[0165] Furthermore, if the grid points covered by the spatial quadrilateral have no surface, it means that the plane passes through the inside of the object or is outside the object. Fig.11 The plane formed by the midline segments L1 and L3. If the grid points covered by the spatial quadrilateral have a surface, check whether the surface normal vector is parallel to the normal vector of the spatial quadrilateral. If the two normal vectors are parallel, it means that a plane in space is detected by a pair of spatial lines, such as Fig.11 The plane formed by line segment L2 and line segment L3. If the plane normal vector is not parallel to the plane normal vector of the grid point, it means that the surface is not the plane formed by plane L1 and line segment L2. All grids where the plane is detected are recorded as the grid where the plane is located.

[0166] The following is an exemplary explanation of the process of distinguishing space line segments in combination with the formula.

[0167] (1) The formula for calculating the direction vectors d1 and d2 of two line segments L1 and L2 is as follows:

[0168]

[0169] (2) Select the starting point p1 of L1 and calculate two vectors v1 and v2 to the other line segment.

[0170]

[0171] (3) Calculate the mixed product V of the three vectors starting from p1.

[0172]

[0173] It should be noted that if the absolute value of V is very small, less than a certain threshold, then the two line segments are considered to be coplanar. The judgment condition of V is shown in the following formula:

[0174] ||V||=0,or,||V|| <T (19)

[0175] Second, when extracting long line segment information, the electronic device records the long line segment information for the grid through which the long line segment passes. The spatial points in the grid through which the long line segment passes are no longer used to initialize the three-dimensional Gaussian ellipsoid.

[0176] Third, when extracting short line segment information, the electronic device records the length of the corresponding short line segment for the grid where the short line segment exists, calculates the average value of all short line segments, and sets the average value as the threshold information. According to the threshold information, the electronic device will suppress the length of the three-dimensional ellipsoid during the optimization process and delete the three-dimensional ellipsoid that is too long.

[0177] In some embodiments of the present application, the above target operation includes at least one of the following:

[0178] Adjusting the distribution of at least one three-dimensional Gaussian ellipsoid in the three-dimensional Gaussian ellipsoid set;

[0179] adjusting the shape of at least one three-dimensional Gaussian ellipsoid in the three-dimensional Gaussian ellipsoid set;

[0180] The size of at least one three-dimensional Gaussian ellipsoid in the three-dimensional Gaussian ellipsoid set is adjusted.

[0181] In some embodiments of the present application, the distribution in the three-dimensional Gaussian ellipsoid set may be a distribution in the three-dimensional Gaussian ellipsoid set generated in a region with specific geometric features, for example, a distribution in the three-dimensional Gaussian ellipsoid set generated according to a space line segment.

[0182] In some embodiments of the present application, the shape of the three-dimensional Gaussian ellipsoid can be determined based on the lengths of the three axes of the three-dimensional Gaussian ellipsoid, and appears as the shape of a three-dimensional Gaussian ellipsoid macroscopically.

[0183] Illustratively, when the three axes of the ellipsoid are of equal length, the shape is a perfect sphere.

[0184] As another example, when the length of one of the axes is much smaller than the lengths of the other two axes, the ellipsoid forms a flat ellipse.

[0185] As another example, when the length of one of the axes is much greater than the lengths of the other two axes, the ellipsoid forms a cylindrical ellipsoid.

[0186] In some embodiments of the present application, the size of the three-dimensional Gaussian ellipsoid may be determined based on the lengths of the three axes of the three-dimensional Gaussian ellipsoid, and may be macroscopically represented as the volume of the three-dimensional Gaussian ellipsoid.

[0187] During the optimization process, the electronic device optimizes the three-dimensional Gaussian ellipsoid using the above-mentioned spatial feature information.

[0188] In some examples, in a mesh where a plane passes through, during the optimization of the shape of the ellipsoid, the normal vector and the flattened shape of the ellipsoid are maintained, and the center of the ellipsoid is ensured to always remain near the plane.

[0189] Specifically, for a Gaussian ellipsoid in a plane, the smallest scale in the S matrix is ​​always smaller than T small .

[0190] Specifically, the axis direction corresponding to the minimum scale is always parallel to the plane normal vector, that is, the angle between the two vectors is less than a certain threshold θ small .

[0191] In other examples, the size of a spatial ellipsoid is supervised in a grid with short line segments, and the ellipsoid is split into smaller ellipsoids after the length of the longest axis is found to be greater than a threshold of the calculated object size.

[0192] Specifically, each segmentation starts from the center of the line segment representing the longest axis of the ellipsoid, and a line segment is divided into two line segments. Each line segment is reinitialized according to the above process of generating a three-dimensional Gaussian ellipsoid set.

[0193] In some other examples, the electronic device utilizes the strong correlation between the long line segment and the plane to constrain the optimization of the long line segment ellipsoid. For example, the long line segment is often located at the junction of two planes, such as the junction of walls; or the long line segment exists on a plane, such as the boundary of a window on a wall.

[0194] Specifically, the longest axis of the long line segment ellipsoid is always perpendicular to the normal vector of the adjacent plane.

[0195] Specifically, the distance from the center of the long line segment ellipsoid to the adjacent plane is made as small as possible.

[0196] In this way, the electronic device uses line segments to extract spatial planes and adds plane information to the optimization process, ensuring the geometric accuracy of the reconstruction results, thereby simplifying the amount of data in the three-dimensional scene reconstruction and improving the reconstruction efficiency.

[0197] The following is an exemplary description of the three-dimensional scene reconstruction method provided in the embodiment of the present application through an embodiment.

[0198] Exemplarily, the above three-dimensional scene reconstruction method may include the following steps:

[0199] Step 11: Use 3D scene reconstruction technology to restore the camera pose and the spatial structure of the scene.

[0200] For example, the electronic device uses 3D reconstruction technology (SFM) to reconstruct the camera lens parameters, the spatial position and posture of the camera, and the sparse structure of the scene for a set of photos. The sparse point cloud and spatial line segments in the scene can be restored by extracting feature points and feature lines on the surface of the photo.

[0201] For example, after obtaining the lens parameters of the camera, the photo is dedistorted to obtain a photo without lens distortion. After obtaining the camera position and posture parameters, the multi-view stereo vision technology MVS can be used to reconstruct the depth map of each photo and obtain a dense point cloud expression of the scene, such as Figure 2 The value of each pixel in the depth map of the photo is the distance from the observed object to the center of the lens.

[0202] Step 12: Use the depth map fusion technology to establish the distance field (SDF) of the space where the reconstructed scene is located.

[0203] Exemplarily, first, at the reconstructed scene, a spatial grid with a certain resolution and a size that can cover the scene is set, and each point in the grid is projected onto several photos that are closest to it and can be observed. The distance from the grid point to the surface of the object is the difference between the spatial distance and the depth map value of the projection point.

[0204] Furthermore, the surface of the figure in the scene is extracted by a zero level set surface extraction algorithm based on SDF, such as marching cube, and the normal vector is calculated for the surface. The zero level set surface is the set of surfaces whose distance to the object is 0, that is, the object surface.

[0205] Step 13: Classification of spatial line segments.

[0206] Exemplarily, the electronic device can find a suitable threshold value through the Otsu binarization algorithm according to the histogram of the lengths of all line segments in the scene, and classify the line segments into two categories: long line segments and short line segments.

[0207] The three-dimensional scene reconstruction method provided in the embodiment of the present application utilizes the strong correlation between the distribution of long line segments and planes, and uses long line segments to find spatial planes in the SDF space established in the previous step. Short line segments will constrain the size of non-line segment objects in space. The algorithm will require that the longest axis size of the ellipsoid representing the non-line segment object is smaller than the average value of the short line segments.

[0208] Step 14: Extract and record spatial information to obtain a grid with spatial information.

[0209] Exemplarily, the electronic device will use the previously extracted spatial line segments, spatial surfaces, and normal vectors to infer the spatial plane information. The obtained information is marked in the spatial grid. In the spatial grid, each three-dimensional grid cube will record the spatial information inside its location. This spatial information includes whether there is a plane, if there is a plane, then record the plane normal vector; whether there is a long line segment; whether there is a short line segment, etc. This information will initialize and constrain the shape, position and size of the Gaussian ellipsoid when it is reconstructed using a three-dimensional Gaussian ellipsoid in the future.

[0210] Step 15: Initialize the Gaussian kernel from spatial points and spatial line segments.

[0211] Exemplarily, the electronic device initializes the reconstructed spatial points and spatial line segments to form a three-dimensional Gaussian ellipsoid.

[0212] Step 16: Optimization process of 3D Gaussian sputtering rendering and its spatial ellipsoid constraint.

[0213] Exemplarily, after the grid with spatial feature information is established, the grid through which the plane passes is marked, and the grid with short line segments marks the line segment size of the local area. The average length of the short line segments will represent the upper threshold of the size of the object in the area. During the optimization process, the electronic device uses the above spatial feature information to supervise the optimization process of the ellipsoid.

[0214] The embodiment of the present application provides a radiation field rendering and reconstruction method based on a three-dimensional Gaussian kernel that combines spatial line segments and spatial plane features to initialize and constrain the reconstruction process in a targeted manner. In order to address the shortcomings of traditional methods that do not combine spatial geometric information, the present application designs a data structure and method for extracting and storing spatial geometric information. In view of the shortcomings of traditional methods that have a single initialization, the present application improves on different features and proposes different initialization methods. The present application uses the extracted geometric information to increase the spatial constraints of the reconstruction and reduce the geometric distortion in the reconstructed scene.

[0215] In the embodiment of the present application, on the one hand, the spatial geometric information is extracted by storing the information of the extracted spatial straight line and spatial plane features in the spatial grid, and combining the spatial line segments and spatial planes. On the other hand, in view of the problem that the spatial points cannot be targeted at a specific spatial structure, the present application provides a new initialization method for the line feature objects in the scene. The shape of the three-dimensional ellipsoid is made to fit the detected spatial lines, reducing the amount of algorithm data. On the other hand, according to the spatial line segment and spatial plane features, the distribution, shape and size of the three-dimensional Gaussian kernel in the scene are constrained to achieve the geometric effect of improving reconstruction. For example, by dividing the line segments into long line segments and short line segments. Using the length of the short line segment to constrain the granularity of the local space, the three-dimensional Gaussian ellipsoid with unreasonable size can be eliminated. Through the strong correlation between the long line segments in space and the spatial plane, the distribution of the spatial lines in the reconstruction is controlled. For example, long lines in space often exist at the junction of two planes, or fit on one plane. Through the normal vector information of the plane, the three-dimensional Gaussian ellipsoid can be constrained to be flatter, thereby reducing geometric distortion.

[0216] The above-mentioned method embodiments, or various possible implementation methods in each method embodiment, can be executed separately, or, under the premise that there is no contradiction, can also be executed in combination with each other. The specific implementation can be determined according to actual usage requirements, and the embodiments of the present application do not limit this.

[0217] The 3D scene reconstruction method provided in the embodiment of the present application can be executed by a 3D scene reconstruction device. In the embodiment of the present application, the 3D scene reconstruction device executing the 3D scene reconstruction method is taken as an example to illustrate the 3D scene reconstruction device provided in the embodiment of the present application.

[0218] Fig.13 A schematic diagram of the structure of a three-dimensional scene reconstruction device 600 provided in an embodiment of the present application is shown in FIG. Fig.13 As shown, the three-dimensional scene reconstruction device may include: a processing module 601; the processing module 601 is used to determine the spatial geometric information of the shooting scene based on M two-dimensional images, the M two-dimensional images are images of the shooting scene at different shooting angles, the spatial geometric information at least includes: at least one target space point and at least one space line segment, M is an integer greater than 1; the processing module 601 is also used to generate a three-dimensional Gaussian ellipsoid set based on the spatial geometric information, the three-dimensional Gaussian ellipsoid set includes at least one three-dimensional Gaussian ellipsoid, and the three-dimensional Gaussian ellipsoid set is used to characterize the three-dimensional scene information of the shooting scene.

[0219] In some embodiments of the present application, the above-mentioned processing module is specifically used to: perform line segment detection on the above-mentioned M two-dimensional images to determine the above-mentioned spatial line segments; determine the above-mentioned at least one target spatial point based on the target feature points in the dense point cloud of the above-mentioned shooting scene, and the above-mentioned target feature points are feature points in the above-mentioned dense point cloud except for the feature points corresponding to the above-mentioned spatial line segments.

[0220] In some embodiments of the present application, the above-mentioned processing module is specifically used to: determine the target parameters of the above-mentioned at least one three-dimensional Gaussian ellipsoid based on the above-mentioned spatial geometric information, and the above-mentioned target parameters include at least one of the following: the center position of the above-mentioned three-dimensional Gaussian ellipsoid, the scale of the above-mentioned three-dimensional Gaussian ellipsoid, and the rotation direction of the above-mentioned three-dimensional Gaussian ellipsoid; generate at least one three-dimensional Gaussian ellipsoid based on the above-mentioned spatial geometric information.

[0221] In some embodiments of the present application, the above-mentioned target parameters include the center position of the above-mentioned three-dimensional Gaussian ellipsoid; the above-mentioned processing module is specifically used to: obtain the endpoint information of the above-mentioned at least one spatial line segment; and determine the center position of the above-mentioned at least one three-dimensional Gaussian ellipsoid based on the above-mentioned endpoint information.

[0222] In some embodiments of the present application, the above-mentioned target parameters include the size of the above-mentioned three-dimensional Gaussian ellipsoid; the above-mentioned processing module is specifically used to: obtain the length of the above-mentioned at least one spatial line segment; determine the length of the above-mentioned at least one spatial line segment as the length of the longest axis of the above-mentioned at least one three-dimensional Gaussian ellipsoid.

[0223] In some embodiments of the present application, the above-mentioned target parameters include the rotation direction of the above-mentioned three-dimensional Gaussian ellipsoid, and the above-mentioned processing module is specifically used to: determine the angle information corresponding to the above-mentioned at least one spatial line segment, the above-mentioned angle information includes: the first angle between the above-mentioned at least one spatial line segment and the X-axis, the second angle between the above-mentioned at least one spatial line segment and the Y-axis, and the third angle between the above-mentioned at least one spatial line segment and the Z-axis; determine the normal vector of the above-mentioned spatial line segment; based on the above-mentioned angle information and the above-mentioned normal vector, determine the rotation matrix of the above-mentioned three-dimensional Gaussian ellipsoid, and the above-mentioned rotation matrix is ​​used to characterize the rotation direction of the above-mentioned three-dimensional Gaussian ellipsoid.

[0224] In some embodiments of the present application, the above-mentioned processing module is also used to obtain the spatial feature information of the above-mentioned shooting scene after generating the three-dimensional Gaussian ellipsoid set according to the above-mentioned spatial geometric information; the above-mentioned processing module is also used to perform target operations on the above-mentioned three-dimensional Gaussian ellipsoid set according to the above-mentioned spatial feature information; wherein the above-mentioned target operation includes at least one of the following: adjusting the distribution of the above-mentioned at least one three-dimensional Gaussian ellipsoid in the above-mentioned three-dimensional Gaussian ellipsoid set; adjusting the shape of at least one three-dimensional Gaussian ellipsoid in the above-mentioned three-dimensional Gaussian ellipsoid set; adjusting the size of at least one three-dimensional Gaussian ellipsoid in the above-mentioned three-dimensional Gaussian ellipsoid set.

[0225] In some embodiments of the present application, the above-mentioned spatial feature information includes at least one of the following:

[0226] Normal vector information of the plane in the above shooting scene;

[0227] Information about the spatial line segments in the above-mentioned shooting scene;

[0228] Among them, the above-mentioned spatial line segments include long line segments and short line segments. The above-mentioned long line segments are spatial line segments in the above-mentioned shooting scene whose length is greater than or equal to the length threshold, and the above-mentioned short line segments are spatial line segments in the above-mentioned shooting scene whose length is less than the above-mentioned length threshold.

[0229] The three-dimensional scene reconstruction device provided in the embodiment of the present application determines the spatial geometric information of the shooting scene based on the dense point cloud of the shooting scene corresponding to M two-dimensional images, and the spatial geometric information at least includes: at least one target space point and at least one space line segment, M is an integer greater than 1; based on the above spatial geometric information, a three-dimensional Gaussian ellipsoid set is generated, and the above three-dimensional Gaussian ellipsoid set includes at least one three-dimensional Gaussian ellipsoid for characterizing the three-dimensional scene information of the shooting scene. Through this method, the three-dimensional scene reconstruction device can generate a three-dimensional Gaussian ellipsoid set in combination with the space points and the space line segments, and realize the use of a smaller number of three-dimensional Gaussian ellipsoids to characterize the scene space of a specific area, that is, directly generating a three-dimensional Gaussian ellipsoid from the space line segments, thereby reducing the amount of data in the three-dimensional scene reconstruction, and thus improving the reconstruction efficiency.

[0230] The three-dimensional scene reconstruction device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0231] The 3D scene reconstruction device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0232] The three-dimensional scene reconstruction device provided in the embodiment of the present application can implement each process implemented in the above-mentioned three-dimensional scene reconstruction method embodiment, and will not be described again here to avoid repetition.

[0233] Alternatively, if Fig.14 As shown, an embodiment of the present application also provides an electronic device 700, including a processor 701 and a memory 702, wherein the memory 702 stores programs or instructions that can be executed on the processor 701, and when the program or instructions are executed by the processor 701, the various steps of the above-mentioned three-dimensional scene reconstruction method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0234] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0235] Fig.15 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0236] The electronic device 100 includes but is not limited to components such as a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110.

[0237] Those skilled in the art will appreciate that the electronic device 100 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.15 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0238] Among them, the above-mentioned processor 110 is used to determine the spatial geometric information of the shooting scene based on M two-dimensional images, the above-mentioned M two-dimensional images are images of the above-mentioned shooting scene at different shooting angles, and the above-mentioned spatial geometric information at least includes: at least one target space point and at least one spatial line segment, and M is an integer greater than 1; the above-mentioned processor 110 is also used to generate a three-dimensional Gaussian ellipsoid set based on the above-mentioned spatial geometric information, the above-mentioned three-dimensional Gaussian ellipsoid set includes at least one three-dimensional Gaussian ellipsoid, and the above-mentioned three-dimensional Gaussian ellipsoid set is used to characterize the three-dimensional scene information of the above-mentioned shooting scene.

[0239] In some embodiments of the present application, the processor 110 is specifically used to: perform line segment detection on the M two-dimensional images to determine the spatial line segments; determine at least one target spatial point based on the target feature points in the dense point cloud of the shooting scene, where the target feature points are feature points in the dense point cloud except for the feature points corresponding to the spatial line segments.

[0240] In some embodiments of the present application, the processor 110 is specifically used to: determine the target parameters of the at least one three-dimensional Gaussian ellipsoid based on the spatial geometric information, the target parameters including at least one of the following: the center position of the three-dimensional Gaussian ellipsoid, the scale of the three-dimensional Gaussian ellipsoid, and the rotation direction of the three-dimensional Gaussian ellipsoid; generate at least one three-dimensional Gaussian ellipsoid based on the spatial geometric information.

[0241] In some embodiments of the present application, the above-mentioned target parameters include the center position of the above-mentioned three-dimensional Gaussian ellipsoid; the above-mentioned processor 110 is specifically used to: obtain the endpoint information of the above-mentioned at least one spatial line segment; and determine the center position of the above-mentioned at least one three-dimensional Gaussian ellipsoid based on the above-mentioned endpoint information.

[0242] In some embodiments of the present application, the above-mentioned target parameters include the size of the above-mentioned three-dimensional Gaussian ellipsoid; the above-mentioned processor 110 is specifically used to: obtain the length of the above-mentioned at least one spatial line segment; determine the length of the above-mentioned at least one spatial line segment as the length of the longest axis of the above-mentioned at least one three-dimensional Gaussian ellipsoid.

[0243] In some embodiments of the present application, the above-mentioned target parameters include the rotation direction of the above-mentioned three-dimensional Gaussian ellipsoid, and the above-mentioned processor 110 is specifically used to: determine the angle information corresponding to the above-mentioned at least one spatial line segment, the above-mentioned angle information includes: the first angle between the above-mentioned at least one spatial line segment and the X-axis, the second angle between the above-mentioned at least one spatial line segment and the Y-axis, and the third angle between the above-mentioned at least one spatial line segment and the Z-axis; determine the normal vector of the above-mentioned spatial line segment; based on the above-mentioned angle information and the above-mentioned normal vector, determine the rotation matrix of the above-mentioned three-dimensional Gaussian ellipsoid, and the above-mentioned rotation matrix is ​​used to characterize the rotation direction of the above-mentioned three-dimensional Gaussian ellipsoid.

[0244] In some embodiments of the present application, the processor 110 is further used to obtain spatial feature information of the shooting scene after generating a three-dimensional Gaussian ellipsoid set according to the spatial geometric information; the processor 110 is further used to perform a target operation on the three-dimensional Gaussian ellipsoid set according to the spatial feature information; wherein the target operation includes at least one of the following: adjusting the distribution of at least one three-dimensional Gaussian ellipsoid in the three-dimensional Gaussian ellipsoid set; adjusting the shape of at least one three-dimensional Gaussian ellipsoid in the three-dimensional Gaussian ellipsoid set; adjusting the size of at least one three-dimensional Gaussian ellipsoid in the three-dimensional Gaussian ellipsoid set.

[0245] In some embodiments of the present application, the above-mentioned spatial feature information includes at least one of the following:

[0246] Normal vector information of the plane in the above shooting scene;

[0247] Information about the spatial line segments in the above-mentioned shooting scene;

[0248] Among them, the above-mentioned spatial line segments include long line segments and short line segments. The above-mentioned long line segments are spatial line segments in the above-mentioned shooting scene whose length is greater than or equal to the length threshold, and the above-mentioned short line segments are spatial line segments in the above-mentioned shooting scene whose length is less than the above-mentioned length threshold.

[0249] The electronic device provided in the embodiment of the present application determines the spatial geometric information of the shooting scene based on M two-dimensional images, wherein the M two-dimensional images are images of the shooting scene at different shooting angles, and the spatial geometric information at least includes: at least one target space point and at least one space line segment, and M is an integer greater than 1; based on the spatial geometric information, a three-dimensional Gaussian ellipsoid set is generated, and the three-dimensional Gaussian ellipsoid set includes at least one three-dimensional Gaussian ellipsoid for characterizing the three-dimensional scene information of the shooting scene. Through this method, the electronic device can generate a three-dimensional Gaussian ellipsoid set in combination with the space points and the space line segments, and realize the use of a smaller number of three-dimensional Gaussian ellipsoids to characterize the scene space of a specific area, that is, directly generating a three-dimensional Gaussian ellipsoid from the space line segments, thereby reducing the amount of data in the three-dimensional scene reconstruction, and thus improving the reconstruction efficiency.

[0250] It should be understood that in the embodiment of the present application, the input unit 104 may include a graphics processor (Graphics Processing Unit, GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 107 includes a touch panel 1071 and at least one of other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0251] The memory 109 can be used to store software programs and various data. The memory 109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 109 may include a volatile memory or a non-volatile memory, or the memory 109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0252] The processor 110 may include one or more processing units; optionally, the processor 110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 110.

[0253] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned three-dimensional scene reconstruction method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0254] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0255] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned three-dimensional scene reconstruction method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0256] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0257] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned three-dimensional scene reconstruction method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0258] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0259] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0260] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A three-dimensional scene reconstruction method, characterized in that: The method comprises: Determine spatial geometric information of a shooting scene according to M two-dimensional images, wherein the M two-dimensional images are images of the shooting scene at different shooting angles, and the spatial geometric information at least includes: at least one target space point and at least one space line segment, and M is an integer greater than 1; A three-dimensional Gaussian ellipsoid set is generated according to the spatial geometric information, wherein the three-dimensional Gaussian ellipsoid set includes at least one three-dimensional Gaussian ellipsoid, and the three-dimensional Gaussian ellipsoid set is used to characterize the three-dimensional scene information of the shooting scene.

2. The method according to claim 1, characterized in that Determining spatial geometric information according to the M two-dimensional images includes: Performing line segment detection on the M two-dimensional images to determine the spatial line segments; The at least one target space point is determined according to target feature points in the dense point cloud of the shooting scene, wherein the target feature points are feature points in the dense point cloud except feature points corresponding to the space line segments.

3. The method according to claim 1, characterized in that Generating a three-dimensional Gaussian ellipsoid set according to the spatial geometric information includes: Determine, according to the spatial geometric information, target parameters of the at least one three-dimensional Gaussian ellipsoid, the target parameters comprising at least one of the following: a center position of the three-dimensional Gaussian ellipsoid, a scale of the three-dimensional Gaussian ellipsoid, and a rotation direction of the three-dimensional Gaussian ellipsoid; At least one three-dimensional Gaussian ellipsoid is generated according to the spatial geometric information.

4. The method according to claim 3, characterized in that: The target parameter includes a rotation direction of the three-dimensional Gaussian ellipsoid, and determining the target parameter of the at least one three-dimensional Gaussian ellipsoid according to the spatial geometric information includes: Determine angle information corresponding to the at least one space line segment, the angle information comprising: a first angle between the at least one space line segment and the X-axis, a second angle between the at least one space line segment and the Y-axis, and a third angle between the at least one space line segment and the Z-axis; Determining a normal vector of the space segment; Based on the angle information and the normal vector, a rotation matrix of the three-dimensional Gaussian ellipsoid is determined, and the rotation matrix is ​​used to characterize the rotation direction of the three-dimensional Gaussian ellipsoid.

5. The method according to claim 1, characterized in that After generating a three-dimensional Gaussian ellipsoid set according to the spatial geometric information, the method further comprises: Acquiring spatial feature information of the shooting scene; performing a target operation on the three-dimensional Gaussian ellipsoid set according to the spatial feature information; The target operation includes at least one of the following: Adjusting the distribution of the at least one three-dimensional Gaussian ellipsoid in the three-dimensional Gaussian ellipsoid set; Adjusting the shape of at least one three-dimensional Gaussian ellipsoid in the three-dimensional Gaussian ellipsoid set; The size of at least one three-dimensional Gaussian ellipsoid in the three-dimensional Gaussian ellipsoid set is adjusted.

6. A three-dimensional scene reconstruction device, characterized in that: The device comprises: a processing module; The processing module is used to determine the spatial geometric information of the shooting scene according to M two-dimensional images, wherein the M two-dimensional images are images of the shooting scene at different shooting angles, and the spatial geometric information at least includes: at least one target space point and at least one space line segment, and M is an integer greater than 1; The processing module is further used to generate a three-dimensional Gaussian ellipsoid set according to the spatial geometric information, wherein the three-dimensional Gaussian ellipsoid set includes at least one three-dimensional Gaussian ellipsoid, and the three-dimensional Gaussian ellipsoid set is used to characterize the three-dimensional scene information of the shooting scene.

7. The device according to claim 6, characterized in that The processing module is specifically used for: Performing line segment detection on the M two-dimensional images to determine the spatial line segments; The at least one target space point is determined according to target feature points in the dense point cloud of the shooting scene, wherein the target feature points are feature points in the dense point cloud except feature points corresponding to the space line segments.

8. The device according to claim 6, characterized in that The processing module is specifically used for: Determine, according to the spatial geometric information, target parameters of the at least one three-dimensional Gaussian ellipsoid, the target parameters comprising at least one of the following: a center position of the three-dimensional Gaussian ellipsoid, a scale of the three-dimensional Gaussian ellipsoid, and a rotation direction of the three-dimensional Gaussian ellipsoid; At least one three-dimensional Gaussian ellipsoid is generated according to the spatial geometric information.

9. The device according to claim 8, characterized in that The target parameter includes the rotation direction of the three-dimensional Gaussian ellipsoid, and the processing module is specifically used for: Determine angle information corresponding to the at least one space line segment, the angle information comprising: a first angle between the at least one space line segment and the X-axis, a second angle between the at least one space line segment and the Y-axis, and a third angle between the at least one space line segment and the Z-axis; Determining a normal vector of the space segment; Based on the angle information and the normal vector, a rotation matrix of the three-dimensional Gaussian ellipsoid is determined, and the rotation matrix is ​​used to characterize the rotation direction of the three-dimensional Gaussian ellipsoid.

10. The device according to claim 6, characterized in that The processing module is further used to obtain the spatial feature information of the shooting scene after generating the three-dimensional Gaussian ellipsoid set according to the spatial geometric information; The processing module is further used to perform a target operation on the three-dimensional Gaussian ellipsoid set according to the spatial feature information; The target operation includes at least one of the following: Adjusting the distribution of the at least one three-dimensional Gaussian ellipsoid in the three-dimensional Gaussian ellipsoid set; Adjusting the shape of at least one three-dimensional Gaussian ellipsoid in the three-dimensional Gaussian ellipsoid set; The size of at least one three-dimensional Gaussian ellipsoid in the three-dimensional Gaussian ellipsoid set is adjusted.

Images