Grid linear structure recovery method, device, electronic device and storage medium

Through linear feature inspection and back projection algorithm based on linear segment detection algorithm, the feature line segments of aerial images are extracted and matched, and the problem of high complexity of three-dimensional reconstruction of aerial scenes in the prior art is solved, and more efficient three-dimensional model reconstruction is achieved.

CN114202632BActive Publication Date: 2024-11-12BEIHANG UNIV +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111469581.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-03
Publication Date
2024-11-12
Estimated Expiration
2041-12-03

Smart Images

  • Figure CN114202632B_ABST
    Figure CN114202632B_ABST
Patent Text Reader

Abstract

The present application provides a method, device, electronic device and storage medium for restoring linear structure of a mesh, the method comprising: extracting characteristic line segments of an aerial image according to a linear feature inspection algorithm based on a straight line segment detection algorithm; matching three-dimensional space line segments for each characteristic line segment of the aerial image, clustering the three-dimensional space line segments to obtain three-dimensional line cluster results of the characteristic line segments; for each three-dimensional line segment in the three-dimensional line cluster results, using a back-projection algorithm, projecting the three-dimensional line segment onto a mesh model composed of triangular facets, and adjusting the vertex positions of the triangular facets covered by the three-dimensional line segment according to the topological constraints of the three-dimensional line segment. By extracting characteristic line segments and matching three-dimensional space line segments, the mesh nodes are adjusted and optimized according to the three-dimensional topological constraints, thereby reducing the complexity of the reconstruction of three-dimensional scenes of aerial images.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to virtual reality technology, and in particular to a method, device, electronic device and storage medium for restoring a grid linear structure. Background Art

[0002] The main information in large-scale aerial scene images has extremely high linear structural features, and the optimization of linear detail structural features can make the final generated 3D model closer to the real scene and improve the model quality. The reconstruction of the 3D scene can be completed by restoring and optimizing the linear structure of the grid. There are three main methods: incremental, energy function and dense point cloud.

[0003] However, in the prior art, the algorithms used for three-dimensional reconstruction of aerial photography scenes do not work well for large-scale data, and the complexity of the algorithms is too high. Summary of the invention

[0004] The present application provides a grid linear structure recovery method, device, electronic device and storage medium to reduce the complexity of three-dimensional scene reconstruction of aerial images.

[0005] In a first aspect, the present application provides a grid linear structure recovery method, comprising:

[0006] According to the linear feature detection algorithm based on the straight line segment detection algorithm, the characteristic line segments of the aerial image are extracted;

[0007] For each characteristic line segment of the aerial image, matching the characteristic line segment with a three-dimensional space line segment, and clustering the three-dimensional space line segment to obtain a three-dimensional line cluster result of the characteristic line segment;

[0008] For each three-dimensional line segment in the three-dimensional line cluster result, a back-projection algorithm is used to project the three-dimensional line segment onto a mesh model composed of triangular facets, and the vertex positions of the triangular facets covered by the three-dimensional line segment are adjusted according to the topological constraints of the three-dimensional line segment.

[0009] Optionally, extracting characteristic line segments of the aerial image according to a linear feature detection algorithm based on a straight line segment detection algorithm includes:

[0010] Calculating the gradient value and gradient direction of each pixel point in the aerial image, and pseudo-sorting the points according to the gradient values ​​of the points;

[0011] For each pixel point of the aerial image, a regional diffusion algorithm is used to estimate and fit a line segment to obtain a line segment in the aerial image;

[0012] Continuous line segments whose angle differences are not greater than a set angle threshold are selected from the line segments as the characteristic line segments.

[0013] Optionally, matching a three-dimensional space line segment to each feature line segment of the aerial image, and clustering the three-dimensional space line segment to obtain a three-dimensional line cluster result of the feature line segment includes:

[0014] Calculating the affinity between the feature line segment and the matching three-dimensional space line segment according to the angle similarity and the space similarity;

[0015] According to the affinity, based on the detection line confidence measurement function, a confidence measurement function value is calculated and a three-dimensional space line segment with the highest confidence measurement function value is used as the three-dimensional space position of the feature line segment;

[0016] For the three-dimensional spatial position, an affinity matrix is ​​generated based on a pairwise relationship or proximity-based replicator graph clustering strategy. For each item of the affinity matrix, an element with maximized consistency is queried within its visual neighborhood set to obtain a three-dimensional line cluster result of the feature line segment.

[0017] Optionally, for each three-dimensional line segment in the three-dimensional line cluster result, projecting the three-dimensional line segment onto a mesh model composed of triangular facets by using a back-projection algorithm includes:

[0018] According to the camera pose information and the visibility relationship between the three-dimensional line cluster result and the aerial image, projecting each three-dimensional line segment in the three-dimensional line cluster result onto the grid model;

[0019] Calculate the vertical projection distance from each vertex of the triangular facet covered by the three-dimensional line segment to the three-dimensional line segment, and select the vertex with the shortest distance as the vertex to be adjusted;

[0020] A vertical vector from the vertex to be adjusted to the three-dimensional line segment is calculated, and the vertex to be adjusted is translated along the direction of the vertical vector until the vertical projection distance from the vertex to be adjusted to the three-dimensional line segment is equal to 0.

[0021] Optionally, the feature line segments are represented by an undirected graph, the nodes in the undirected graph represent pixel points on the aerial image, and the edges in the undirected graph represent line segments connecting two pixel points.

[0022] In a second aspect, the present application provides a grid linear structure recovery device, comprising:

[0023] A line segment extraction module is used to extract characteristic line segments of the aerial image according to a linear feature detection algorithm based on a straight line segment detection algorithm;

[0024] A line segment matching module is used to match a three-dimensional space line segment for each characteristic line segment of the aerial image, and cluster the three-dimensional space line segment to obtain a three-dimensional line cluster result of the characteristic line segment;

[0025] A structure adjustment module is used to project each three-dimensional line segment in the three-dimensional line cluster result onto a mesh model composed of triangular facets using a back-projection algorithm, and to adjust the vertex positions of the triangular facets covered by the three-dimensional line segment according to the topological constraints of the three-dimensional line segment.

[0026] Optionally, the line segment extraction module is specifically used to:

[0027] Calculating the gradient value and gradient direction of each pixel point in the aerial image, and pseudo-sorting the points according to the gradient values ​​of the points;

[0028] For each pixel point of the aerial image, a regional diffusion algorithm is used to estimate and fit a line segment to obtain a line segment in the aerial image;

[0029] Continuous line segments whose angle differences are not greater than a set angle threshold are selected from the line segments as the characteristic line segments.

[0030] Optionally, the line segment matching module is specifically used to:

[0031] Calculating the affinity between the feature line segment and the matching three-dimensional space line segment according to the angle similarity and the space similarity;

[0032] According to the affinity, based on the detection line confidence measurement function, a confidence measurement function value is calculated and a three-dimensional space line segment with the highest confidence measurement function value is used as the three-dimensional space position of the feature line segment;

[0033] For the three-dimensional spatial position, an affinity matrix is ​​generated based on a pairwise relationship or proximity-based replicator graph clustering strategy. For each item of the affinity matrix, an element with maximized consistency is queried within its visual neighborhood set to obtain a three-dimensional line cluster result of the feature line segment.

[0034] Optionally, the structure adjustment module is specifically used to:

[0035] According to the camera pose information and the visibility relationship between the three-dimensional line cluster result and the aerial image, projecting each three-dimensional line segment in the three-dimensional line cluster result onto the grid model;

[0036] Calculate the vertical projection distance from each vertex of the triangular facet covered by the three-dimensional line segment to the three-dimensional line segment, and select the vertex with the shortest distance as the vertex to be adjusted;

[0037] A vertical vector from the vertex to be adjusted to the three-dimensional line segment is calculated, and the vertex to be adjusted is translated along the direction of the vertical vector until the vertical projection distance from the vertex to be adjusted to the three-dimensional line segment is equal to 0.

[0038] Optionally, the feature line segments are represented by an undirected graph, the nodes in the undirected graph represent pixel points on the aerial image, and the edges in the undirected graph represent line segments connecting two pixel points.

[0039] In a third aspect, the present application provides an electronic device, including:

[0040] at least one processor; and

[0041] a memory communicatively connected to the at least one processor; wherein,

[0042] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to the first aspect.

[0043] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect.

[0044] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0045] The present application provides a method, device, electronic device and storage medium for restoring linear structure of a mesh, the method comprising: extracting characteristic line segments of an aerial image according to a linear feature inspection algorithm based on a straight line segment detection algorithm; matching three-dimensional space line segments for each characteristic line segment of the aerial image, clustering the three-dimensional space line segments to obtain three-dimensional line cluster results of the characteristic line segments; for each three-dimensional line segment in the three-dimensional line cluster results, using a back-projection algorithm, projecting the three-dimensional line segment onto a mesh model composed of triangular facets, and adjusting the vertex positions of the triangular facets covered by the three-dimensional line segment according to the topological constraints of the three-dimensional line segment. By extracting characteristic line segments and matching three-dimensional space line segments, the mesh nodes are adjusted and optimized according to the three-dimensional topological constraints, thereby reducing the complexity of the reconstruction of three-dimensional scenes of aerial images. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0047] Figure 1 Schematic diagram of application scenarios provided for this application example;

[0048] Figure 2 A schematic diagram of a grid linear structure recovery method provided in Example 1 of the present application;

[0049] Figure 3 A schematic diagram of a grid linear structure recovery method provided in Example 2 of the present application;

[0050] Figure 4 A schematic diagram of a grid linear structure recovery method provided in Example 3 of the present application;

[0051] Figure 5 A schematic diagram of a flow chart of a grid linear structure recovery method provided in Embodiment 4 of the present application;

[0052] Figure 6 A schematic diagram of the structure of a grid linear structure recovery device provided in Embodiment 5 of the present application;

[0053] Figure 7 A schematic diagram of the structure of an electronic device provided in Example 6 of the present application.

[0054] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0055] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0056] Virtual reality technology is based on computer technology and integrates multidisciplinary knowledge. It outputs a digital simulation system that is highly similar to the real environment from a multi-sensory perspective and allows users to interact with it to gain an immersive experience. The reconstruction of three-dimensional virtual scenes has an important impact on the user's sense of immersion.

[0057] Usually, the reconstruction of three-dimensional scenes is performed based on aerial images. Figure 1 The application scenario diagram provided for this application example is as follows: Figure 1 As shown, Figure 1An aerial image and a schematic diagram of its linear structure are given to illustrate the existing technology of grid linear structure recovery.

[0058] Large-scale urban scene images acquired through aerial photography are rich in information mainly consisting of buildings, which have extremely high linear structural features. The optimization of linear detail structural features can make the final generated three-dimensional model closer to the real scene and improve the model quality. Therefore, there are many methods in the prior art that use three-dimensional line segments to construct scene models, mainly including three types: incremental-based, energy function-based, and dense point cloud-based.

[0059] Among them, the incremental algorithm first selects and calculates the initial image set, and then increases it on this basis, but the cumulative error value of this method increases sharply with the increase in the number of images, so it is not suitable for large-scale scene aerial photography data sets; although the energy function-based method can produce more accurate reconstruction results for scenes with interference from external environmental factors, it uses repeated iterative optimization operations on vertices, so for large-scale scenes, it takes a long time and has high requirements on computing resources; the dense point cloud-based method processes the point cloud. Due to the large amount of data, the generated dense point cloud is usually complex, which makes the algorithm computationally complex and inefficient. In the prior art, the algorithm for three-dimensional reconstruction of aerial scenes is not effective for large-scale data, and the complexity of the algorithm is too high.

[0060] The technical solutions of the present application and the technical solutions of the present application are described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. In the description of the present application, unless otherwise clearly specified and limited, each term should be understood in a broad sense in the art. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0061] Embodiment 1

[0062] Figure 2 A schematic diagram of a grid linear structure recovery method provided in Example 1 of the present application is shown in FIG. Figure 2 As shown, the method includes:

[0063] S101: extracting characteristic line segments of the aerial image according to a linear feature detection algorithm based on a straight line segment detection algorithm;

[0064] S102: For each feature line segment of the aerial image, match the feature line segment with a three-dimensional space line segment, and cluster the three-dimensional space line segment to obtain a three-dimensional line cluster result of the feature line segment;

[0065] S103: For each three-dimensional line segment in the three-dimensional line cluster result, use a back-projection algorithm to project the three-dimensional line segment onto a mesh model composed of triangular facets, and adjust the vertex positions of the triangular facets covered by the three-dimensional line segment according to the topological constraints of the three-dimensional line segment.

[0066] This embodiment is explained in conjunction with a specific application scenario: In virtual reality technology, the establishment of a three-dimensional scene is usually based on multiple triangular facets, and the three-dimensional reconstruction of the real model is completed by building and connecting the triangular facets. Therefore, in virtual reality technology, the position, shape, size, etc. of the triangular mesh used to form the triangular facets, or other shapes of the mesh, will significantly affect the effect of the three-dimensional scene reconstruction.

[0067] For large-scale three-dimensional scenes, such as building images obtained from aerial photography, mesh detail optimization based on linear features can be performed to obtain better quality of three-dimensional reconstruction scene models. First, linear features are extracted, screened and evaluated for images in the image set, and a linear feature detection algorithm based on LSD is used to extract feature line segments in aerial images. Then, the spatial positions of feature line segments are matched using angle similarity and spatial similarity, and clustering operations are performed on three-dimensional space line segments to generate three-dimensional line cluster results corresponding to feature line segments. The three-dimensional linear model and mesh are integrated using back-projection operations, and the three-dimensional line segments are projected onto the mesh model composed of triangular facets to constrain the movement of triangular facet vertices. The vertices are further optimized and adjusted according to the topological constraints of the three-dimensional line segments to obtain an optimized high-quality mesh model.

[0068] In one example, the feature line segment can be represented by an undirected graph, which is a line segment representation method based on graph theory, wherein the nodes in the undirected graph represent the pixels on the aerial image, and the edges in the undirected graph represent the line segments connecting two pixels, that is, when there is a line segment connection between the pixels represented by the nodes, there is an edge between the two nodes in the undirected graph. This representation method can be used to represent the connection relationship between pixels in large-scale images.

[0069] This embodiment provides a method for restoring linear structures of a mesh, including: extracting characteristic line segments of an aerial image according to a linear feature detection algorithm based on a straight line segment detection algorithm; matching three-dimensional space line segments for each characteristic line segment of the aerial image, and clustering the three-dimensional space line segments to obtain three-dimensional line cluster results of the characteristic line segments; for each three-dimensional line segment in the three-dimensional line cluster results, using a back-projection algorithm, projecting the three-dimensional line segment onto a mesh model composed of triangular facets, and adjusting the vertex positions of the triangular facets covered by the three-dimensional line segment according to the topological constraints of the three-dimensional line segment. By extracting characteristic line segments and matching three-dimensional space line segments, the mesh nodes are adjusted and optimized according to three-dimensional topological constraints, thereby reducing the complexity of three-dimensional scene reconstruction of aerial images.

[0070] Embodiment 2

[0071] Figure 3 A schematic diagram of a grid linear structure recovery method provided in the second embodiment of the present application is shown in FIG. Figure 3 As shown, based on any other embodiment, S101 specifically includes:

[0072] S201: Calculate the gradient value and gradient direction of each pixel point in the aerial image, and pseudo-sort the points according to the gradient values ​​of the points;

[0073] S202: For each pixel point of the aerial image, estimate and fit a line segment using a regional diffusion algorithm to obtain a line segment in the aerial image;

[0074] S203: Filter out continuous line segments whose angle differences are not greater than a set angle threshold from the line segments as the feature line segments.

[0075] This embodiment is exemplarily described in conjunction with a specific application scenario: According to the LSD algorithm, the gradient value and gradient direction of each pixel in the image can be calculated. Since the area where the pixel with a higher gradient amplitude is located usually has a stronger edge, the middle pixel in the edge generally has the highest gradient amplitude. It is feasible to start with the pixel with the highest gradient amplitude to perform straight line segmentation detection. Therefore, all pixels can be pseudo-sorted according to the gradient value, and for each pixel of the aerial image, the regional diffusion algorithm is used to estimate and fit the rectangle to obtain all detected line segments. Subsequently, the detection line representation method based on graph theory can be used to represent the extracted feature line segments using an undirected graph. The nodes in the undirected graph represent the pixels on each image in the aerial image set, and the edges in the undirected graph represent the detected line segments connecting two nodes. Finally, all line segments are sorted and filtered according to the length and angle relationship, and continuous line segments with an angle difference not greater than the set angle threshold are screened out from the line segments as the feature line segments. Typically, the angle threshold may be 20°, that is, when the direction angle deviation between consecutive line segments does not exceed 20°, all line segments that meet the conditions and have sufficient length are used as the feature line segments.

[0076] This embodiment provides a grid linear structure recovery method, which calculates the gradient value and gradient direction of each pixel point in the aerial image, and pseudo-sorts the points according to the gradient values ​​of each point; for each pixel point of the aerial image, the regional diffusion algorithm is used to estimate and fit the line segment to obtain the line segment in the aerial image; and the continuous line segments whose angle difference is not greater than the set angle threshold are screened out from the line segments as the feature line segments. Through the fitting and screening of the line segments in the aerial image, the continuous line segments that meet the length and angle requirements are selected as the linear framework of the grid, which reduces the complexity of the reconstruction of the three-dimensional scene of the aerial image.

[0077] Embodiment 3

[0078] Figure 4 A schematic diagram of a grid linear structure recovery method provided in Example 3 of the present application is shown in FIG. Figure 4 As shown, based on any other embodiment, S102 may specifically be:

[0079] S301: Calculating affinity between the feature line segment and the matching three-dimensional space line segment according to angle similarity and space similarity;

[0080] S302: According to the affinity, based on the detection line confidence measurement function, calculate to obtain a confidence measurement function value and use the three-dimensional space line segment with the highest confidence measurement function value as the three-dimensional space position of the feature line segment;

[0081] S303: For the three-dimensional spatial position, an affinity matrix is ​​generated based on a pairwise relationship or proximity-based replicator graph clustering strategy, and for each item of the affinity matrix, an element with maximized consistency is queried within its visual neighborhood set to obtain a three-dimensional line cluster result of the feature line segment.

[0082] This embodiment is illustrated with reference to a specific application scenario: the error can be measured according to the angular similarity and the spatial similarity, and based on this, the affinity between the feature line segment and the matching three-dimensional space line segment can be calculated, and based on the affinity, the confidence measurement function value can be calculated, and the three-dimensional space line segment with the highest confidence measurement function value is used as the three-dimensional space position of the feature line segment.

[0083] Specifically, S a and S p The angle and spatial errors in the three-dimensional geometric characteristics are calculated respectively. a The definition is as follows:

[0084]

[0085] ∠(h1, h1) represents the angle between two similar line segments, which is measured in degrees in actual calculations. is a regularization term.

[0086] Each 2D line segment relationship can be converted to a 3D line segment by intersecting the two planes passing through the corresponding camera centers and the 2D line segments. and It represents the three-dimensional similar line segment in the visual neighborhood set.

[0087] S p The definition is as follows:

[0088]

[0089] where d ⊥ (Z, h2) represents the normal distance between a point Z in three-dimensional space and the line passing through the three-dimensional line segment h2, d i (Z)=‖C i -Z‖2 represents image I i The camera center coordinates C i The Euclidean distance from a point Z in three-dimensional space, i.e. the depth along the ray emitted from its corresponding camera center, is a regularization term.

[0090] Expression A calculates the value of the two-dimensional line segment Projected 3D line segment The affinity between it and its similar line segments is defined as follows:

[0091]

[0092] According to the affinity, the confidence measurement function of the detection line can be calculated according to the following formula:

[0093]

[0094] For all the hypothetical matches after screening, we first need to estimate their most likely three-dimensional spatial positions. Since each two-dimensional line segment can only have a unique projection in a specific three-dimensional space, for each two-dimensional line segment The three-dimensional hypothesized line segment with the highest confidence metric function value can be as its three-dimensional spatial position.

[0095] For the three-dimensional spatial position, an affinity matrix is ​​generated based on a pairwise relationship or proximity-based replicator graph clustering strategy. Specifically, the affinity matrix can be expressed as follows:

[0096]

[0097] Where S a The angle similarity is measured. It measures the spatial similarity. For each item w in the matrix ij , query the element with maximum consistency in its visual neighborhood set, diffuse based on this, and finally output a set of three-dimensional line cluster result set ∏ = {∏1,…,∏ T}, each 3D line cluster result in the result set consists of a 3D line segment And composition A set of one or more collinear hypothetical line segments composition.

[0098] This embodiment provides a grid linear structure recovery method. For each feature line segment of the aerial image, a three-dimensional space line segment is matched for the feature line segment, and the affinity between the feature line segment and the matched three-dimensional space line segment is calculated based on the angle similarity and space similarity; based on the affinity, the confidence measurement function value is calculated based on the detection line confidence measurement function and the three-dimensional space line segment with the highest confidence measurement function value is used as the three-dimensional space position of the feature line segment; for the three-dimensional space position, an affinity matrix is ​​generated based on a pairwise relationship or a proximity-based replicator graph clustering strategy, and for each item of the affinity matrix, an element with maximized consistency is queried in its visual neighborhood set to obtain a three-dimensional line cluster result of the feature line segment. The affinity between the line segment and the possible position in the three-dimensional space is described by angle similarity and space similarity, and the most likely position is matched according to the affinity to obtain the three-dimensional line cluster result corresponding to the feature line segment, thereby reducing the complexity of the three-dimensional scene reconstruction of the aerial image.

[0099] Embodiment 4

[0100] Figure 5 A schematic diagram of a grid linear structure recovery method provided in Example 4 of the present application is shown in FIG. Figure 5 As shown, based on any other embodiment, S103 may specifically be:

[0101] S401: projecting each 3D line segment in the 3D line cluster result onto the grid model according to the camera pose information and the visibility relationship between the 3D line cluster result and the aerial image;

[0102] S402: Calculate the vertical projection distance from each vertex of the triangular facet covered by the three-dimensional line segment to the three-dimensional line segment, and select the vertex with the shortest distance as the vertex to be adjusted;

[0103] S403: Calculate and obtain a vertical vector from the vertex to be adjusted to the three-dimensional line segment, and translate the vertex to be adjusted along the direction of the vertical vector until the vertical projection distance from the vertex to be adjusted to the three-dimensional line segment is equal to 0.

[0104] This embodiment is exemplified in combination with a specific application scenario: for the three-dimensional line cluster result set Π={Π1,…,Π T}, using the camera pose information and its visibility relationship with the image set, each 3D line segment in the result set is projected onto the input mesh model. After this operation, each 3D line segment in the 3D space will cover a set of triangle face sets F = {F1, F2, ...}.

[0105] For each triangle F in the above triangle set F i, calculate the vertical projection distance of each vertex to each three-dimensional line segment H in the result set, and select the vertex with the shortest distance as the vertex to be adjusted in the next step and put it into the vertex set V. For each vertex V in the vertex set V i , find the perpendicular vector to the corresponding three-dimensional linear topological constraint line segment H and moves it along the vector , until the vertical projection distance is equal to 0, which means the three-dimensional vertex V i Completely coincide with the three-dimensional linear topology constraint line segment H. For example, for a linear structure such as the wall of a building, the initial mesh model composed of triangular facets may lose certain linear features, but by moving the positions of the triangular facet vertices, the details of this linear structure can be restored.

[0106] This embodiment provides a method for restoring the linear structure of a mesh. According to the camera posture information and the visibility relationship between the three-dimensional line cluster result and the aerial image, each three-dimensional line segment in the three-dimensional line cluster result is projected onto the mesh model; the vertical projection distance from each vertex of the triangle patch covered by the three-dimensional line segment to the three-dimensional line segment is calculated, and the vertex with the shortest distance is selected as the vertex to be adjusted; the vertical vector from the vertex to be adjusted to the three-dimensional line segment is calculated, and the vertex to be adjusted is translated along the direction of the vertical vector until the vertical projection distance of the vertex to be adjusted to the three-dimensional line segment is equal to 0. The projection of the three-dimensional line segment to the mesh model is completed through spatial topological constraints, and the vertex with the shortest vertical projection distance to the three-dimensional line segment is selected as the vertex to be adjusted for position adjustment, thereby reducing the complexity of the reconstruction of the three-dimensional scene of the aerial image.

[0107] Embodiment 5

[0108] Embodiment 5 of the present application also provides a grid linear structure recovery device to implement the above method. Figure 6 A schematic diagram of a grid linear structure recovery device provided in Embodiment 5 of the present application is shown in FIG. Figure 6 As shown, based on any other embodiment, the device includes:

[0109] A line segment extraction module 51 is used to extract characteristic line segments of the aerial image according to a linear feature detection algorithm based on a straight line segment detection algorithm;

[0110] A line segment matching module 52 is used to match a three-dimensional space line segment for each feature line segment of the aerial image, and cluster the three-dimensional space line segment to obtain a three-dimensional line cluster result of the feature line segment;

[0111] The structure adjustment module 53 is used to project each three-dimensional line segment in the three-dimensional line cluster result onto a mesh model composed of triangular facets using a back-projection algorithm, and to adjust the vertex positions of the triangular facets covered by the three-dimensional line segment according to the topological constraints of the three-dimensional line segment.

[0112] The characteristic line segments are represented by an undirected graph, the nodes in the undirected graph represent the pixel points on the aerial image, and the edges in the undirected graph represent the line segments connecting two pixel points.

[0113] In one example, the line segment extraction module 51 is specifically used for:

[0114] Calculating the gradient value and gradient direction of each pixel point in the aerial image, and pseudo-sorting the points according to the gradient values ​​of the points;

[0115] For each pixel point of the aerial image, a regional diffusion algorithm is used to estimate and fit a line segment to obtain a line segment in the aerial image;

[0116] Continuous line segments whose angle differences are not greater than a set angle threshold are selected from the line segments as the characteristic line segments.

[0117] By fitting and screening line segments in aerial images, continuous line segments that meet the length and angle requirements are selected as the linear framework of the grid, reducing the complexity of three-dimensional scene reconstruction from aerial images.

[0118] For example, the line segment matching module 52 is specifically used for:

[0119] Calculating the affinity between the feature line segment and the matching three-dimensional space line segment according to the angle similarity and the space similarity;

[0120] According to the affinity, based on the detection line confidence measurement function, a confidence measurement function value is calculated and a three-dimensional space line segment with the highest confidence measurement function value is used as the three-dimensional space position of the feature line segment;

[0121] For the three-dimensional spatial position, an affinity matrix is ​​generated based on a pairwise relationship or proximity-based replicator graph clustering strategy. For each item of the affinity matrix, an element with maximized consistency is queried within its visual neighborhood set to obtain a three-dimensional line cluster result of the feature line segment.

[0122] The affinity between the line segment and the possible position in the three-dimensional space is described by angular similarity and spatial similarity, and the most likely position is matched according to the affinity to obtain the three-dimensional line cluster result corresponding to the characteristic line segment, thereby reducing the complexity of three-dimensional scene reconstruction of aerial images.

[0123] In one example, the structure adjustment module 53 is specifically used for:

[0124] According to the camera pose information and the visibility relationship between the three-dimensional line cluster result and the aerial image, projecting each three-dimensional line segment in the three-dimensional line cluster result onto the grid model;

[0125] Calculate the vertical projection distance from each vertex of the triangular facet covered by the three-dimensional line segment to the three-dimensional line segment, and select the vertex with the shortest distance as the vertex to be adjusted;

[0126] A vertical vector from the vertex to be adjusted to the three-dimensional line segment is calculated, and the vertex to be adjusted is translated along the direction of the vertical vector until the vertical projection distance from the vertex to be adjusted to the three-dimensional line segment is equal to 0.

[0127] The projection of the three-dimensional line segment to the mesh model is completed through spatial topological constraints, and the vertex with the shortest vertical projection distance to the three-dimensional line segment is selected as the vertex to be adjusted for position adjustment, which reduces the complexity of three-dimensional scene reconstruction of aerial images.

[0128] This embodiment provides a cell fault analysis device, including: a line segment extraction module, which is used to extract feature line segments of aerial images according to a linear feature detection algorithm based on a straight line segment detection algorithm; a line segment matching module, which is used to match three-dimensional space line segments for each feature line segment of the aerial image, and cluster the three-dimensional space line segments to obtain three-dimensional line cluster results of the feature line segments; a structure adjustment module, which is used to project each three-dimensional line segment in the three-dimensional line cluster result onto a mesh model composed of triangular facets using a back-projection algorithm, and adjust the vertex positions of the triangular facets covered by the three-dimensional line segment according to the topological constraints of the three-dimensional line segment. By extracting feature line segments and matching three-dimensional space line segments, the mesh nodes are adjusted and optimized according to the three-dimensional topological constraints, thereby reducing the complexity of the reconstruction of the three-dimensional scene of the aerial image.

[0129] Embodiment 6

[0130] Figure 7 A schematic diagram of the structure of an electronic device provided in Example 6 of the present application is shown in FIG. Figure 7 As shown, the electronic device includes:

[0131] The electronic device includes a processor 291 and a memory 292; it may also include a communication interface 293 and a bus 294. The processor 291, the memory 292, and the communication interface 293 may communicate with each other through the bus 294. The communication interface 293 may be used for information transmission. The processor 291 may call the logic instructions in the memory 294 to execute the method of the above embodiment.

[0132] In addition, the logic instructions in the above-mentioned memory 292 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0133] The memory 292 is a computer-readable storage medium that can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the methods in the embodiments of the present application. The processor 291 executes functional applications and data processing by running the software programs, instructions, and modules stored in the memory 292, that is, implementing the methods in the above method embodiments.

[0134] The memory 292 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 292 may include a high-speed random access memory and may also include a non-volatile memory.

[0135] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in any embodiment.

[0136] An embodiment of the present application also provides a computer program product, including a computer program, which implements the method provided in the above embodiment when executed by a processor.

[0137] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0138] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for restoring a grid linear structure, characterized in that: include: According to the linear feature detection algorithm based on the straight line segment detection algorithm, the characteristic line segments of the aerial image are extracted; For each characteristic line segment of the aerial image, matching the characteristic line segment with a three-dimensional space line segment, and clustering the three-dimensional space line segment to obtain a three-dimensional line cluster result of the characteristic line segment; For each three-dimensional line segment in the three-dimensional line cluster result, a back-projection algorithm is used to project the three-dimensional line segment onto a mesh model composed of triangular facets, and the vertex positions of the triangular facets covered by the three-dimensional line segment are adjusted according to the topological constraints of the three-dimensional line segment; For each characteristic line segment of the aerial image, matching the characteristic line segment with a three-dimensional space line segment, and clustering the three-dimensional space line segment to obtain a three-dimensional line cluster result of the characteristic line segment includes: Calculating the affinity between the feature line segment and the matching three-dimensional space line segment according to the angle similarity and the space similarity; According to the affinity, based on the detection line confidence measurement function, a confidence measurement function value is calculated and a three-dimensional space line segment with the highest confidence measurement function value is used as the three-dimensional space position of the feature line segment; For the three-dimensional spatial position, an affinity matrix is ​​generated based on a pairwise relationship or proximity-based replicator graph clustering strategy. For each item of the affinity matrix, an element with maximized consistency is queried within its visual neighborhood set to obtain a three-dimensional line cluster result of the feature line segment.

2. The method according to claim 1, characterized in that The method of extracting characteristic line segments of the aerial image according to the linear feature detection algorithm based on the straight line segment detection algorithm includes: Calculating the gradient value and gradient direction of each pixel point in the aerial image, and pseudo-sorting the points according to the gradient values ​​of the points; For each pixel point of the aerial image, a regional diffusion algorithm is used to estimate and fit a line segment to obtain a line segment in the aerial image; Continuous line segments whose angle differences are not greater than a set angle threshold are selected from the line segments as the characteristic line segments.

3. The method according to claim 1, characterized in that The method of projecting each three-dimensional line segment in the three-dimensional line cluster result onto a mesh model composed of triangular facets by using a back-projection algorithm comprises: According to the camera pose information and the visibility relationship between the three-dimensional line cluster result and the aerial image, projecting each three-dimensional line segment in the three-dimensional line cluster result onto the grid model; Calculate the vertical projection distance from each vertex of the triangular facet covered by the three-dimensional line segment to the three-dimensional line segment, and select the vertex with the shortest distance as the vertex to be adjusted; A vertical vector from the vertex to be adjusted to the three-dimensional line segment is calculated, and the vertex to be adjusted is translated along the direction of the vertical vector until the vertical projection distance from the vertex to be adjusted to the three-dimensional line segment is equal to 0.

4. The method according to any one of claims 1 to 3, characterized in that: The characteristic line segments are represented by an undirected graph, the nodes in the undirected graph represent the pixel points on the aerial image, and the edges in the undirected graph represent the line segments connecting two pixel points.

5. A grid linear structure recovery device, characterized in that: include: A line segment extraction module is used to extract characteristic line segments of the aerial image according to a linear feature detection algorithm based on a straight line segment detection algorithm; A line segment matching module is used to match a three-dimensional space line segment for each characteristic line segment of the aerial image, and cluster the three-dimensional space line segment to obtain a three-dimensional line cluster result of the characteristic line segment; A structure adjustment module, for projecting each 3D line segment in the 3D line cluster result onto a mesh model composed of triangular facets using a back-projection algorithm, and adjusting the vertex positions of the triangular facets covered by the 3D line segment according to the topological constraints of the 3D line segment; The line segment matching module is specifically used to calculate the affinity between the feature line segment and the matching three-dimensional space line segment according to the angle similarity and the space similarity; According to the affinity, based on the detection line confidence measurement function, a confidence measurement function value is calculated and a three-dimensional space line segment with the highest confidence measurement function value is used as the three-dimensional space position of the feature line segment; For the three-dimensional spatial position, an affinity matrix is ​​generated based on a pairwise relationship or proximity-based replicator graph clustering strategy. For each item of the affinity matrix, an element with maximized consistency is queried within its visual neighborhood set to obtain a three-dimensional line cluster result of the feature line segment.

6. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 4 when executed by a processor.

8. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.