A method and device for correcting facade texture of street view image

Abstract

The present application provides a method and device for correcting the facade texture of a street view image, wherein the method includes: placing a preset quadrilateral grid network on the street view image to be corrected; detecting line segments in the image to be corrected; cutting the line segments using the upper and lower edges of the quadrilateral grids in the preset quadrilateral grid network; taking the weighted sum of the mean of the conformal energy of all quadrilateral grids, the mean of the vertical rigid energy of all line segments, and the mean of the line-related energy of all detected line segments as the minimum optimization target, determining the target similarity transformation matrix of each quadrilateral grid, and transforming the cut line segments in each quadrilateral grid according to the target similarity transformation of each quadrilateral grid. The three energy functions defined in the present application constrain the quadrilateral grids to transform so that the weighted sum of the three energy means is minimized, thereby straightening the curves in the street view image to be corrected and repairing the deformed curves in the street view image to be corrected.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method and device for correcting facade texture of street view images. Background Art

[0002] In the construction of new smart cities, digital twin cities have become a hot topic. Compared with traditional smart cities, digital twin cities integrate more technologies and means, and can be used as a comprehensive information carrier to depict urban details, present urban signs, and deduce urban trends in digital space. Constructing a three-dimensional urban building model is the basic step to realize a digital twin city.

[0003] In most applications, there is a strong demand for realistic modeling of urban buildings. However, the reconstructed three-dimensional urban building models are usually lacking in details, such as vegetation occlusion, image deformation, and image blur, which makes the building facades lack sufficiently accurate texture information and cannot achieve high-quality visualization of urban building models. Therefore, in order to achieve a refined expression of building facades, a method that can obtain high-quality building facade texture information is urgently needed.

[0004] The inventors found in their research that street view images, as a type of two-dimensional image data obtained by stitching together several images captured at the same time by a panoramic camera, can reflect the complete scene and record rich and clear texture information. Therefore, street view images are undoubtedly the best choice for achieving realistic expression of building facade appearance. However, due to its imaging method, street view images have significant deformation (bending deformation of straight line structures) in its wide field of view. Summary of the invention

[0005] The present application provides a method and a device, the purpose of which is to solve the problem.

[0006] In order to achieve the above objectives, this application provides the following technical solutions:

[0007] The present application provides a method for correcting facade texture of a street view image, comprising:

[0008] Placing a preset quadrilateral grid network on the street view image to be corrected; the quadrilateral grid network includes a plurality of quadrilateral grids;

[0009] Detecting line segments in the image to be corrected;

[0010] Using the upper and lower edges of the quadrilateral mesh in the preset quadrilateral mesh network, the line segment is cut to obtain a cut line segment;

[0011] The target similarity transformation matrix of each quadrilateral mesh is determined by taking the minimum weighted sum of the mean value of the conformal energy of all quadrilateral meshes, the mean value of the vertical rigid energy of all line segments and the mean value of the line correlation energy of all detected line segments as the optimization goal; wherein the conformal energy of any quadrilateral mesh is used to keep the quadrilateral mesh rigid before and after the similarity transformation; the vertical rigid energy of any line segment is used to keep the slope of the line segment before the similarity transformation close to the slope after the similarity transformation; the line correlation energy of any line segment is used to keep the position of the line segment before the similarity transformation relative to the quadrilateral mesh network before the similarity transformation consistent with the position of the line segment after the similarity transformation relative to the quadrilateral mesh network after the similarity transformation;

[0012] According to the target similarity transformation of each quadrilateral mesh, the cutting line segments in each quadrilateral mesh are transformed to obtain a deformed and restored street view image.

[0013] Optionally, the expression for the mean value of the conformal energy of all quadrilateral meshes is:

[0014] The expression of the mean value of the vertical rigid energy of all the line segments is:

[0015] The expression of the mean value of the line correlation energy of all line segments is:

[0016] Optionally, the sum of the weights corresponding to the mean of the conformal energy of all quadrilateral meshes, the mean of the vertical rigid energy of all line segments, and the mean of the line-related energy of all detected line segments is 1; and the weight of the mean of the vertical rigid energy of all line segments is the largest.

[0017] Optionally, taking the weighted sum of the mean of the conformal energy of all quadrilateral meshes, the mean of the vertical rigid energy of all line segments and the mean of the line-correlated energy of all detected line segments as the minimum optimization target, determining the target similarity transformation matrix of each quadrilateral mesh comprises:

[0018] The parameters of the quadrilateral mesh network are fixed, and the similarity transformation matrix of each quadrilateral mesh is optimized to obtain the optimal rotation matrix of each quadrilateral mesh;

[0019] The optimal rotation matrix of each quadrilateral mesh is fixed, and the parameters of the quadrilateral mesh network are optimized.

[0020] Optionally, the parameters of the quadrilateral mesh network are fixed, and the similarity transformation matrix of each quadrilateral mesh is optimized to obtain the optimal rotation matrix of each quadrilateral mesh, including:

[0021] With the constraint that the same line segment divided in multiple quadrilateral meshes and the divided line segments with similar directions in different quadrilateral meshes still maintain similar rotation angles after transformation, the similarity transformation matrix of each quadrilateral mesh is optimized to obtain the optimal rotation matrix of each quadrilateral mesh.

[0022] Optionally, before the method uses the upper and lower edges of the quadrilateral meshes in the preset quadrilateral mesh network to cut the line segments to obtain the cut line segments, the method further includes:

[0023] receiving a vertical line segment manually located from the street view image to be corrected;

[0024] The detecting of line segments in the image to be corrected is specifically:

[0025] Using an LSD line segment detection algorithm to detect line segments in the street view image to be corrected;

[0026] The method of cutting the line segments by using the upper and lower edges of the quadrilateral mesh in the preset quadrilateral mesh network to obtain the cut line segments is specifically as follows:

[0027] All line segments are cut using the upper and lower edges of the quadrilateral meshes in the preset quadrilateral mesh network to obtain cut line segments.

[0028] Optionally, after the cutting line segments in each of the quadrilateral meshes are transformed according to the target similarity transformation of each of the quadrilateral meshes to obtain the deformed and restored street view image, the method further includes:

[0029] In the case where the deformed and restored street view image is distorted, obtaining a distortion-to-be-corrected area of ​​the deformed and restored street view image;

[0030] Determine the coordinates of each pixel point in the distortion-to-be-corrected area on a virtual panoramic spherical surface, and obtain the virtual panoramic spherical surface coordinates of each pixel point in the distortion-to-be-corrected area; the virtual panoramic sphere is a virtual panoramic sphere corresponding to the deformed and restored street view image;

[0031] Determine the panoramic spherical geographic coordinates of each pixel point in the distortion to be corrected area according to the virtual panoramic spherical coordinates of each pixel point in the distortion to be corrected area;

[0032] Obtaining the coordinates of the tangent points between the projection plane of the deformed and restored street view image and the virtual panoramic sphere; the projection plane is a plane parallel to the facade of the building in the deformed and restored street view image;

[0033] According to the panoramic sphere geographic coordinates of each pixel point in the distortion-to-be-corrected area, the virtual panoramic spherical coordinates, the tangent point coordinates and a preset perspective mathematical model, the coordinates of each pixel point in the distortion-to-be-corrected area projected on the projection plane are calculated; the perspective mathematical model is a relationship between the panoramic sphere geographic coordinates and the panoramic sphere equatorial tangent plane coordinates obtained by projecting rays emitted from the center of the panoramic sphere onto the equatorial tangent plane of the panoramic sphere.

[0034] Optionally, the tangent point between the projection plane of the deformed and restored street view image and the virtual panoramic sphere includes: a boundary inflection point of a road feature in the deformed and restored street view image.

[0035] Optionally, the horizontal field angle of the area to be corrected for distortion is not greater than 120°.

[0036] Optionally, after calculating the coordinates of each pixel point in the distortion-to-be-corrected area projected on the projection plane according to the panoramic spherical geographic coordinates of each pixel point in the distortion-to-be-corrected area, the virtual panoramic spherical coordinates, the tangent point coordinates and a preset perspective mathematical model, the method further includes:

[0037] According to the projection coordinates of the boundary corner points of the distortion-to-be-corrected area on the projection plane, the projected coordinates of each pixel point in the distortion-to-be-corrected area are translated to obtain a processed street view image.

[0038] Optionally, after translating the projected coordinates of each pixel point in the distortion-to-be-corrected area according to the projection coordinates of the boundary corner points of the distortion-to-be-corrected area on the projection plane to obtain a processed street view image, the method further includes:

[0039] The processed street view image is used as a facade texture data source to perform facade texture mapping.

[0040] The present application also provides a device for correcting facade texture of a street view image, comprising:

[0041] A placement module, used for placing a preset quadrilateral grid network on the street view image to be corrected; the quadrilateral grid network includes a plurality of quadrilateral grids;

[0042] A detection module, used for detecting line segments in the image to be corrected;

[0043] A cutting module, used for cutting the line segment by using the upper and lower edges of the quadrilateral grid in the preset quadrilateral grid network to obtain a cutting line segment;

[0044] A determination module is used to determine the target similarity transformation matrix of each quadrilateral mesh with the minimum weighted sum of the mean value of the conformal energy of all quadrilateral meshes, the mean value of the vertical rigid energy of all line segments and the mean value of the line correlation energy of all detected line segments as the optimization target; wherein the conformal energy of any quadrilateral mesh is used to make the quadrilateral mesh rigid before and after the similarity transformation; the vertical rigid energy of any line segment is used to make the slope of the line segment before the similarity transformation close to the slope after the similarity transformation; the line correlation energy of any line segment is used to make the position of the line segment before the similarity transformation relative to the quadrilateral mesh network before the similarity transformation consistent with the position of the line segment after the similarity transformation relative to the quadrilateral mesh network after the similarity transformation;

[0045] The transformation module is used to transform the cut line segments in each of the quadrilateral meshes according to the target similarity transformation of each of the quadrilateral meshes to obtain a deformed and repaired street view image. The method and device for correcting textures in street view images described in the present application place a preset quadrilateral mesh network on the street view image to be corrected; detect the line segments in the image to be corrected; use the upper and lower edges of the quadrilateral meshes in the preset quadrilateral mesh network to cut the line segments to obtain the cut line segments; determine the target similarity transformation matrix of each quadrilateral mesh with the minimum weighted sum of the mean of the conformal energy of all quadrilateral meshes, the mean of the vertical rigid energy of all line segments, and the mean of the line-related energy of all line segments as the optimization target, transform the cut line segments in each quadrilateral mesh according to the target similarity transformation of each quadrilateral mesh, and obtain a deformed and repaired street view image.

[0046] In the present application, during the similarity transformation of the quadrilateral mesh, since the shape-preserving energy function, the vertical rigidity energy function and the line-correlation energy function are defined, these three energy functions constrain the quadrilateral mesh to be transformed so that the weighted sum of the three energy means is minimized, thereby straightening the curves in the street view image to be corrected, thereby achieving the repair of the deformed curves in the street view image to be corrected.

[0047] In addition, in this embodiment, the three energies also constrain the vertex positions of the quadrilateral mesh to not change arbitrarily. While repairing the curves in the facade structure of the street view image to be corrected, the shape of the related neighborhood of the quadrilateral mesh is kept unchanged. Therefore, while repairing the curves on the facade of the street view image to be corrected, the image features of the street view image are maintained as much as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A flowchart of a method for correcting facade texture of a street view image disclosed in an embodiment of the present application;

[0050] FIG. 2( a ) is an example diagram of a street view image to be corrected disclosed in an embodiment of the present application;

[0051] FIG2( b ) is an example diagram of a street view image after a quadrilateral grid network is placed on the street view image to be corrected according to an embodiment of the present application;

[0052] FIG2( c ) is a schematic diagram of a street view image after detecting line segments from the street view image to be corrected shown in FIG2( a ) according to an embodiment of the present application;

[0053] FIG2( d ) is a schematic diagram of the result of similarity transformation of a quadrilateral mesh network disclosed in an embodiment of the present application;

[0054] FIG2( e ) is a schematic diagram of a street view image after similarity transformation disclosed in an embodiment of the present application;

[0055] FIG3( a ) is a schematic diagram of a distorted image to be corrected according to an embodiment of the present application;

[0056] FIG3( b ) is a schematic diagram of an image after distortion correction disclosed in an embodiment of the present application;

[0057] Figure 4 A flowchart of a process of correcting distortion of an area to be corrected in a street view image disclosed in an embodiment of the present application;

[0058] Figure 5 A schematic structural diagram of a device for correcting facade texture of a street view image is disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0060] Figure 1A method for correcting the facade texture of a street view image provided in an embodiment of the present application may include the following steps:

[0061] S101: placing a preset quadrilateral grid network on the street view image to be corrected.

[0062] In this embodiment, the preset quadrilateral mesh network is a network composed of quadrilateral meshes formed by the intersection of multiple parallel lines and multiple vertical lines. In practice, the quadrilateral mesh network may include 20×40 quadrilateral meshes. Of course, in practice, the number of quadrilateral meshes included in the quadrilateral mesh network may also be other values, and this embodiment does not limit the specific value.

[0063] In this embodiment, the quadrilateral mesh network is parameterized as V = {P, F}, where P = {p i (x i ,y i )}(i=1,2,3...861), indicating the coordinate position of the quadrilateral mesh vertex. i}(j=1,2,3…800) represents the face of the quadrilateral mesh.

[0064] In this embodiment, in order to intuitively display the street view image to be corrected, an example of the street view image to be corrected is given as shown in Figure 2(a). Figure 2(b) is a street view image after a quadrilateral grid network is placed on the street view image to be corrected.

[0065] S102: Detect line segments in the street view image to be corrected.

[0066] In this embodiment, an LSD line segment detection algorithm may be used to detect line segments in the street view image to be corrected, wherein the specific implementation process of the detection is an existing technology and will not be described in detail here.

[0067] In this embodiment, the line segment set obtained in this step is represented as L = {l i (o i ,e i )}(i=1,2,3...n), where o represents the starting point of the line segment and e represents the end point of the line segment.

[0068] FIG. 2( c ) is a street view image provided by this embodiment after line segments are detected from the street view image to be corrected shown in FIG. 2( a ).

[0069] S103 , using the upper and lower edges of the quadrilateral mesh in the preset quadrilateral mesh network to cut the detected line segments to obtain cut line segments.

[0070] In this step, the detected line segments are cut. For the convenience of description, the line segments obtained by cutting are called cut line segments.

[0071] In this embodiment, a cutting line segment is completely located in a quadrilateral grid.

[0072] S104: Associating the information of the cutting line segment with the corresponding quadrilateral mesh.

[0073] In this embodiment, the information of the cutting line segment may include: position information and endpoint information.

[0074] In this step, the information of the cutting line segments is respectively associated with the corresponding quadrilateral meshes to obtain the association relationship between the quadrilateral meshes and the cutting line segment information.

[0075] S105 , determining a target similarity transformation matrix of each quadrilateral mesh by performing similarity transformation on the quadrilateral meshes.

[0076] In this embodiment, the target similarity transformation matrix refers to a similarity transformation matrix that minimizes the weighted sum of multiple preset rigidity preserving energy terms.

[0077] In this embodiment, three rigidity-preserving energy terms are constructed, namely: shape-preserving energy function, vertical rigidity energy function and line-dependent energy function. The functions and specific expressions of these three energy functions are introduced below.

[0078] Among them, the shape-preserving energy function is used to maintain the rigidity of the quadrilateral mesh, that is, to try to keep the original shape of the quadrilateral mesh under similarity transformation.

[0079] The expression of the shape-preserving energy function is shown in the following formula (1):

[0080]

[0081] In the formula, E r (p') represents the shape-preserving energy of the quadrilateral mesh network after similarity transformation, p' represents the vertex coordinates of the quadrilateral mesh surface after similarity transformation, N represents the total number of quadrilateral meshes in the quadrilateral mesh network, f represents the index of the quadrilateral mesh surface, A p Represents an 8*4 matrix consisting of the original coordinates of the four vertices of the quadrilateral mesh, P f ' represents an 8*1 vector consisting of the coordinates of the four vertices of the quadrilateral mesh after similarity transformation, and I represents the unit matrix.

[0082] Among them, A p and P f The specific expression of ' is

[0083] Among them, (x1, y1), (x2, y2), (x3, y3), (x4, y4) represent the original coordinates of the four vertices of the quadrilateral mesh surface. (x1', y1'), (x'2, y'2), (x'3, y'3), (x'4, y'4) represent the coordinates of the four vertices of the quadrilateral mesh surface after similarity transformation.

[0084] The applicant has discovered that since the quadrilateral mesh may destroy certain line structure properties of the street view image when applying a similarity transformation, in order to preserve the line structure properties (i.e., to ensure that the slopes of multiple segmented line segments obtained by segmenting the detected line segment by the quadrilateral mesh remain consistent), this embodiment proposes a vertical rigid energy function to perform vertical rigid constraints on the line segments within the quadrilateral mesh network.

[0085] The expression of the vertical rigid energy function is shown in the following formula (2):

[0086]

[0087] In the formula, l represents a line segment, Represents the total number of line segments, R l Represents a known rotation matrix, ψ represents the direction vector of the line segment, ψ' represents the direction vector of the line segment after similarity transformation, and I represents the identity matrix.

[0088] Where ψ = eo, and the scaling factor of the line segment is

[0089] In practice, since some hard contours in the street view image to be corrected are easily confused by other buildings in the background, in this embodiment, the vertical line segments can also be interactively selected and located by manually using a tool similar to a lasso in the existing image editing software. In this embodiment, in order to maintain the smooth change of the nodes obtained by segmenting the detected line segments by the quadrilateral grid, the vertical rigid energy calculated in formula (2) includes the vertical rigid energy of the vertical line segments, that is, l in formula (2) includes both the detected line segments and the vertical line segments. That is, before S103, it also includes: receiving the vertical line segments manually located from the street view image to be corrected. In this case, S103 is to use the upper and lower edges of the quadrilateral grid in the preset quadrilateral grid network to cut all the line segments (the detected line segments and the vertical line segments manually located) to obtain the cut line segments. Calculate the average value of the vertical rigid energy of all the line segments (including the detected line segments and the vertical line segments manually located).

[0090] In this embodiment, in order to maintain the parallel rigidity of the line segment, the same line segment across multiple quadrilateral meshes is constrained, and the segmented line segments with similar directions in different quadrilateral meshes still maintain the same (similar) rotation angle θ after transformation. l.

[0091] In this embodiment, the line-correlated energy function is used to ensure that the position of the detected line segment relative to the original quadrilateral mesh network before the similarity transformation is consistent with the position of the detected line segment relative to the quadrilateral mesh network after the similarity transformation.

[0092] In this embodiment, the expression of the line correlation energy function is shown in formula (3):

[0093]

[0094] Wherein, o represents the starting point of the detected line segment before the similarity transformation, e represents the end point of the detected line segment before the similarity transformation, o' represents the starting point of the detected line segment after the similarity transformation, e' represents the end point of the detected line segment after the similarity transformation, and p' represents the vertex coordinates of the quadrilateral mesh after the similarity transformation. represents the total number of line segments, P o ' represents the coordinates of the starting point in the vertex of the quadrilateral mesh after similarity transformation, P e ' represents the coordinates of the end point in the vertex of the quadrilateral mesh after similarity transformation. o and A e The coefficient vector that forms the bilinear interpolation function.

[0095] In this embodiment, the coordinates of the two endpoints o and e of the detected line segment before the similarity transformation can be represented by the bilinear interpolation of the four vertices p' of the transformed quadrilateral mesh.

[0096] In this embodiment, the values ​​of the above three energy functions under similarity transformation are weighted and summed, and the specific expression is shown in formula (4):

[0097] E=λ r E r +λ v E v +λ c E c (4)

[0098] In this embodiment, since the human eye is more sensitive to curved straight lines than distorted shapes, better restoration results can be obtained when the importance of maintaining the line structure is higher than the importance of maintaining the shape. Therefore, in this embodiment, a greater weight is given to the vertical rigid energy function.

[0099] Optionally, in this embodiment, the weight value corresponding to the shape-preserving energy function is 0.1, the weight value corresponding to the vertical rigidity energy function is 0.4, and the weight value corresponding to the line-correlated energy function is 0.1. Of course, in practice, the weight values ​​corresponding to the three energy functions can also be other values, and this embodiment does not limit the specific values ​​of the weights.

[0100] In this step, the target similarity transformation matrix of each quadrilateral mesh can be determined by iteratively executing the following steps A1 to A2:

[0101] A1. Fix the parameters of the quadrilateral mesh network, optimize the similarity transformation matrix of each quadrilateral mesh, and obtain the optimal rotation matrix of each quadrilateral mesh.

[0102] In this embodiment, the rotation matrix R in each direction l are independent of each other. In the above formulas (1), (2) and (3), each term in the summation part of each formula can be considered independent. Therefore, the θ of each term in the summation part is l Independent of each other, each term can be optimized separately l , find the optimal common rotation θ for each quadrilateral mesh l , that is, the optimal rotation matrix of each quadrilateral mesh is obtained.

[0103] Specifically, in the optimization process of this step, the similarity transformation matrix of each quadrilateral mesh can be optimized by taking the same line segment divided in multiple quadrilateral meshes and the segmented line segments with similar directions in different quadrilateral meshes as the constraint that they still maintain similar rotation angles after transformation. The similarity transformation matrix obtained by optimization is decomposed into scaling and rotation, where the rotation is R l The optimal solution of , thus, the optimal rotation matrix of each quadrilateral mesh is obtained.

[0104] A2. Fix the optimal rotation matrix and optimize the parameters of the quadrilateral mesh network.

[0105] In this embodiment, when the optimal rotation matrix R l When it is known and fixed, the total energy E is a quadratic function on the quadrilateral mesh network. The optimization of this step is a standard least squares problem. The specific optimization details will not be repeated here.

[0106] FIG2( d ) shows a schematic diagram of the result after similarity transformation of the quadrilateral mesh network according to the operation of this step.

[0107] S106: transform the street view image to be corrected according to the target similarity transformation of each quadrilateral grid.

[0108] In this step, according to the target similarity transformation of each quadrilateral mesh, the cutting line segments in each quadrilateral mesh are transformed to achieve deformation repair of the street view image to be corrected, and obtain a deformed repaired street view image. Among them, according to the target similarity transformation of the deformed grid, the specific implementation of transforming the cutting line segments in each quadrilateral mesh is a prior art and will not be repeated here.

[0109] For the street scene image to be corrected shown in FIG2(a), after the operation of this embodiment is performed, the street scene image after similarity transformation is obtained as shown in FIG2(e).

[0110] In this embodiment, during the similarity transformation of the quadrilateral mesh, since the shape-preserving energy function, the vertical rigidity energy function and the line-correlated energy function are defined, these three energy functions constrain the vertex positions of the quadrilateral mesh and cannot be changed arbitrarily. While repairing the curves in the facade structure of the street view image to be corrected, the shape of the related neighborhood of the quadrilateral mesh is kept unchanged. Therefore, while repairing the curves on the facade of the street view image to be corrected, the image features of the street view image are kept as much as possible.

[0111] In the embodiment of the present application, due to the imaging method of equidistant cylindrical projection, the street view image is distorted (the street view image introduces severe stretching deformation, resulting in very high distortion). In order to intuitively show the effect comparison before and after distortion correction, taking Figure 3(a) and Figure 3(b) as examples, Figure 3(a) is the distorted image to be corrected, and Figure 3(b) is the image after distortion correction.

[0112] In the embodiment of the present application, each distorted street view image corresponds to a virtual panoramic sphere, and the entire circumference of the virtual panoramic sphere is the width of the plane street view image. If the radius of the panoramic sphere is R, and the width of the plane street view image is W, then R = W / 2Π. According to the equidistant cylindrical projection model, any point on the virtual panoramic sphere The mapping relationship between the expanded 2D street view image and any point (m, n) is as shown in the following formula (5):

[0113]

[0114] This embodiment establishes a panoramic spherical geographic coordinate system, using θ E Indicates the longitude of the coordinate system, using Indicates the latitude of the coordinate system. The latitude of the coordinate system is expressed by the angle between the line connecting a point on the virtual panoramic sphere and the center of the sphere and the XOY plane, with north as the positive direction. relationship.

[0115] In this embodiment, a general formula of spherical center projection is obtained by projecting rays emitted from the center of the panoramic sphere onto a plane tangent to the equator of the panoramic sphere, and the formula is used as a perspective mathematical model of the corresponding relationship between any point (m, n) on the plane street view image and the corresponding point (x, y) on the projection plane. The specific expression of the perspective mathematical model is shown in the following formula (6):

[0116]

[0117] In the formula, r represents the radius of the panoramic sphere, θ represents the longitude of the virtual panoramic sphere, and θ E represents the longitude in the pansphere geographic coordinate system, Represents the latitude in the pansphere geographic coordinate system, The coordinates of the tangent point between the projection plane of the area to be corrected and the panoramic sphere.

[0118] Therefore, for the area to be corrected on the street view image, when the projection plane and the tangent point of the area to be corrected are known, the projected coordinates of each pixel in the area to be corrected can be obtained by formula (6), and the area after distortion correction can be obtained. The process of performing distortion correction on the area to be corrected in the street view image in this embodiment is as follows: Figure 4 As shown, the following steps may be included:

[0119] S401: Obtain a distorted street view image to be corrected.

[0120] In this embodiment, the distorted image to be corrected may be Figure 1 The deformed and repaired street view image obtained in the corresponding embodiment. Of course, in practice, the distorted street view image to be corrected can also be other distorted street view images, not necessarily street view images obtained after deformation and repair. This embodiment does not limit the specific content of the distorted street view image to be corrected.

[0121] This example assumes that Figure 1 The deformed and restored street view image obtained in the corresponding embodiment is distorted. Figure 1 The deformed repaired image obtained in the corresponding embodiment is a distorted street scene image to be corrected, which is introduced below.

[0122] S402: Obtain the coordinates of the tangent points between the projection plane of the deformation-corrected street view image and the virtual panoramic sphere.

[0123] In this embodiment, the selected projection plane needs to be roughly parallel to the building facade, and the distortion correction effect is determined by the tangent point of the selected projection plane. In this embodiment, the tangent point can be the boundary inflection point of the road feature. When the feature in the street view image is blocked, the tangent point can also be determined by approximating the building facade.

[0124] In this step, the obtained tangent point coordinates can be expressed as

[0125] In practice, important objects in street view images are generally concentrated in the middle of the street view images, which is equivalent to the "equator" position of the virtual panoramic sphere. Therefore, this embodiment assumes that the sensor is horizontal and the tangent point is located at the midline of the street view image, that is, Therefore, we only need to determine the horizontal coordinate θ of the tangent point. EO Among them, the horizontal coordinate of the tangent point is θ EOThe coordinates of the corner points of the distortion-to-be-corrected area on the deformed restored street view image can be determined, wherein the specific determination process is prior art and will not be described in detail here. As an example, the distortion-to-be-corrected area is shown as the framed area in FIG3(a).

[0126] S403: Obtain a distortion-to-be-corrected area of ​​the deformed and restored street view image.

[0127] In this embodiment, the coordinates of the four corner points of the distortion to be corrected area can be expressed as

[0128] Optionally, in this embodiment, the distortion area to be corrected can be selected manually. Of course, in practice, in addition to manually selecting the distortion area to be corrected, it can also be selected automatically by a machine. This embodiment does not limit the specific selection method of the distortion area to be corrected.

[0129] S404: Determine whether the horizontal field angle of the distortion-to-be-corrected area is greater than 120°. If not, execute S405; if yes, execute S403.

[0130] In this embodiment, when the horizontal field of view angle of the distortion-to-be-corrected area is greater than 120°, there is edge stretching of the street view image and unnecessary upsampling and sampling of the ground area. Therefore, this embodiment requires a distortion-to-be-corrected area with a horizontal field of view angle of not more than 120° in the horizontal and vertical fields of view on the street view image.

[0131] It should be noted that, in this embodiment, this step is an optional step. That is, in practice, after obtaining the distortion area to be corrected in S403, S405 is directly executed. Only executing this step can make the final distortion repair effect better.

[0132] S405 : Determine the coordinates of each pixel point in the distortion-to-be-corrected area on the virtual panoramic spherical surface, and obtain the virtual panoramic spherical surface coordinates of each pixel point in the distortion-to-be-corrected area.

[0133] In this embodiment, the virtual panoramic sphere is a virtual panoramic sphere corresponding to the deformed and restored street view image.

[0134] In this step, the method for determining the coordinates of each pixel point in the distortion-to-be-corrected area on the virtual panoramic sphere is the same. For the convenience of description, an arbitrary pixel point in the distortion-to-be-corrected area is used as an example for description. Specifically, the coordinates of the pixel point in the distortion-to-be-corrected area in the deformed and restored street view image are used as (m, n) in formula (5), and the coordinates of the pixel point in the distortion-to-be-corrected area on the virtual panoramic sphere are calculated according to formula (5).

[0135] S406 , determining the panoramic spherical geographic coordinates of each pixel point in the distortion-to-be-corrected area according to the virtual panoramic spherical coordinates of each pixel point in the distortion-to-be-corrected area.

[0136] In this embodiment, the method of determining the panoramic spherical geographic coordinates of each pixel point in the distortion-to-be-corrected area is the same. For the convenience of description, an arbitrary pixel point in the distortion-to-be-corrected area is taken as an example for description. Specifically, the panoramic spherical geographic coordinates of the pixel point in the distortion-to-be-corrected area include longitude and latitude, wherein the longitude can be expressed as θ E Indicates that latitude can be expressed as express.

[0137] Determine the latitude of the pixel in the area to be corrected The methods include: That is, the latitude of the pixel point in the distortion-to-be-corrected area on the virtual panoramic sphere is used Determine the latitude of the pixel in the panoramic sphere geographic coordinate system

[0138] S407, calculating the coordinates of each pixel point in the distortion-to-be-corrected area projected on the projection plane according to the panoramic spherical geographic coordinates, virtual panoramic spherical coordinates, tangent point coordinates and a preset perspective mathematical model of each pixel point in the distortion-to-be-corrected area.

[0139] In this step, the perspective mathematical model is obtained by projecting rays from the center of the panoramic sphere onto the equatorial section of the panoramic sphere, and obtaining the relationship between the geographical coordinates of the panoramic sphere and the coordinates of the equatorial section of the panoramic sphere, that is, the perspective mathematical model given by formula (6).

[0140] In this step, based on formula (6), the projection coordinates of each pixel point in the distortion-to-be-corrected area on the projection plane can be calculated. Taking a pixel point as an example, the projection coordinates (x, y) of the pixel point can be obtained by formula (6).

[0141] S408 , translating the projected coordinates of each pixel point in the distortion-to-be-corrected area according to the projection coordinates of the boundary corner points of the distortion-to-be-corrected area on the projection plane.

[0142] Optionally, in this embodiment, the distortion area to be corrected of the planar street view image is centrally symmetrically distributed with the tangent point, and the coordinate index of the street view image pixel is a non-negative value, therefore, the projected coordinates of each pixel point in the distortion area to be corrected can be translated according to the projection coordinates of the boundary corner points of the distortion area to be corrected on the projection plane. The specific implementation method of this step is the existing technology and will not be described in detail here.

[0143] In this embodiment, the characteristic of spherical center projection, that is, any great circle is projected as a straight line, is used to correct the distortion of the image. Specifically, the relationship between the projection plane and the equator of the panoramic sphere is used to map the points on the panoramic sphere surface to the projection plane starting from the center of the panoramic sphere, keeping the mapping angle unchanged. Therefore, in this embodiment, as long as the selected projection plane is roughly parallel to the facade of the building, the distortion of the street view image can be reduced after the projection conversion. In addition, after the estimation of the appropriate tangent point and the determination of the projection target area range (distortion to be corrected area), the street view image is corrected for distortion.

[0144] Through the above-mentioned deformation repair and distortion correction, a processed street view image is obtained. In the embodiment of the present application, the processed street view image can be used as a facade texture data source to perform facade texture mapping to obtain a realistic and semantically perceptual realistic building facade model. Among them, the facade texture mapping can be implemented in LoD-3. Of course, in practice, the specific implementation of the facade texture mapping can also be other methods. This embodiment does not limit the specific implementation of the facade texture mapping.

[0145] Figure 5 The apparatus for correcting the facade texture of a street view image provided in the embodiment of the present application includes: a placement module 501, a detection module 502, a cutting module 503, a determination module 504 and a transformation module 505, wherein:

[0146] The placement module 501 is used to place a preset quadrilateral mesh network on the street view image to be corrected; the quadrilateral mesh network includes a plurality of quadrilateral meshes.

[0147] The detection module 502 is used to detect line segments in the image to be corrected.

[0148] The cutting module 503 is used to cut the line segment by using the upper and lower edges of the quadrilateral meshes in the preset quadrilateral mesh network to obtain a cut line segment.

[0149] The determination module 504 is used to determine the target similarity transformation matrix of each quadrilateral mesh by taking the weighted sum of the mean value of the conformal energy of all quadrilateral meshes, the mean value of the vertical rigid energy of all line segments and the mean value of the line correlation energy of all detected line segments as the minimum optimization target; wherein the conformal energy of any quadrilateral mesh is used to make the quadrilateral mesh remain rigid before and after the similarity transformation; the vertical rigid energy of any line segment is used to make the slope of the line segment before the similarity transformation close to the slope after the similarity transformation; the line correlation energy of any line segment is used to make the position of the line segment before the similarity transformation relative to the quadrilateral mesh network before the similarity transformation and the position of the line segment after the similarity transformation relative to the quadrilateral mesh network after the similarity transformation consistent.

[0150] The transformation module 505 is used to transform the cutting line segments in each of the quadrilateral meshes according to the target similarity transformation of each of the quadrilateral meshes to obtain a deformed and restored street view image.

[0151] Optionally, the expression for the mean value of the conformal energy of all quadrilateral meshes is: The expression of the mean value of the vertical rigid energy of all the line segments is: The expression of the mean value of the line correlation energy of all line segments is:

[0152] Optionally, the sum of the weights corresponding to the mean of the conformal energy of all quadrilateral meshes, the mean of the vertical rigid energy of all line segments, and the mean of the line-related energy of all detected line segments is 1; and the weight of the mean of the vertical rigid energy of all line segments is the largest.

[0153] Optionally, the determination module 504 is used to determine the target similarity transformation matrix of each quadrilateral mesh by taking the weighted sum of the mean value of the conformal energy of all quadrilateral meshes, the mean value of the vertical rigid energy of all line segments and the mean value of the line correlation energy of all detected line segments as the optimization target, including:

[0154] The determination module 504 is specifically used to fix the parameters of the quadrilateral mesh network, optimize the similarity transformation matrix of each quadrilateral mesh, and obtain the optimal rotation matrix of each quadrilateral mesh; fix the optimal rotation matrix of each quadrilateral mesh and optimize the parameters of the quadrilateral mesh network.

[0155] Optionally, the determination module 504 is used to fix the parameters of the quadrilateral mesh network, optimize the similarity transformation matrix of each quadrilateral mesh, and obtain the optimal rotation matrix of each quadrilateral mesh, including:

[0156] Determination module 504 is specifically used to optimize the similarity transformation matrix of each quadrilateral mesh based on the constraint that the same line segment divided in multiple quadrilateral meshes and the divided line segments with similar directions in different quadrilateral meshes still maintain similar rotation angles after transformation, so as to obtain the optimal rotation matrix of each quadrilateral mesh.

[0157] Optionally, the device may further include:

[0158] A receiving module, configured to receive a vertical line segment manually located from the street view image to be corrected before the cutting module 503 cuts the line segment using the upper and lower edges of the quadrilateral grid in the preset quadrilateral grid network to obtain the cut line segment;

[0159] The detection module 502 is used to detect line segments in the image to be corrected, specifically:

[0160] The detection module 502 is specifically configured to detect line segments in the street view image to be corrected by using an LSD line segment detection algorithm;

[0161] The cutting module 503 is used to cut the line segment using the upper and lower edges of the quadrilateral mesh in the preset quadrilateral mesh network to obtain the cut line segment, specifically:

[0162] The cutting module 503 is specifically used to cut all line segments using the upper and lower edges of the quadrilateral meshes in the preset quadrilateral mesh network to obtain cut line segments.

[0163] Optionally, the device may further include: a distortion correction module, configured to transform the cutting line segments in each of the quadrilateral meshes according to the target similarity transformation of each of the quadrilateral meshes in the transformation module 505 to obtain the deformed and restored street view image, and then, if the deformed and restored street view image is distorted, obtain a distortion-to-be-corrected area of ​​the deformed and restored street view image;

[0164] Determine the coordinates of each pixel point in the distortion-to-be-corrected area on a virtual panoramic spherical surface, and obtain the virtual panoramic spherical surface coordinates of each pixel point in the distortion-to-be-corrected area; the virtual panoramic sphere is a virtual panoramic sphere corresponding to the deformed and restored street view image;

[0165] Determine the panoramic spherical geographic coordinates of each pixel point in the distortion to be corrected area according to the virtual panoramic spherical coordinates of each pixel point in the distortion to be corrected area;

[0166] Obtaining the coordinates of the tangent points between the projection plane of the deformed and restored street view image and the virtual panoramic sphere; the projection plane is a plane parallel to the facade of the building in the deformed and restored street view image;

[0167] According to the panoramic sphere geographic coordinates of each pixel point in the distortion-to-be-corrected area, the virtual panoramic spherical coordinates, the tangent point coordinates and a preset perspective mathematical model, the coordinates of each pixel point in the distortion-to-be-corrected area projected on the projection plane are calculated; the perspective mathematical model is a relationship between the panoramic sphere geographic coordinates and the panoramic sphere equatorial tangent plane coordinates obtained by projecting rays emitted from the center of the panoramic sphere onto the equatorial tangent plane of the panoramic sphere.

[0168] Optionally, the tangent point between the projection plane of the deformed and restored street view image and the virtual panoramic sphere obtained by the distortion restoration module includes: a boundary inflection point of a road feature in the deformed and restored street view image.

[0169] Optionally, the horizontal field angle of the distortion to be corrected area obtained by the distortion repair module is no greater than 120°.

[0170] Optionally, the device may also include: a translation module, which is used to calculate the coordinates of each pixel point in the distortion area to be corrected projected on the projection plane based on the panoramic spherical geographic coordinates, virtual panoramic spherical coordinates, the tangent point coordinates and a preset perspective mathematical model of each pixel point in the distortion area to be corrected in the distortion repair module, and then translate the projected coordinates of each pixel point in the distortion area to be corrected according to the projection coordinates of the boundary corner points of the distortion area to be corrected on the projection plane to obtain a processed street view image.

[0171] Optionally, the device may also include: a facade texture mapping module, which is used to translate the projected coordinates of each pixel point in the distortion area to be corrected according to the projection coordinates of the boundary corner points of the distortion area to be corrected on the projection plane in the distortion repair module, and after obtaining the processed street view image, use the processed street view image as the facade texture data source to perform facade texture mapping.

[0172] If implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium readable by a computing device. Based on this understanding, the part of the embodiment of the present application that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computing device (which can be a personal computer, server, mobile computing device or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program code.

[0173] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0174] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for correcting facade texture of a street view image, characterized in that: include: placing a preset quadrilateral grid network on the street view image to be corrected; The quadrilateral mesh network includes a plurality of quadrilateral meshes; Detecting line segments in the street view image to be corrected; Using the upper and lower edges of the quadrilateral mesh in the preset quadrilateral mesh network, the line segment is cut to obtain a cut line segment; The target similarity transformation matrix of each quadrilateral mesh is determined by taking the minimum weighted sum of the mean value of the conformal energy of all quadrilateral meshes, the mean value of the vertical rigid energy of all line segments and the mean value of the line correlation energy of all detected line segments as the optimization goal; wherein the conformal energy of any quadrilateral mesh is used to keep the quadrilateral mesh rigid before and after the similarity transformation; the vertical rigid energy of any line segment is used to keep the slope of the line segment before the similarity transformation close to the slope after the similarity transformation; the line correlation energy of any line segment is used to keep the position of the line segment before the similarity transformation relative to the quadrilateral mesh network before the similarity transformation consistent with the position of the line segment after the similarity transformation relative to the quadrilateral mesh network after the similarity transformation; According to the target similarity transformation of each quadrilateral mesh, the cutting line segments in each quadrilateral mesh are transformed to obtain a deformed and restored street view image; Wherein, before the cutting module cuts the line segment using the upper and lower edges of the quadrilateral mesh in the preset quadrilateral mesh network to obtain the cut line segment, it also includes: receiving a vertical line segment manually located from the street view image to be corrected; The detecting of line segments in the street view image to be corrected is specifically: Using an LSD line segment detection algorithm to detect line segments in the street view image to be corrected; The method of cutting the line segments by using the upper and lower edges of the quadrilateral mesh in the preset quadrilateral mesh network to obtain the cut line segments is specifically as follows: Using the upper and lower edges of the quadrilateral meshes in the preset quadrilateral mesh network, all line segments are cut to obtain cut line segments; The sum of the weights corresponding to the mean of the conformal energy of all quadrilateral meshes, the mean of the vertical rigid energy of all line segments, and the mean of the line-related energy of all detected line segments is 1; and the weight of the mean of the vertical rigid energy of all line segments is the largest; Among them, the expression of the mean value of the conformal energy of all quadrilateral meshes is: In the formula, E r (p') represents the shape-preserving energy of the quadrilateral mesh network after similarity transformation, p' represents the vertex coordinates of the quadrilateral mesh surface after similarity transformation, N represents the total number of quadrilateral meshes in the quadrilateral mesh network, f represents the index of the quadrilateral mesh surface, A p Represents an 8*4 matrix consisting of the original coordinates of the four vertices of the quadrilateral mesh, P f ' represents an 8*1 vector consisting of the coordinates of the four vertices of the quadrilateral mesh after similarity transformation, and I represents the unit matrix; The expression of the mean value of the vertical rigid energy of all the line segments is: In the formula, l represents a line segment, Represents the total number of line segments, R l represents the known rotation matrix, ψ represents the direction vector of the line segment, and ψ' represents the direction vector of the line segment after similarity transformation; The expression of the mean value of the line correlation energy of all line segments is: Wherein, o represents the starting point of the detected line segment before the similarity transformation, e represents the end point of the detected line segment before the similarity transformation, o' represents the starting point of the detected line segment after the similarity transformation, e' represents the end point of the detected line segment after the similarity transformation, and p' represents the vertex coordinates of the quadrilateral mesh after the similarity transformation. represents the total number of line segments, P o ' represents the coordinates of the starting point in the vertex of the quadrilateral mesh after similarity transformation, P e ' represents the coordinates of the end point of the vertex of the quadrilateral mesh after similarity transformation, A o and A e The coefficient vector of the bilinear interpolation function is formed; wherein the weighted sum of the mean of the conformal energy of all quadrilateral meshes, the mean of the vertical rigid energy of all line segments and the mean of the line-correlated energy of all detected line segments is minimized as the optimization target, and the target similarity transformation matrix of each quadrilateral mesh is determined, including: The parameters of the quadrilateral mesh network are fixed, and the similarity transformation matrix of each quadrilateral mesh is optimized to obtain the optimal rotation matrix of each quadrilateral mesh; wherein the rotation matrix R in each direction i are independent of each other. In the expression of the mean value of the conformal energy of all quadrilateral meshes, the expression of the mean value of the vertical rigid energy of all line segments, and the expression of the mean value of the line-related energy of all line segments, each term in the summation part of each formula is independent, and the θ of each term in the summation part is l Independently, optimize each term separately l , find the optimal common rotation θ for each quadrilateral mesh l , get the optimal rotation matrix for each quadrilateral mesh; The optimal rotation matrix of each quadrilateral mesh is fixed to optimize the parameters of the quadrilateral mesh network; wherein, after the cutting line segments in each quadrilateral mesh are transformed according to the target similarity transformation of each quadrilateral mesh to obtain the deformed and repaired street view image, the method further includes: In the case where the deformed and restored street view image is distorted, obtaining a distortion-to-be-corrected area of ​​the deformed and restored street view image; Determine the coordinates of each pixel point in the distortion-to-be-corrected area on a virtual panoramic spherical surface, and obtain the virtual panoramic spherical surface coordinates of each pixel point in the distortion-to-be-corrected area; the virtual panoramic sphere is a virtual panoramic sphere corresponding to the deformed and restored street view image; Determine the panoramic spherical geographic coordinates of each pixel point in the distortion to be corrected area according to the virtual panoramic spherical coordinates of each pixel point in the distortion to be corrected area; Obtaining the coordinates of the tangent points between the projection plane of the deformed and restored street view image and the virtual panoramic sphere; the projection plane is a plane parallel to the facade of the building in the deformed and restored street view image; According to the panoramic sphere geographic coordinates of each pixel point in the distortion-to-be-corrected area, the virtual panoramic spherical coordinates, the tangent point coordinates and a preset perspective mathematical model, the coordinates of each pixel point in the distortion-to-be-corrected area projected on the projection plane are calculated; the perspective mathematical model is a relationship between the panoramic sphere geographic coordinates and the panoramic sphere equatorial tangent plane coordinates obtained by projecting rays emitted from the center of the panoramic sphere onto the equatorial tangent plane of the panoramic sphere.

2. The method according to claim 1, characterized in that The parameters of the quadrilateral mesh network are fixed, and the similarity transformation matrix of each quadrilateral mesh is optimized to obtain the optimal rotation matrix of each quadrilateral mesh, including: With the constraint that the same line segment divided in multiple quadrilateral meshes and the divided line segments with similar directions in different quadrilateral meshes still maintain similar rotation angles after transformation, the similarity transformation matrix of each quadrilateral mesh is optimized to obtain the optimal rotation matrix of each quadrilateral mesh.

3. The method according to claim 1, characterized in that The tangent point between the projection plane of the deformed and restored street view image and the virtual panoramic sphere includes: a boundary inflection point of a road feature in the deformed and restored street view image.

4. The method according to claim 1, characterized in that: The horizontal field angle of the distortion to be corrected area is not greater than 120°.

5. The method according to claim 1, characterized in that After calculating the coordinates of each pixel point in the distortion-to-be-corrected area projected on the projection plane according to the panoramic spherical geographic coordinates of each pixel point in the distortion-to-be-corrected area, the virtual panoramic spherical coordinates, the tangent point coordinates and the preset perspective mathematical model, the method further includes: According to the projection coordinates of the boundary corner points of the distortion-to-be-corrected area on the projection plane, the projected coordinates of each pixel point in the distortion-to-be-corrected area are translated to obtain a processed street view image.

6. The method according to claim 5, characterized in that After translating the projected coordinates of each pixel point in the distortion-to-be-corrected area according to the projection coordinates of the boundary corner points of the distortion-to-be-corrected area on the projection plane to obtain a processed street view image, the method further includes: The processed street view image is used as a facade texture data source to perform facade texture mapping.

7. A device for correcting facade texture of street view images, characterized in that: include: A placement module, used for placing a preset quadrilateral grid network on the street view image to be corrected; The quadrilateral mesh network includes a plurality of quadrilateral meshes; A detection module, used for detecting line segments in the street view image to be corrected; A cutting module, used for cutting the line segment by using the upper and lower edges of the quadrilateral grid in the preset quadrilateral grid network to obtain a cutting line segment; A determination module is used to determine the target similarity transformation matrix of each quadrilateral mesh with the minimum weighted sum of the mean value of the conformal energy of all quadrilateral meshes, the mean value of the vertical rigid energy of all line segments and the mean value of the line correlation energy of all detected line segments as the optimization target; wherein the conformal energy of any quadrilateral mesh is used to make the quadrilateral mesh rigid before and after the similarity transformation; the vertical rigid energy of any line segment is used to make the slope of the line segment before the similarity transformation close to the slope after the similarity transformation; the line correlation energy of any line segment is used to make the position of the line segment before the similarity transformation relative to the quadrilateral mesh network before the similarity transformation consistent with the position of the line segment after the similarity transformation relative to the quadrilateral mesh network after the similarity transformation; A transformation module, used for transforming the cutting line segments in each of the quadrilateral meshes according to the target similarity transformation of each of the quadrilateral meshes to obtain a deformed and restored street view image; Wherein, before the upper and lower edges of the quadrilateral meshes in the preset quadrilateral mesh network are used to cut the line segments to obtain the cut line segments, the method further includes: receiving a vertical line segment manually located from the street view image to be corrected; The detecting of line segments in the street view image to be corrected is specifically: Using an LSD line segment detection algorithm to detect line segments in the street view image to be corrected; The method of cutting the line segments by using the upper and lower edges of the quadrilateral mesh in the preset quadrilateral mesh network to obtain the cut line segments is specifically as follows: Using the upper and lower edges of the quadrilateral meshes in the preset quadrilateral mesh network, all line segments are cut to obtain cut line segments; The sum of the weights corresponding to the mean of the conformal energy of all quadrilateral meshes, the mean of the vertical rigid energy of all line segments, and the mean of the line-related energy of all detected line segments is 1; and the weight of the mean of the vertical rigid energy of all line segments is the largest; Among them, the expression of the mean value of the conformal energy of all quadrilateral meshes is: In the formula, E r (p') represents the shape-preserving energy of the quadrilateral mesh network after similarity transformation, p' represents the vertex coordinates of the quadrilateral mesh surface after similarity transformation, N represents the total number of quadrilateral meshes in the quadrilateral mesh network, f represents the index of the quadrilateral mesh surface, A p Represents an 8*4 matrix consisting of the original coordinates of the four vertices of the quadrilateral mesh, P′ f Represents an 8*1 vector consisting of the coordinates of the four vertices of the quadrilateral mesh after similarity transformation, and I represents the unit matrix; The expression of the mean value of the vertical rigid energy of all the line segments is: In the formula, l represents a line segment, Represents the total number of line segments, R l represents the known rotation matrix, ψ represents the direction vector of the line segment, and ψ' represents the direction vector of the line segment after similarity transformation; The expression of the mean value of the line correlation energy of all line segments is: Wherein, o represents the starting point of the detected line segment before the similarity transformation, e represents the end point of the detected line segment before the similarity transformation, o' represents the starting point of the detected line segment after the similarity transformation, e' represents the end point of the detected line segment after the similarity transformation, and p' represents the vertex coordinates of the quadrilateral mesh after the similarity transformation. represents the total number of line segments, P′o represents the coordinates of the starting point in the vertex of the quadrilateral mesh after similarity transformation, and P e ' represents the coordinates of the end point of the vertex of the quadrilateral mesh after similarity transformation, A o and A e The coefficient vector that constitutes the bilinear interpolation function; The target similarity transformation matrix of each quadrilateral mesh is determined by minimizing the weighted sum of the mean value of the conformal energy of all quadrilateral meshes, the mean value of the vertical rigid energy of all line segments, and the mean value of the line-correlated energy of all detected line segments, including: The parameters of the quadrilateral mesh network are fixed, and the similarity transformation matrix of each quadrilateral mesh is optimized to obtain the optimal rotation matrix of each quadrilateral mesh; wherein the rotation matrix R in each direction l are independent of each other. In the expression of the mean value of the conformal energy of all quadrilateral meshes, the expression of the mean value of the vertical rigid energy of all line segments, and the expression of the mean value of the line-related energy of all line segments, each term in the summation part of each formula is independent, and the θ of each term in the summation part is l Independently, optimize each term separately l , find the optimal common rotation θ for each quadrilateral mesh l , get the optimal rotation matrix for each quadrilateral mesh; The optimal rotation matrix of each quadrilateral mesh is fixed to optimize the parameters of the quadrilateral mesh network; wherein, after the cutting line segments in each quadrilateral mesh are transformed according to the target similarity transformation of each quadrilateral mesh to obtain the deformed and repaired street view image, the method further includes: In the case where the deformed and restored street view image is distorted, obtaining a distortion-to-be-corrected area of ​​the deformed and restored street view image; Determine the coordinates of each pixel point in the distortion-to-be-corrected area on a virtual panoramic spherical surface, and obtain the virtual panoramic spherical surface coordinates of each pixel point in the distortion-to-be-corrected area; the virtual panoramic sphere is a virtual panoramic sphere corresponding to the deformed and restored street view image; Determine the panoramic spherical geographic coordinates of each pixel point in the distortion to be corrected area according to the virtual panoramic spherical coordinates of each pixel point in the distortion to be corrected area; Obtaining the coordinates of the tangent points between the projection plane of the deformed and restored street view image and the virtual panoramic sphere; the projection plane is a plane parallel to the facade of the building in the deformed and restored street view image; According to the panoramic sphere geographic coordinates of each pixel point in the distortion-to-be-corrected area, the virtual panoramic spherical coordinates, the tangent point coordinates and a preset perspective mathematical model, the coordinates of each pixel point in the distortion-to-be-corrected area projected on the projection plane are calculated; the perspective mathematical model is a relationship between the panoramic sphere geographic coordinates and the panoramic sphere equatorial tangent plane coordinates obtained by projecting rays emitted from the center of the panoramic sphere onto the equatorial tangent plane of the panoramic sphere.

Images