Facade texture repairing method and device, electronic equipment and computer storage medium

Abstract

The invention relates to a facade texture repairing method and device, electronic equipment and a computer storage medium, and belongs to the technical field of geographic information surveying and mapping, and the facade texture repairing method comprises the steps: obtaining a multi-view texture image of a building facade, and determining a main texture image and an auxiliary texture image in the multi-view texture image; the view angle of the main texture image is taken as a main view angle, the auxiliary texture image is converted to the main view angle by adopting a homography matrix, a candidate texture image is obtained, and covered areas of the main texture image and the candidate texture image are determined based on the structural similarity of the main texture image and the candidate texture image; and calculating a cost value during texture synthesis of the covered area based on the main texture image and the candidate texture image by adopting a preset texture synthesis cost function, determining a texture synthesis mode with the minimum cost value, and obtaining a texture restoration image of the building facade. According to the invention, accurate texture repair of the building facade can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of geographic information surveying and mapping, and in particular to a facade texture repairing method and device, electronic equipment and computer storage medium. BACKGROUND

[0002] In the fields of digital city construction, cultural heritage protection, building information model (BIM) management, etc., high-precision three-dimensional building models are important basic data. Realistic textures, as a key component of three-dimensional models, directly affect the visual effect and application value of the models. The texture data of existing three-dimensional building models is mainly obtained through multi-source remote sensing technologies such as unmanned aerial vehicle oblique photography and ground laser scanning. However, due to factors such as shooting angle obstruction, uneven lighting conditions, and sensor noise, the model surface often has problems such as texture loss, blur, misplacement, or color distortion, which seriously restricts the practicality of three-dimensional models.

[0003] Traditional texture repairing techniques mainly rely on manual intervention and manually fill in the missing areas through image editing software. Although this method can ensure a certain accuracy, it is low in efficiency and requires high professional skills of the operator, and it is difficult to meet the automated processing needs of large-scale city-level three-dimensional models. In existing automatic image repairing techniques, whether traditional methods based on neighborhood matching or deep learning models relying on data-driven, the fundamental limitation is the neglect of the uniqueness of the original content of the occluded area. Traditional methods fill in the missing area by searching for local samples, which is essentially a statistical texture speculation based on two-dimensional images. Deep learning models can learn high-order semantic associations, but the generated textures are still the mean mapping of the training data distribution. Both methods take a single image as the basis for repairing and treat the occluded area as an unknown random variable, which leads to the possibility that the repaired result deviates from the true facade features of the building.

[0004] Therefore, the existing texture image repairing techniques cannot accurately restore the true facade features of the building. SUMMARY

[0005] Therefore, it is necessary to provide a facade texture repairing method, device, electronic equipment and computer storage medium to solve the problem that the existing texture image repairing techniques cannot accurately restore the true facade features of the building.

[0006] To solve the above problems, in a first aspect, the present application provides a facade texture repairing method, comprising: obtaining multi-view texture images of a building facade, and determining a main texture image and a secondary texture image in the multi-view texture images; The main perspective of the main texture image is taken as a main perspective, a homography matrix is used to convert the secondary texture image to the main perspective to obtain a candidate texture image, and a covered area of the main texture image and the candidate texture image is determined based on structural similarity of the main texture image and the candidate texture image. A preset texture synthesis cost function is used to calculate a cost value when the covered area of the main texture image and the candidate texture image is synthesized, a texture synthesis mode with the minimum cost value is determined, and a texture repaired image of the building facade is obtained.

[0007] In a possible implementation, the multi-perspective texture images of the building facade are obtained, and the main texture image and the secondary texture image in the multi-perspective texture images are determined, including: The multi-perspective images of the building facade are collected by using an image collection device, and the multi-perspective images are projected based on a camera projection matrix to obtain the multi-perspective texture images. The line connecting the center point of the image collection device and the corresponding perspective texture image when each perspective texture image is collected is determined, and the distance between the image collection device and the building facade is determined. The included angle between the line and the normal vector of the building facade is calculated, the texture image with the minimum included angle and within a preset distance range is taken as the main texture image, and the texture images of the remaining perspectives are taken as the secondary texture images.

[0008] In a possible implementation, the camera projection matrix is:

[0009] wherein, K is a 3×3 intrinsic matrix composed of intrinsic elements of the texture image, and are principal distances of the intrinsic elements, R is a 3×3 rotation matrix composed of three angle elements of extrinsic elements of the texture image, t is a 3×1 offset vector composed of three line elements of the extrinsic elements of the texture image, (u, v) represents coordinates of a three-dimensional point in the building facade in a pixel coordinate system, represents coordinates of the three-dimensional point in the building facade in a camera coordinate system, , and represent coordinates of the three-dimensional point in the building facade in a world coordinate system, represents coordinates of the three-dimensional point in the building facade in the camera coordinate system.

[0010] In a possible implementation, the homography matrix is:

[0011] wherein, is a first row of the homography matrix, is a second row of the homography matrix, and is a third row of the homography matrix.i The homography matrix that transforms the secondary texture image to the main view, i Indicates the sequence number of the secondary texture image, Indicates the i The internal parameter matrix corresponding to the secondary texture image is Represents the intrinsic parameter matrix of the main texture image, Indicates the i The rotation matrix of the secondary texture image from its own camera coordinate system to the camera coordinate system of the main texture image, Indicates the i The translation vector of the secondary texture image from its own camera coordinate system to the camera coordinate system of the primary texture image, and Indicates the building facade i The plane equation of the camera coordinate system of the sub-texture image, Indicates the number of sub-texture images.

[0012] In a possible implementation, determining the covered areas of the main texture image and the candidate texture image based on the structural similarity between the main texture image and the candidate texture image includes: The structural similarity calculation formula is used to calculate the structural similarity of each pixel in the main texture image and the candidate texture image. The structural similarity calculation formula is:

[0013]

[0014]

[0015]

[0016] in, represents the brightness function, represents the contrast function, represents the structure function, x and y represent the pixel values ​​of the corresponding pixels in the first texture image and the second texture, Represents the average grayscale value of each pixel in the first texture image, Represents the average grayscale value of each pixel in the second texture image, is the covariance of the grayscale values ​​of each pixel in the first texture image and the second texture image, is the standard deviation of the grayscale values ​​of each pixel in the first texture image, is the standard deviation of the grayscale value of each pixel in the second texture image, 、 、 、 、 and is a preset constant; determining the pixel point with the structure similarity less than the preset threshold as an occluded pixel point, and determining all the occluded pixel points as an occluded region.

[0017] In a possible implementation, the texture synthesis cost function is:

[0018]

[0019]

[0020] wherein, E(x) is a total energy value, is a data item energy value, is a smooth item energy item, x represents an energy value of a pixel point to be synthesized, i , j indicates a number of a texture image corresponding to the pixel point to be synthesized; indicates a view priority of the texture image in which the pixel point to be synthesized is located; indicates a resolution priority of the texture image in which the pixel point to be synthesized is located; indicates a color priority of the texture image in which the pixel point to be synthesized is located, , p is a preset constant.

[0021] In a possible implementation, the texture synthesis mode with the minimum cost value is determined, and a texture repair image of a building facade is obtained, including: determining a synthesized texture of the occluded region based on the texture synthesis mode with the minimum cost value; taking the synthesized texture of the occluded region and a mask image of the occluded region as inputs of a preset image repair deep learning model, and obtaining a texture repair image of the building facade.

[0022] In a second aspect, the present application further provides a facade texture repair device, including: an image acquisition module, configured to acquire multi-view texture images of a building facade, and determine a main texture image and a secondary texture image in the multi-view texture images; an occluded region determination module, configured to take a view angle of the main texture image as a main view angle, convert the secondary texture image to the main view angle by using a homography matrix to obtain a candidate texture image, and determine an occluded region of the main texture image and the candidate texture image based on structure similarities of the main texture image and the candidate texture image; The texture repairing module is configured to calculate a value of a preset texture synthesis cost function when texture synthesis is performed based on the covered area of the main texture image and the candidate texture image, determine a texture synthesis mode with the minimum value of the cost function, and obtain a texture repairing image of the building facade.

[0023] In a third aspect, the present application provides an electronic device, comprising a memory and a processor, wherein, The memory is configured to store a program. The processor, coupled to the memory, is configured to execute the program stored in the memory to implement the steps of the facade texture repairing method according to any one of the above embodiments.

[0024] In a fourth aspect, the present application provides a computer readable storage medium for storing computer readable programs or instructions, which can implement the steps of the facade texture repairing method according to any one of the above embodiments when executed by a processor.

[0025] The facade texture repairing method provided by the present application can obtain multi-view texture images of a building facade, divide the multi-view texture images into a main texture image and a secondary texture image, determine the optimal texture image viewing angle, and ensure the accuracy of texture repairing. The viewing angle of the main texture image is used as the main viewing angle, the secondary texture image is converted to the main viewing angle by using a homography matrix to obtain a candidate texture image, the covered area of the main texture image and the candidate texture image is determined based on the structural similarity of the main texture image and the candidate texture image, the covered area of the main texture image and the candidate texture image is extracted by using the structural similarity of the multi-view texture images, the covered area of the main texture image and the candidate texture image can be accurately determined, a preset texture synthesis cost function is used to calculate the value of the cost function when texture synthesis is performed based on the covered area of the main texture image and the candidate texture image, the texture synthesis mode with the minimum value of the cost function is determined, the texture repairing image of the building facade is obtained, the texture repairing problem is converted into a problem of finding the minimum cost value, the calculation of texture repairing is facilitated, the accuracy of building facade texture repairing is further improved, the texture of the building facade is repaired by using multi-view texture images, the structural similarity of the multi-view texture images is used to effectively restore the real texture of the facade when the occlusion range of the facade texture is high. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0027] Figure 1 A schematic flow chart of a facade texture repair method provided by an embodiment of the present invention; Figure 2 A flowchart of a method for implementing S101 provided in an embodiment of the present invention; Figure 3 A texture image provided by an embodiment of the present invention; Figure 4 A flowchart of a method for implementing S103 provided in an embodiment of the present invention; Figure 5 A real texture synthesis result image provided by an embodiment of the present invention; Figure 6 A comparison diagram of the results of comparing a real occlusion area and an actual occlusion extraction area provided by an embodiment of the present invention; Figure 7 A schematic structural diagram of a facade texture repair device provided by an embodiment of the present invention; Figure 8 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0029] The terms "first," "second," and so on, used in the embodiments of the present invention are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, technical features designated as "first" or "second" may explicitly or implicitly include at least one such feature.

[0030] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0031] A specific embodiment of the present invention, as Figure 1 As shown, a facade texture repair method is disclosed, comprising: S101: Acquire a multi-view texture image of a building facade, and determine a primary texture image and a secondary texture image in the multi-view texture image.

[0032] In the embodiment of the present application, the building facade refers to the three-dimensional outer facade of a building, and the texture image can accurately display the appearance of the three-dimensional building facade in the three-dimensional model of the building. The multi-view texture image of the building facade refers to the texture images collected from different angles for the same facade of the building. The multi-view texture image can display the appearance of the same facade of the building from different angles. For the case where the facade is blocked, the blocked areas of the texture images of different angles may be different. For example, in a texture image of a certain angle, the lower left corner of the facade of the building is blocked by trees, but in a texture image of another angle, the lower left corner of the facade of the building is not blocked. Based on this, the texture of the facade can be repaired in combination with the texture images of multiple angles.

[0033] In the embodiment of the present application, for the obtained multi-view texture images, the main texture image and the auxiliary texture image need to be determined. The main texture image is the texture image with the best angle, which can most accurately and realistically represent the texture of the facade of the building. The specific determination method of the main texture image and the auxiliary texture image will be described in detail later in the present application.

[0034] S102, taking the angle of the main texture image as the main angle, converting the auxiliary texture image to the main angle by using the homography matrix to obtain the candidate texture image, and determining the blocked area of the main texture image and the candidate texture image based on the structural similarity of the main texture image and the candidate texture image.

[0035] In the embodiment of the present application, after the main texture image and the auxiliary texture image are determined, all auxiliary texture images are converted to the main angle to obtain the candidate texture image. Specifically, the camera coordinate system of the image acquisition device when the main texture image is collected can be taken as the target, and the auxiliary texture image is converted from the camera coordinate system of the auxiliary texture image to the camera coordinate system of the main texture image. The specific conversion method will be described in detail later in the present application.

[0036] In the embodiment of the present application, after all candidate texture images are obtained, the structural similarity of each pixel point in each candidate texture image and the corresponding pixel point in the main texture image is calculated, and the blocked area of the main texture image and the candidate texture image is determined based on the structural similarity. The specific calculation method of the structural similarity and the determination method of the blocked area will be described in detail later in the present application.

[0037] S103, using a preset texture synthesis cost function to calculate the cost value when the texture synthesis is performed based on the blocked area of the main texture image and the candidate texture image, determining the texture synthesis mode with the minimum cost value, and obtaining the texture repair image of the building facade.

[0038] In the embodiment of the present application, after the covered areas of the main texture image and the candidate texture image are determined, a texture synthesis cost function is constructed, and a texture synthesis cost value of the covered areas of the main texture image and the candidate texture image is calculated based on the texture synthesis cost function, wherein the texture synthesis cost value is used to represent the cost of synthesizing the covered areas by using different texture images, and for a certain covered area, the synthesis can be performed by synthesizing the corresponding areas of the main texture image and the secondary texture image, or the synthesis can be performed by synthesizing the corresponding areas of two secondary texture images, so the cost values of various synthesis modes need to be calculated to determine the optimal texture synthesis mode, and then the texture repair image of the building facade is obtained.

[0039] The facade texture repair method provided by the present application can obtain the multi-view texture images of the building facade, and divide the multi-view texture images into the main texture image and the secondary texture image, so as to determine the optimal texture image view angle and ensure the accuracy of texture repair; the view angle of the main texture image is used as the main view angle, the secondary texture image is converted to the main view angle by using the homography matrix to obtain the candidate texture image, and the covered areas of the main texture image and the candidate texture image are determined based on the structural similarity of the main texture image and the candidate texture image; the covered areas of the main texture image and the candidate texture image are extracted by using the structural similarity of the multi-view texture images, so the covered areas of the main texture image and the candidate texture image can be accurately determined; the preset texture synthesis cost function is used to calculate the cost value of the texture synthesis based on the covered areas of the main texture image and the candidate texture image, the texture synthesis mode with the minimum cost value is determined, and the texture repair image of the building facade is obtained; the texture repair problem is converted into the problem of finding the minimum cost value, so the calculation of texture repair is facilitated, the accuracy of the texture repair of the building facade is further improved, the texture of the building facade is repaired by combining the multi-view texture images, the structural similarity of the multi-view texture images is used, and the real texture of the facade can be effectively restored when the occlusion range of the facade texture is high.

[0040] In some possible embodiments of the present application, as shown in Figure 2 the multi-view texture images of the building facade are obtained, and the main texture image and the secondary texture image in the multi-view texture images are determined, including: S201, the multi-view images of the building facade are collected by using an image collection device, and the multi-view images are projected to obtain the multi-view texture images based on a camera projection matrix; S202, the line connecting the center point of the image collection device and the corresponding view angle texture image when each view angle texture image is collected, and the distance between the image collection device and the building facade are determined; S203, the included angle between the line and the normal vector of the building facade is calculated, the texture image with the smallest included angle and within the preset distance range is taken as the main texture image, and the texture images of the remaining view angles are taken as the secondary texture images.

[0041] In the embodiment of the present application, when the multi-view texture image of the building facade is acquired, the multi-view image of the building facade is acquired by the image acquisition device, wherein the image acquisition device can be a drone carrying an image acquisition device, which facilitates the acquisition of multi-view images of the higher building facade. After the multi-view image of the building facade is acquired, the multi-view image is projected based on the camera projection matrix to obtain the multi-view texture image, wherein the camera projection matrix is:

[0042] wherein, K is a 3x3 intrinsic matrix composed of intrinsic elements of the texture image, and are principal distances of the intrinsic elements, R is a 3x3 rotation matrix composed of three corner elements of extrinsic elements in the texture image, t is a 3x1 offset vector composed of three line elements of extrinsic elements in the texture image, (u, v) represents the coordinates of the three-dimensional point in the pixel coordinate system in the building facade, represents the coordinates of the three-dimensional point in the camera coordinate system in the building facade, , and represent the coordinates of the three-dimensional point in the world coordinate system in the building facade, represents the coordinates of the three-dimensional point in the camera coordinate system in the building facade.

[0043] In the embodiment of the present application, after the multi-view texture image of the building facade is obtained, the center point of the image acquisition device and the corresponding view texture image are determined, and the distance between the image acquisition device and the building facade is determined, and then the included angle between the line and the normal vector of the building facade is calculated. The texture image with the smallest included angle within the preset distance range is used as the main texture image, and the remaining view texture images are used as the auxiliary texture images. Based on this, the texture image with the best view can be determined as the main texture image, which ensures the accuracy of the building facade texture image obtained after texture repair.

[0044] In some possible embodiments of the present application, the homography matrix for converting the auxiliary texture image to the main view is:

[0045] wherein, is the homography matrix for converting the i-th auxiliary texture image to the main view, i represents the serial number of the auxiliary texture image, i represents the i-th auxiliary texture image, i ​an intrinsic matrix corresponding to the main texture image, an intrinsic matrix corresponding to the main texture image, a rotation matrix, i a rotation matrix, a translation vector, i a translation vector, and a plane equation of the building facade in the camera coordinate system of the i-th sub texture image, i a plane equation of the building facade in the camera coordinate system of the i-th sub texture image, a number of sub texture images.

[0046] In the embodiment of the application, because the shooting angles of the texture images are different, the sizes of the facade textures are not uniform, and the texture images cannot be directly used for texture synthesis. However, the essence of the multi-view texture images is the projection of the same three-dimensional facade, and therefore, the candidate texture images can be converted to the shooting angle of the main texture image by using the homography matrix, the main texture image and the candidate texture image are aligned, and the candidate texture with the best shooting angle is obtained, thereby laying a foundation for subsequent occlusion extraction of the best shooting angle.

[0047] In some possible embodiments of the application, the covered area of the main texture image and the candidate texture image is determined based on the structural similarity of the main texture image and the candidate texture image, and the method comprises the following steps. The structural similarity of each pixel point in the main texture image and the candidate texture image is calculated by using a structural similarity calculation formula, wherein the structural similarity calculation formula is as follows:

[0048]

[0049]

[0050]

[0051] wherein, represents a luminance function, represents a contrast function, represents a structural function, x and y represent the pixel values of the corresponding pixel points in the first texture image and the second texture image, represents the average gray value of each pixel point in the first texture image, represents the average gray value of each pixel point in the second texture image, is the covariance of the gray values of each pixel point in the first texture image and the second texture image, is the standard deviation of the gray values of each pixel point in the first texture image, a standard deviation of gray scale values of each pixel in the second texture image, 、 、 、 、 and is a preset constant; determining the pixel points with a structure similarity less than a preset threshold as occluded pixel points, and determining all the occluded pixel points as an occluded region.

[0052] In the embodiment of the present application, as shown in Figure 3 under the premise that all facade textures maintain the same view angle and size, the occluded regions of the main texture and the candidate textures are extracted by using the structure similarity, the structure similarity is usually used to evaluate the similarity between two images, when the structure similarity value is low, it indicates that there is occlusion between any one of the two images, by calculating the structure similarity between the main texture image and the multiple candidate texture images, the occluded region affected by the main texture can be extracted; similarly, the candidate texture can also extract its own occluded range and unoccluded region range.

[0053] In some possible embodiments of the present application, the texture synthesis cost function is:

[0054]

[0055]

[0056] wherein, E(x) is a total energy value, is a data item energy value, is a smooth item energy item, x represents an energy value of a pixel to be synthesized, i , j indicates the number of the texture image corresponding to the pixel to be synthesized; indicates the view angle priority of the texture image where the pixel to be synthesized is located; indicates the resolution priority of the texture image where the pixel to be synthesized is located; indicates the color priority of the texture image where the pixel to be synthesized is located, 、 p is a preset constant.

[0057] In the embodiment of the present application, the synthesis cost function is used to calculate the total energy value of texture synthesis, the total energy value includes the data item energy value and the smooth item energy value, the data item is mainly determined by the texture image quality, in order to synthesize as much as possible unoccluded and high-quality facade texture, the energy item is constructed by fully considering the factors such as view angle, resolution, color, represents the perspective priority of the texture image where the pixel to be synthesized is located, the closer to the perspective of the main texture, the smaller the energy value is; represents the resolution priority of the texture image where the pixel to be synthesized is located, the higher the source texture resolution is, the smaller the energy value is; represents the color priority of the texture image where the pixel to be synthesized is located, the smaller the color difference with the main texture, the smaller the energy value is. The smoothing term is used to evaluate the spatial consistency of the synthesized texture, avoiding the synthesized texture being too fragmented and the obvious texture misplacement. In the solving process, the adjacent pixels from the same image are preferentially selected. The present application sets a penalty term in the smoothing term, if the adjacent pixel texture comes from the same texture image, the penalty value is 0, otherwise, the penalty value is p.

[0058] In some possible embodiments of the present application, as shown in Figure 4 the texture synthesis mode with the minimum generation value is determined to obtain the texture repair image of the building facade, comprising: S401, determining the synthesized texture of the covered area based on the texture synthesis mode with the minimum generation value; S402, taking the synthesized texture of the covered area and the mask image of the covered area as inputs of a preset image repair deep learning model to obtain the texture repair image of the building facade.

[0059] In the embodiments of the present application, the synthesis of the texture restores the occluded area of the best perspective main texture image to the greatest extent, but is affected by factors such as the difference in shooting angle, light factor, systematic error, etc. of different texture images, and the synthesized texture has phenomena such as occlusion, inconsistent color, texture misplacement, etc. that cannot be eliminated to some extent, and further texture repair is needed. The image repair deep learning model can be an image repair model based on a generative adversarial network (such as a Deepfill model), which replaces the traditional convolution with a gated convolution, dynamically learns the features of the missing area, extracts features from the known area of the image, generates the content of the area to be repaired, and ensures that the repaired area and the surrounding environment are consistent in semantics and texture. The model mainly consists of two stages, generating a rough structure of the area to be repaired in the coarse repair stage, and optimizing the details in the fine repair stage to improve the texture quality of the repaired area. The mask image and the synthesized texture of the occluded area are taken as input data of the model, and the complete facade texture after repair is output.

[0060] In a specific embodiment of the present application, a real three-dimensional monomer model of a city is taken as a data source to construct an experimental data set containing different types of occlusions such as eaves, billboards, auxiliary building structures and buildings, etc. The XML file obtained by aerial triangulation is used to parse the interior and exterior orientation elements of the UAV oblique image, and the four vertex coordinates of the building facade are projected onto the UAV oblique image to obtain a visible facade texture image data set. Therefore, each facade data set contains facade three-dimensional data, a visible facade texture image data set and a corresponding XML file of interior and exterior orientation elements. In terms of parameter setting, in the texture alignment process, one texture image with the best view angle and five candidate texture images with the closest view angle are selected. In the process of using structural similarity to extract occlusions, , , , , the window size is 7; in the process of texture synthesis, , , in the process of texture repair, a pre-trained model is used, and texture image training samples are added for optimization. The occlusion range in the original best view angle texture is extracted through structural similarity, and the real texture is synthesized through the visible area in the multi-view texture image, and the real texture is repaired through deep learning. The results are shown in Figure 5 . The real texture synthesis result basically eliminates the facade texture occlusion. From the texture occlusion extraction result, the area not only contains the real occlusion area, but also contains the area that is not actually occluded, Figure 6 . The comparison between the real occlusion area and the actual occlusion extraction area is shown, and the actual texture occlusion range reaches 44.73%, which seriously affects the visibility of the three-dimensional model. The redundant occlusion range is less than 20%, and the pseudo-occlusion area in this part is mainly optimized in the texture synthesis process to eliminate the influence. There is still some occlusion in facade textures a and c, but the effect is also significantly improved after texture repair.

[0061] The texture synthesis of the embodiment of the present application restores the actual situation of the facade texture to the greatest extent, which provides more abundant information for texture repair. Due to the differences in shooting time, errors, etc. of multi-view images, there are phenomena such as inconsistent color tone and texture misplacement in the texture synthesis result, but after the texture repair, the color transition is more natural in visual effect, and the occluded texture is also eliminated to a certain extent.

[0062] In order to better implement the facade texture repair method in the embodiment of the present application, on the basis of the facade texture repair method, as shown in Figure 7 , the embodiment of the present application also provides a facade texture repair device. The facade texture repair device 700 comprises: An image acquisition module 701 is used to acquire a multi-view texture image of a building facade and determine a primary texture image and a secondary texture image in the multi-view texture image; The covered area determination module 702 is configured to use the primary texture image as the primary perspective, convert the secondary texture image to the primary perspective using a homography matrix, obtain a candidate texture image, and determine the covered areas of the primary texture image and the candidate texture image based on the structural similarity between the primary texture image and the candidate texture image; The texture restoration module 703 is used to calculate the cost of texture synthesis based on the covered area of ​​the main texture image and the candidate texture image using a preset texture synthesis cost function, determine the texture synthesis method with the minimum cost, and obtain the texture restoration image of the building facade.

[0063] The facade texture repair device 700 provided in the above embodiment can implement the technical solution described in the above facade texture repair method embodiment. The specific implementation principles of the above modules or units can refer to the corresponding contents in the above facade texture repair method embodiment, which will not be repeated here.

[0064] like Figure 8 As shown, the present invention also provides an electronic device 800. The electronic device 800 includes a processor 801, a memory 802 and a display 803. Figure 8 Only some of the components of the electronic device 800 are shown, but it should be understood that implementation of all of the shown components is not required, and more or fewer components may be implemented instead.

[0065] In some embodiments, the processor 801 may be a central processing unit (CPU), a microprocessor, or other data processing chip, configured to execute program codes or process data stored in the memory 802 , such as the facade texture restoration method of the present invention.

[0066] In some embodiments, processor 801 may be a single server or a server group. The server group may be centralized or distributed. In some embodiments, processor 801 may be local or remote. In some embodiments, processor 801 may be implemented on a cloud platform. In some embodiments, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, multiple clouds, or any combination thereof.

[0067] The memory 802 can be an internal storage unit of the electronic device 800, such as a hard disk or a memory of the electronic device 800 in some embodiments. The memory 802 can also be an external storage device of the electronic device 800, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the electronic device 800 in other embodiments.

[0068] Further, the memory 802 can include both an internal storage unit and an external storage device of the electronic device 800. The memory 802 is used to store application software installed on the electronic device 800 and various types of data.

[0069] The display 803 can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, etc. in some embodiments. The display 803 is used to display information of the electronic device 800 and to display a visualized user interface. The components 801-803 of the electronic device 800 communicate with each other through a system bus.

[0070] In some embodiments, the following steps can be implemented when the processor 801 executes the facade texture repair program in the memory 802: Obtaining multi-view texture images of a building facade, and determining a main texture image and a secondary texture image in the multi-view texture images; Taking a view angle of the main texture image as a main view angle, converting the secondary texture image to the main view angle by using a homography matrix to obtain a candidate texture image, and determining a covered area of the main texture image and the candidate texture image based on a structural similarity of the main texture image and the candidate texture image; Calculating a cost value when performing texture synthesis based on the covered area of the main texture image and the candidate texture image by using a preset texture synthesis cost function, determining a texture synthesis mode with a minimum cost value, and obtaining a texture repair image of the building facade.

[0071] It should be understood that, in addition to the above functions, the processor 801 can also implement other functions when executing the facade texture repair program in the memory 802. For details, refer to the description of the corresponding method embodiments.

[0072] Further, the embodiments of the present application do not make specific limitation on the type of the electronic device 800 mentioned above, and the electronic device 800 can be a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop computer, or the like. Exemplary embodiments of the portable electronic device include, but are not limited to, a portable electronic device running an IOS, an android, a microsoft, or other operating system. The portable electronic device can also be other portable electronic devices, such as a laptop computer having a touch-sensitive surface (e.g., a touch panel), and the like. It should also be understood that in some other embodiments of the present application, the electronic device 800 can also not be a portable electronic device, but a desktop computer having a touch-sensitive surface (e.g., a touch panel).

[0073] Correspondingly, the embodiments of the present application also provide a computer readable storage medium for storing computer readable programs or instructions, which, when executed by a processor, can implement the steps or functions in the facade texture repairing method provided by the above-mentioned method embodiments.

[0074] Those skilled in the art can understand that all or part of the processes of the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. The computer readable storage medium includes a magnetic disk, an optical disk, a read-only memory, a random access memory, and the like.

[0075] The above description is only the preferred embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A facade texture repair method, characterized in that: include: Acquire a multi-view texture image of a building facade, and determine a primary texture image and a secondary texture image in the multi-view texture image; Taking the perspective of the primary texture image as the primary perspective, converting the secondary texture image to the primary perspective using a homography matrix to obtain a candidate texture image, and determining the covered areas of the primary texture image and the candidate texture image based on the structural similarity between the primary texture image and the candidate texture image; A preset texture synthesis cost function is used to calculate a cost value when performing texture synthesis based on the covered area of ​​the main texture image and the candidate texture image, and a texture synthesis method with the minimum cost value is determined to obtain a texture restoration image of the building facade.

2. The facade texture repair method according to claim 1, characterized in that: The acquiring of a multi-view texture image of a building facade and determining a primary texture image and a secondary texture image in the multi-view texture image comprises: An image acquisition device is used to acquire multi-perspective images of a building facade, and the multi-perspective images are projected based on a camera projection matrix to obtain a multi-perspective texture image; Determining the line connecting the center point of the image acquisition device and the texture image of the corresponding perspective when acquiring the texture image of each perspective, and the distance between the image acquisition device and the building facade; The angle between the connecting line and the building facade normal vector is calculated, and the texture image with the smallest angle within a preset distance range is used as the main texture image, and the texture images of the remaining perspectives are used as secondary texture images.

3. The facade texture repair method according to claim 2, characterized in that: The camera projection matrix is: in, K Is a 3×3 internal parameter matrix, which is composed of the internal orientation elements of the texture image. and is the principal distance of the interior orientation element, R Is a 3×3 rotation matrix consisting of the three corner elements of the exterior orientation elements in the texture image. t is a 3×1 offset vector consisting of three line elements of the exterior orientation elements in the texture image, ( u,v ) represents the coordinates of the three-dimensional point in the building facade in the pixel coordinate system, Represents the coordinates of the three-dimensional point in the building facade in the camera coordinate system, 、 and Represents the coordinates of a three-dimensional point in the building facade in the world coordinate system, Represents the coordinates of a 3D point in the building facade in the camera coordinate system.

4. The facade texture repair method according to claim 1, characterized in that: The homography matrix is: in, For the i The homography matrix that transforms the secondary texture image to the main view, i Indicates the sequence number of the secondary texture image, Indicates the i The internal parameter matrix corresponding to the secondary texture image is Represents the intrinsic parameter matrix of the main texture image, Indicates the i The rotation matrix of the secondary texture image from its own camera coordinate system to the camera coordinate system of the main texture image, Indicates the i The translation vector of the secondary texture image from its own camera coordinate system to the camera coordinate system of the primary texture image, and Indicates the building facade i The plane equation of the camera coordinate system of the sub-texture image, Indicates the number of sub-texture images.

5. The facade texture repair method according to claim 1, characterized in that: The determining of the covered areas of the main texture image and the candidate texture image based on the structural similarity between the main texture image and the candidate texture image comprises: The structural similarity calculation formula is used to calculate the structural similarity of each pixel in the main texture image and the candidate texture image, wherein the structural similarity calculation formula is: in, represents the brightness function, represents the contrast function, represents the structure function, x and y represent the pixel values ​​of the corresponding pixels in the first texture image and the second texture, Represents the average grayscale value of each pixel in the first texture image, Represents the average grayscale value of each pixel in the second texture image, is the covariance of the grayscale values ​​of each pixel in the first texture image and the second texture image, is the standard deviation of the grayscale values ​​of each pixel in the first texture image, is the standard deviation of the grayscale value of each pixel in the second texture image, 、 、 、 、 and is a preset constant; Pixels whose structural similarity is less than a preset threshold are determined as blocked pixels, and all blocked pixels are determined as blocked areas.

6. The facade texture repair method according to claim 5, characterized in that: The texture synthesis cost function is: in, E(x) is the total energy value, is the energy value of the data item, is the smoothing term and the energy term, x Represents the energy value of the pixel to be synthesized, i , j Indicates the number of the texture image corresponding to the pixel to be synthesized; Indicates the viewing angle priority of the texture image where the pixel to be synthesized is located; Indicates the resolution priority of the texture image where the pixel to be synthesized is located; Indicates the color priority of the texture image where the pixel to be synthesized is located. 、 p is a preset constant.

7. The facade texture repair method according to claim 1, characterized in that: Determining the texture synthesis method with the minimum cost to obtain the texture restoration image of the building facade includes: Determining a synthesized texture of the covered area based on the texture synthesis method with the minimum cost; The synthesized texture of the covered area and the mask image of the covered area are used as inputs of a preset image restoration deep learning model to obtain a texture restoration image of the building facade.

8. A facade texture repair device, characterized in that: include: An image acquisition module is used to acquire a multi-view texture image of a building facade and determine a primary texture image and a secondary texture image in the multi-view texture image; a covered area determination module, configured to use a homography matrix to transform the secondary texture image to the primary perspective using the primary texture image as the primary perspective, thereby obtaining a candidate texture image, and determine covered areas of the primary texture image and the candidate texture image based on structural similarity between the primary texture image and the candidate texture image; The texture restoration module is used to calculate the cost value when performing texture synthesis based on the covered area of ​​the main texture image and the candidate texture image using a preset texture synthesis cost function, determine the texture synthesis method with the minimum cost value, and obtain the texture restoration image of the building facade.

9. An electronic device, characterized in that: comprising a memory and a processor, wherein, The memory is used to store programs; The processor is coupled to the memory and is configured to execute the program stored in the memory to implement the steps of the facade texture repair method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps of the facade texture repair method according to any one of claims 1 to 7.

Images