Binocular vision three-dimensional reconstruction method, system and program product for non-abandoned buildings in severe meteorological environment

Abstract

The invention relates to a binocular vision three-dimensional reconstruction method and system for non-abandoned buildings in a severe meteorological environment and a program product. The method comprises the following steps: synchronously acquiring left and right images and a plurality of meteorological data corresponding to non-abandoned buildings in a severe meteorological environment; recovering the left and right images through an image recovery model suitable for a severe meteorological environment; generating a building structure key point thermodynamic diagram of a non-abandoned building; generating a meteorological degradation factor probability graph according to the meteorological data; generating a disparity map according to the recovered left and right images, the building structure key point thermodynamic diagram and the meteorological degradation factor probability graph; and performing three-dimensional reconstruction based on the disparity map. According to the method, the binocular camera with the infrared filter is used for shooting, meteorological noise is suppressed, and the feature matching failure rate is reduced; information such as texture details of non-abandoned buildings can be enhanced by using a building structure key point thermodynamic diagram; and a meteorological degradation factor probability graph is used to further reduce meteorological noise interference, and high-precision digital modeling and all-weather acquisition of non-abandoned buildings are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, and in particular to a binocular vision three-dimensional reconstruction method, system and program product for intangible cultural heritage buildings in severe weather environment. BACKGROUND

[0002] The three-dimensional reconstruction of intangible cultural heritage buildings is a process of high-precision digital modeling and virtual restoration of buildings in intangible cultural heritage using modern digital technology. Intangible cultural heritage building monitoring requires uninterrupted data collection throughout the year, but existing technologies cannot cope with sudden severe weather. Existing technologies often use the following methods:

[0003] (1) Laser radar scanning: Laser radar emits high-density point clouds to obtain high-precision three-dimensional surface information of the building. This method has a large ranging error in rain and fog, which can reach ± 15mm.

[0004] (2) Traditional photogrammetry: images are taken by ordinary photographic equipment, and feature extraction and fusion technology is used to complete the three-dimensional reconstruction of intangible cultural heritage buildings. This method has a high failure rate in low visibility, with a failure rate of more than 60%.

[0005] (3) Existing image restoration algorithms do not consider the specificity of building textures when removing rain and fog interference (such as DehazeNet), resulting in the loss of details such as corbel arches and carved decorations. SUMMARY

[0006] The technical problem to be solved by the present application is to provide a binocular vision three-dimensional reconstruction method, system and program product for intangible cultural heritage buildings in severe weather environment.

[0007] The technical solution adopted by the present application to solve the technical problem is: a binocular vision three-dimensional reconstruction method for intangible cultural heritage buildings in severe weather environment, comprising the following steps:

[0008] S1, synchronously acquiring left and right images corresponding to intangible cultural heritage buildings in severe weather environment and a plurality of weather data; wherein the left and right images are obtained by a binocular camera with an infrared filter;

[0009] S2, restoring the left and right images by an image restoration model suitable for the severe weather environment;

[0010] S3, generating a building structure key point heat map of the intangible cultural heritage buildings;

[0011] S4, generating a weather degradation factor probability map according to the weather data;

[0012] S5, generating a disparity map according to the restored left and right images, the building structure key point heat map and the weather degradation factor probability map;

[0013] S6, performing three-dimensional reconstruction based on the disparity map.

[0014] Further, when the severe weather environment is foggy, the step S2 comprises:

[0015] An end-to-end defogging model is constructed by taking an atmospheric scattering model as a constraint.

[0016] The left and right images are defogged using the defogging model to obtain recovered left and right images.

[0017] Further, the defogging model outputs an atmospheric light map and a transmittance map after defogging the left and right images.

[0018] Further, when the severe weather environment is rainy or snowy, the step S2 comprises:

[0019] A rain removal model or a snow removal model is constructed based on a generative adversarial network model.

[0020] The left and right images are defogged using the defogging model to obtain recovered left and right images.

[0021] Further, the step S4 comprises:

[0022] Multi-scale features are extracted from the weather data through a shared encoder, and the weather degradation factor probability map is generated through parallel decoding.

[0023] Further, the step S5 comprises:

[0024] Key points in the building structure key point heat map are extracted.

[0025] The key points are used as feature points, and stereo matching is performed in combination with the recovered left and right images, and the cost aggregation weight is dynamically adjusted according to the weather degradation factor probability map to generate the disparity map.

[0026] Further, the weather data comprises one or more of visibility, humidity, temperature, light intensity, particulate matter data, and wind direction.

[0027] Further, the binocular camera adopts a 490nm infrared filter.

[0028] The application also provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the binocular vision three-dimensional reconstruction method of non-heritage buildings in severe weather environments according to any one of the above.

[0029] The application also provides a binocular vision three-dimensional reconstruction system for non-heritage buildings in severe weather environments, comprising:

[0030] Binocular camera with infrared filter

[0031] Several meteorological sensors

[0032] The edge computing unit comprises a memory and a processor, the memory stores a computer program, and the processor executes the steps of the binocular vision three-dimensional reconstruction method of the intangible cultural heritage building in severe weather environment by calling the computer program stored in the memory.

[0033] The implementation of the present application has the following beneficial effects: using a binocular camera with an infrared filter to shoot an intangible cultural heritage building in severe weather environment can suppress weather noise and reduce the feature matching failure rate; generating a disparity map based on a building structure key point heat map can enhance the texture details and other information of the intangible cultural heritage building; using a weather degradation factor probability map when generating the disparity map can further reduce the interference of weather noise, thereby realizing high-precision digital modeling of the intangible cultural heritage building. BRIEF DESCRIPTION OF DRAWINGS

[0034] The present application will be further described below in conjunction with the drawings and embodiments, wherein:

[0035] Figure 1 is a flowchart of the binocular vision three-dimensional reconstruction method of the intangible cultural heritage building in severe weather environment according to an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to have a clearer understanding of the technical features, objectives and effects of the present application, the specific embodiments of the present application will be described in detail with reference to the drawings. In the following description, specific details such as specific system structures, techniques, etc. are presented in order to explain, not to limit, so as to thoroughly understand the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices and methods are omitted to avoid unnecessary details that hinder the description of the present application.

[0037] Intangible cultural heritage building monitoring requires uninterrupted data collection throughout the year for three-dimensional reconstruction. During this period, fog, haze, rain, snow and other severe weather conditions that affect the clarity of collected images will pose the following challenges to three-dimensional reconstruction: how to solve the problem of unclear images caused by weather noise in severe weather environment and avoid loss of building texture.

[0038] As shown in FIG. 1, in an embodiment of the binocular vision three-dimensional reconstruction method of the intangible cultural heritage building in severe weather environment according to the present application, the following steps are included: Figure 1

[0039] ​S1, synchronously acquire left and right images and meteorological data corresponding to the non-heritage building in the severe weather environment; wherein the left and right images are obtained by a binocular camera with an infrared filter.

[0040] In this embodiment, the left and right lens spacing of the binocular camera is adjustable, and the images obtained by the left and right lenses are left and right images respectively. The binocular camera uses a 490nm infrared filter. In other embodiments, the binocular camera can also use other specifications of filters. Using this binocular camera to shoot non-heritage buildings in severe weather environments, the infrared filter can suppress weather noise and improve the feature matching success rate of the acquired left and right images.

[0041] When shooting images, meteorological data of the area where the non-heritage building is located is synchronously acquired, including one or more of visibility, humidity, temperature, light intensity, particulate matter data, and wind direction. These meteorological data are collected by corresponding meteorological sensors, for example, a visibility meter is used to monitor visibility, a rain gauge is used to monitor humidity, a temperature sensor is used to monitor temperature, a light intensity sensor is used to monitor light intensity, a particulate matter sensor is used to monitor particulate matter data, and a wind direction sensor is used to collect wind direction. In other embodiments, other meteorological data can also be added according to actual needs.

[0042] S2, restore the left and right images by using an image restoration model suitable for severe weather environments.

[0043] It can be understood that the images shot in severe weather environments are disturbed by weather noise and have low clarity, so it is necessary to use an image restoration model to restore the clarity. Different weather environments have different effects on images, so it is necessary to select a suitable image restoration model for the corresponding weather environment of the image, as follows:

[0044] (1) When the severe weather environment is foggy, step S2 includes: using an atmospheric scattering model as a constraint to construct an end-to-end defogging model; using the defogging model to defog the left and right images to obtain the restored left and right images.

[0045] Specifically, using a large amount of foggy image-clear image pairing data to train a neural network with an atmospheric scattering model as a constraint, a lightweight end-to-end defogging model is constructed, which is suitable for foggy and hazy environments. The defogging model is used to defog the left and right images in the foggy environment to obtain clear left and right images.

[0046] Further, the dehazing model can also be trained to output a clear image, an atmospheric light map and a transmittance map after dehazing the left and right images. The atmospheric light map is used to show the atmospheric light data, and the transmittance map is used to show the transmittance data. They can further optimize the dehazing effect. For example, if there is still some local fog residue in the dehazed left and right images, the atmospheric light map and the transmittance map can be used to adjust these areas to further improve the dehazing effect.

[0047] (2) When the severe weather environment is rain or snow, step S2 includes: constructing a de-rain model or a de-snow model based on the generative adversarial network model; using the de-rain model to de-rain the left and right images, or using the de-snow model to de-snow the left and right images to obtain the restored left and right images.

[0048] Specifically, a generative adversarial network (cGAN) is trained using a large amount of rain map-clear image paired data to construct a de-rain model, which aims to remove raindrops, rain streaks and other interference from the image to restore a clear background image. The de-rain model is used to de-rain the rainy left and right images to obtain clear de-rained left and right images.

[0049] A generative adversarial network (cGAN) is trained using a large amount of snow map-clear image paired data to construct a de-snow model, which is suitable for snowy environments. The de-snow model is used to de-snow the snowy left and right images to obtain clear de-snowed left and right images.

[0050] S3, generating a building structure key point heat map of the non-heritage building.

[0051] In this step, the building structure map with clear building texture corresponding to the left image or the right image is collected, the key points are labeled, and a key point detection model is constructed to generate a heat map.

[0052] S4, generating a weather degradation factor probability map according to the weather data.

[0053] In this embodiment, the weather degradation factor probability map is generated in the following manner: multi-scale features are extracted from the weather data by a shared encoder, and a weather degradation factor probability map is generated by parallel decoding.

[0054] Specifically, the various weather data obtained in step S1 is input into the shared encoder, which extracts the variation rules of the weather data at different spatial and temporal scales, and multiple decoders are used for parallel decoding to generate a probability map that integrates various weather degradation factors. The weather degradation factor probability map represents the probability of each pixel position being affected by weather factors. The higher the probability value, the more serious the weather noise interference on the pixel.

[0055] In other embodiments, the weather degradation factor probability map can also be generated by other existing methods.

[0056] S5, generating a disparity map based on the recovered left and right images, the building structure key point heat map, and the weather degradation factor probability map. The building structure key point heat map is used as auxiliary information for stereo matching of the left and right images to improve the accuracy and efficiency of matching and enhance the details of the non-heritage building. Meanwhile, based on the weather degradation factor probability map and a weather attention mechanism, the stereo matching cost aggregation process is dynamically adjusted to further suppress weather noise interference.

[0057] In an embodiment, step S5 includes: extracting key points in the building structure key point heat map; using the key points as feature points to perform stereo matching in combination with the recovered left and right images, and dynamically adjusting the cost aggregation weight based on the weather degradation factor probability map to generate a disparity map.

[0058] Specifically, high-confidence key points are extracted from the building structure key point heat map and input into a stereo matching algorithm as feature points. For each feature point, the matching cost in the left and right images is calculated, and a weather attention weight is set according to the weather degradation factor probability value of the feature point. For example, for areas with high weather degradation factor probability values, a lower weight is given during cost aggregation to avoid weather noise interference with the matching result; for areas with small weather degradation factor probability values, a higher weight is given. After adjusting the matching cost using the weather attention weight, the costs are aggregated, and the disparity with the smallest aggregated cost is selected as the final disparity value. A complete high-precision disparity map is generated based on the final disparity values of all feature points.

[0059] S6, performing three-dimensional reconstruction based on the disparity map.

[0060] In this step, a high-precision disparity map is used to perform three-dimensional reconstruction of the non-heritage building. The specific three-dimensional reconstruction method is described in the prior art.

[0061] It should be noted that the arrangement order of the above steps is only exemplary, and the execution order of the steps can be adjusted according to specific conditions in actual applications.

[0062] The application uses a binocular camera with an infrared filter to capture non-heritage buildings in harsh weather environments, which can suppress weather noise and reduce the feature matching failure rate. The building structure key point heat map is used to generate a disparity map, which can enhance the texture details and other information of the non-heritage building. The weather degradation factor probability map is used when generating the disparity map to further reduce weather noise interference, thereby realizing high-precision digital modeling of non-heritage buildings in harsh weather environments and all-weather acquisition.

[0063] In an embodiment of the binocular vision three-dimensional reconstruction system of the non-heritage building in the severe weather environment of the application, it comprises a binocular camera with an infrared filter, a plurality of weather sensors, an edge computing unit, including a memory and a processor, the memory stores a computer program, and the processor executes the steps of the binocular vision three-dimensional reconstruction method of the non-heritage building in the severe weather environment as in any of the above embodiments by calling the computer program stored in the memory. In this embodiment, the types of weather sensors are determined according to the required weather data, and the processor of the edge computing unit uses NVIDIA GPU.

[0064] In an embodiment of the computer program product of the application, the computer program product comprises a computer program, and the computer program is executed by the processor to realize the binocular vision three-dimensional reconstruction method of the non-heritage building in the severe weather environment as in any of the above embodiments.

[0065] In an embodiment, the computer program product can be a tangible product containing a computer program, such as a computer-readable storage medium storing a computer program. The readable storage medium can be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, etc. signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory (Flash), mechanical hard disk (HDD), solid state disk (SSD), etc. For example, the computer program product can be implemented as a non-volatile storage medium storing a computer program, such as a read-only memory, a NAND flash memory, etc. In an embodiment, the computer program product can be an intangible product containing a computer program. For example, the computer program product can be implemented as a virtual digital product, such as an executable file, an installation package, etc. digital files storing computer programs.

[0066] The code of the computer program can be written in one or more programming languages. Programming languages such as C, Java, C++, Python, etc. The program code can be executed entirely on the user computing device, or partially on the user computing device, or as a separate software package, or partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any kind of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (for example, through an Internet connection provided by an operator).

[0067] The computer program can be carried or transmitted by an electric, magnetic, optical, electromagnetic, infrared, or the like signal. The electronic device can convert the signal carrying the computer program into a digital signal, and then run the computer program. When the computer program is run on the electronic device, its code is used to make the electronic device perform (more specifically, can make the processor of the electronic device perform) the method steps of various exemplary embodiments of the present disclosure.

[0068] It can be understood that the above embodiments only express the preferred embodiments of the present application, and the description is more specific and detailed, but it cannot be understood as a limitation on the scope of the patent of the present application; it should be pointed out that for those skilled in the art, the above technical features can be freely combined without departing from the concept of the present application, and a number of modifications and improvements can be made, which belong to the protection scope of the present application; therefore, any equivalent transformation and modification within the scope of the claims of the present application shall belong to the scope of the claims of the present application.

Claims

1. A binocular vision 3D reconstruction method for intangible cultural heritage buildings in adverse weather environments, characterized by: The following steps are involved: S1. Synchronously acquiring left and right images corresponding to the intangible cultural heritage building and several types of meteorological data under a severe weather environment; wherein the left and right images are captured by a binocular camera with an infrared filter; S2. Restoring the left and right images using an image restoration model suitable for the severe weather environment; S3, generating a heat map of key points of the architectural structure of the intangible cultural heritage building; S4. generating a meteorological degradation factor probability map based on the meteorological data; S5. generating a disparity map based on the restored left and right images, the heat map of key points of the building structure, and the meteorological degradation factor probability map; S6. Perform three-dimensional reconstruction based on the disparity map.

2. The binocular vision 3D reconstruction method for intangible cultural heritage buildings in adverse weather conditions according to claim 1, characterized in that: When the severe weather environment is foggy, step S2 includes: Using the atmospheric scattering model as a constraint, an end-to-end dehazing model is constructed; The left and right images are defogged using the defogging model to obtain restored left and right images.

3. The binocular vision 3D reconstruction method for intangible cultural heritage buildings in adverse weather conditions according to claim 2, characterized in that: After defogging the left and right images, the defogging model also outputs an atmospheric light map and a transmittance map.

4. The binocular vision 3D reconstruction method for intangible cultural heritage buildings in adverse weather conditions according to claim 1, characterized in that: When the severe weather environment is rain or snow, step S2 includes: A rain removal model or snow removal model is constructed based on the generative adversarial network model; The left and right images are derained using the deraining model, or the left and right images are desnowed using the desnowing model to obtain restored left and right images.

5. The binocular vision 3D reconstruction method for intangible cultural heritage buildings in adverse weather conditions according to claim 1, characterized in that: Step S4 includes: Multi-scale features are extracted from the meteorological data through a shared encoder, and parallel decoding is performed to generate the meteorological degradation factor probability map.

6. The binocular vision 3D reconstruction method for intangible cultural heritage buildings in adverse weather conditions according to claim 1, characterized in that: Step S5 includes: Extracting key points from the key point thermal map of the building structure; The key points are used as feature points, and stereo matching is performed in combination with the restored left and right images. The cost aggregation weight is dynamically adjusted according to the meteorological degradation factor probability map to generate the disparity map.

7. The binocular vision 3D reconstruction method for intangible cultural heritage buildings in adverse weather conditions according to claim 1, characterized in that: The several types of meteorological data include one or more of visibility, humidity, temperature, light intensity, particulate matter data, and wind direction.

8. The binocular vision 3D reconstruction method for intangible cultural heritage buildings in adverse weather conditions according to claim 1, characterized in that: The binocular camera uses a 490nm infrared filter.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the binocular vision three-dimensional reconstruction method of intangible cultural heritage buildings in adverse weather conditions as described in any one of claims 1 to 8 are implemented.

10. A binocular vision 3D reconstruction system for intangible cultural heritage buildings in adverse weather conditions, characterized by: include: Binocular camera with infrared filter; Several types of meteorological sensors; An edge computing unit includes a memory and a processor, wherein a computer program is stored in the memory, and the processor executes the steps of the binocular vision three-dimensional reconstruction method of intangible cultural heritage buildings in a severe weather environment as described in any one of claims 1 to 8 by calling the computer program stored in the memory.

Images