Three-dimensional bas-relief reconstruction method based on monocular image

Abstract

The invention provides a bas-relief three-dimensional reconstruction method based on a monocular image, and belongs to the technical field of image processing. A digital camera is used for collecting relief images, dense point clouds containing 3D coordinates and RGB values are generated based on a photogrammetry method, and a bas-relief data set containing monocular images and semantics, soft edges and depth labels of the monocular images is established. Training a soft edge enhanced multi-task neural network framework based on the labeled data, simultaneously extracting semantic, edge and depth features in a monocular image in the framework, and taking the semantic and edge features as effective clues of a depth estimation task through a multi-task loss function and an edge comparison module; the depth estimation capability of the model on the detail part is enhanced, and the precision of three-dimensional reconstruction is improved. According to the method, the corresponding soft edge and semantic information can be directly extracted from a monocular relief photo, a depth estimation task is assisted, the capability of capturing details of an edge region is enhanced, and the precision of subsequent bas-relief three-dimensional reconstruction is effectively improved.

Description

Technical Field

[0001] The invention provides a 3D reconstruction method of bas-relief based on a monocular image, belonging to the technical field of image processing. Background Art

[0002] Relief is a product of the combination of sculpture and painting. It is usually attached to a plane and can only be viewed from one or two sides. Generally, it is divided into bas-relief and high relief according to the height of the carved patterns and patterns protruding from the bottom surface. Among them, bas-relief is more common in buildings and artifacts. For example, some bas-relief cultural heritages are common in historical sites, but due to natural or human factors, these precious cultural heritages often suffer varying degrees of damage and loss. The traditional restoration process usually requires a lot of manual intervention and professional archaeological knowledge. Although scanning and photogrammetry technologies can permanently preserve the three-dimensional digital models of reliefs in their current state, they cannot restore the original appearance of these reliefs before they were damaged. Therefore, using the relief information contained in historical photos to perform three-dimensional reconstruction through monocular depth estimation technology has become an effective solution. However, due to the complex geometric structure of the relief and the compression of the depth value, it is difficult to accurately extract subtle depth changes in the edge area, which affects the accuracy of depth estimation. In summary, the existing technology either requires multi-dimensional prior information other than the monocular image, or cannot restore enough details from the monocular image.

[0003] Pan et al. proposed a 3D reconstruction method based on a monocular depth estimation network, (J. Pan, L. Li, H. Yamaguchi, K. Hasegawa, F.I. Thufail, Brahmantara, and S. Tanaka. 2020. Fused 3d Transparent Visualization for Large-Scale Cultural Heritage Using Deep Learning-Based Monocular Reconstruction. In ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol. V-2-2020. 989–996. https: / / doi.org / 10 / gs6hng.) reconstructed the buried relief in the Borobudur Temple. In the reconstruction process, feature extraction is performed based on the 3D point cloud data of the relief, and it is projected into an edge binarization image as prior knowledge, and it is input as auxiliary information to the end of the monocular depth estimation network, intervening in the decoding process and generating a bas-relief depth map with some details. The 3D reconstruction of the bas-relief is achieved by combining the grayscale image and the depth image and performing coordinate transformation. This technology has the following disadvantages:

[0004] (1) The edge map is extracted from 3D points and used as an additional input, which significantly reduces the accuracy of their proposed model on the test data. This accuracy drop is because during the testing phase, 3D points from old photos are not available, so there is no effective way to obtain an accurate edge map.

[0005] (2) The soft edge map is incorporated into the network in the late part of the decoding stage, which limits its auxiliary ability for the depth estimation task.

[0006] (3) They represent the soft edge map as a binary image and distinguish edge and non-edge regions as a mask for further processing. However, the binary representation cannot convey the 3D curvature changes inherent in soft edges, which can provide effective clues for depth estimation tasks. Summary of the invention

[0007] The present invention provides a method for 3D reconstruction of bas-relief based on monocular images to solve the following technical problems:

[0008] (1) The problem of insufficient auxiliary effect of additional information: In the existing technology, soft edge information is only introduced in the late decoding stage, and its positive impact on the depth estimation task is limited.

[0009] (2) The binary edge representation as additional information cannot effectively characterize the change of 3D curvature, resulting in limited guiding and assisting effect on the network.

[0010] The specific technical solution of the present invention is:

[0011] A method for 3D reconstruction of a bas-relief based on a monocular image comprises the following steps:

[0012] Step (1): using a digital camera to collect bas-relief image information of a specified scene: taking photos with a digital camera, and using a motion multi-view stereo structure method to perform photogrammetry;

[0013] Step (2): Create a small amount of 3D point cloud data based on photogrammetry: Generate a small amount of 3D points from the photogrammetry data. These scanned points contain 3D coordinates and RGB value information.

[0014] Step (3): Establish a small bas-relief image dataset containing monocular images and their semantic, edge, and depth labels;

[0015] Step (3-1): Obtain a monocular image of the relief: project the three-dimensional point cloud of the relief from the front view into the monocular image;

[0016] Step (3-2): Obtaining the semantic label of the relief: Using the semantic segmentation category label, and manually annotating the projected monocular image to obtain a semantic image;

[0017] Step (3-3): Obtain edge labels of the relief: Use the 3D edge feature extraction algorithm (Kawakami K, Hasegawa K, Li L, et al. Opacity-based edge highlighting for transparent visualization of 3D scanned point clouds [J]. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2020, 2: 373-380.) to set the correlation function between the curvature of each point in the point cloud in step (2) and its transparency (Tanaka S, Hasegawa K, Okamoto N, et al. See-through imaging of laser-scanned 3D cultural heritage objects based on stochastic rendering of large-scale point clouds [J]. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2016, 3: 73-80 and Uchida T, Hasegawa K, Li L, et al. Noise-robust transparent visualization of large-scale point clouds acquired by laser scanning[J].ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 161:124-134.) The edge highlighting algorithm is applied to highlight the narrow area around the center line of the edge area with a brighter color, and slowly darken the part where the curvature of the engraved curve decreases, showing the characteristics of a "soft edge". This can retain information about the depth changes around the edge area to a certain extent. Through front view projection and dividing the pixel values ​​into intervals, a classified soft edge map is obtained; specifically, the edge information is classified and divided according to the sharpness in three-dimensional space, and the edge extraction task is transformed from binary classification to multi-classification, which better assists in the acquisition of high-frequency information in relief stone carving scenes;

[0018] The production of edge labels includes the following steps:

[0019] Step (3-3-1): Obtain the projection image of the soft edge in the measured point cloud by orthographic projection, and the pixel value of the edge area depends on the correlation function of curvature and transparency;

[0020] Step (3-3-2): normalize the value range of each pixel value of the above-mentioned projection image to [0-255];

[0021] Step (3-3-3): Divide the soft edge into five categories according to the pixel value, and represent the pixel value belonging to each category with the median value of the category. Scan points. In this algorithm, the local point distribution is evaluated by measuring the change in curvature, which usually disappears in a planar distribution. The local point distribution is related to the opacity of the point. The narrow area around the center line of the soft edge area shows relatively high opacity and is highlighted with a brighter color. This retains information about the depth change around the edge area to a certain extent.

[0022] Step (3-4): Obtain the depth image of the relief: Through the front view projection method, the Z coordinate is linearly transformed to obtain a depth map with a corresponding pixel value of 0-255.

[0023] Step (4): Build and train a multi-task network model, and constrain the depth estimation task through the edge comparison module;

[0024] Step (4-1): Construct an edge category extractor and form an edge comparison module to extract the soft edge map in step (3-3).

[0025] Step (4-1-1): Build an edge category extractor: Build a lightweight dense edge extraction network to extract the edge labels of step (3-3) from the input image.

[0026] Step (4-1-2): Form an edge comparison module: Use the above-mentioned edge category extractor to extract the edge image in step 3-1 from the input image and the depth estimation image of the multi-task network respectively, and compare them to calculate the error between the two edge images.

[0027] Step (4-2): Use a multi-task neural network framework to extract depth, semantic and edge features, and then constrain the depth estimation task through an edge comparison module.

[0028] Step (4-2-1): Use the SAM ViT encoder and the Swin v2 Transformer encoder to extract the underlying features in the input monocular image.

[0029] Step (4-2-2): The depth information decoder gradually restores these features into a depth map, and the semantic information decoder gradually restores these features into a semantic map corresponding to the monocular image. The edge extractor obtains the soft edge classification map from the input image and the output depth image respectively, and calculates the difference between the two through the edge comparison module.

[0030] Step (4-2-3): The above-mentioned depth estimation, semantic segmentation and edge comparison modules are jointly trained and optimized through a multi-task network.

[0031] Step (4-3): Use the relief image dataset obtained in step (3) to train the multi-task network model. By introducing the edge contrast module, the soft edge information extracted from the monocular image and the depth map are compared to help the depth estimation task focus on the details of the edge area more accurately.

[0032] Step (5): Use the above multi-task model to infer different bas-relief monocular images, obtain the corresponding depth map and establish a three-dimensional model.

[0033] The invention proposes a multi-task neural network architecture that simultaneously performs depth estimation, semantic segmentation, and edge detection tasks. By integrating these tasks, multimodal features such as depth maps, semantic information, and edge information can be more comprehensively extracted from a single old photo, providing a deeper understanding and analysis of the relief scene.

[0034] The present invention redefines the soft edge detection task of relief. Traditional edge detection mainly judges the edge by brightness change, while the present invention identifies the "softness" of relief edge by mapping the curvature change in three-dimensional space into a multi-classification problem. This method makes depth estimation more accurate in edge areas.

[0035] The present invention introduces an edge matching module, which optimizes the depth estimation task by comparing the edge information in the input image and the depth map, so that the network pays more attention to the details of the edge area and improves the depth estimation accuracy of the edge area.

[0036] In order to further optimize the training of the neural network, the present invention adopts a dynamic edge enhancement loss function, which dynamically adjusts the weight of the edge detection task according to the accuracy of the depth estimation, thereby improving the performance of the network when processing edge areas.

[0037] Compared with the prior art, the technical effects of this application are:

[0038] The present invention uses a small amount of relief image data and multiple labels to train a multi-task neural network, which can directly extract soft edge images from a single old photo, effectively providing additional information for the depth estimation task. Through the edge matching module, the soft edge information is introduced into the depth estimation task in the early stage of the network, which enhances the ability to capture details of the edge area and effectively improves the accuracy of the overall depth estimation. Through the above-mentioned precise depth estimation task, combined with the monocular image, the corresponding high-precision point cloud data can be generated to complete the three-dimensional reconstruction task of the bas-relief. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flow chart of the present invention;

[0040] Figure 2 A schematic diagram of a multi-classification soft edge extraction network and its results in a preliminary research experiment of the present invention;

[0041] Figure 3 It is a multi-task network framework under edge information enhancement of the present invention;

[0042] Figure 4 The semantic diagram of the Borobudur relief of the embodiment;

[0043] Figure 5 The edge diagram of the Borobudur relief of the embodiment;

[0044] Figure 6 A depth map of the Borobudur relief of an embodiment;

[0045] Figure 7 The figures are compared with the actual values ​​of the embodiment and the best results obtained by the method of the present invention. DETAILED DESCRIPTION

[0046] The specific technical solution of the present invention is explained in conjunction with embodiments.

[0047] like Figure 1 As shown, a method for 3D reconstruction of a bas-relief based on a monocular image comprises the following steps:

[0048] Step (1): Use a digital camera to collect bas-relief image information of a specified scene: Use a digital camera to take photos with a certain pixel resolution, and use the structure of a motion multi-view stereo method to perform photogrammetry.

[0049] Step (2): Create a small amount of 3D point cloud data based on photogrammetry: Use the commercial software Agisoft Metashape to generate dense 3D points from the photogrammetry data. These scan points contain 3D coordinates and RGB value information.

[0050] Step (3): Establish a small bas-relief image dataset containing monocular images and their semantic, edge, and depth labels;

[0051] Step (3-1): Obtain a monocular image of the relief: project the three-dimensional point cloud of the relief from the front view into the monocular image;

[0052] Step (3-2): Obtaining the semantic label of the relief: Using the semantic segmentation category label, and manually annotating the projected monocular image to obtain a semantic image;

[0053] Step (3-3): Obtain the edge label of the relief: Use the 3D edge feature extraction algorithm to set the correlation function between the curvature of each point in the point cloud in step (2) and its transparency, highlight the narrow area around the center line of the edge area with a brighter color, and slowly darken the part where the curvature of the engraving curve decreases, showing the characteristics of a "soft edge". This can retain information about the depth changes around the edge area to a certain extent. Through front view projection and dividing the pixel values ​​into intervals, a classified soft edge map is obtained; specifically, the edge information is classified and divided according to the sharpness in three-dimensional space, and the edge extraction task is transformed from binary classification to multi-classification, which better assists in the acquisition of high-frequency information in the relief stone carving scene;

[0054] The production of edge labels includes the following steps:

[0055] Step (3-3-1): Obtain the projection image of the soft edge in the measured point cloud by orthographic projection, and the pixel value of the edge area depends on the correlation function of curvature and transparency;

[0056] Step (3-3-2): normalize the value range of each pixel value of the above-mentioned projection image to [0-255];

[0057] Step (3-3-3): Divide the soft edges into the following six categories according to the pixel values, and represent the pixel values ​​belonging to each category with the median value of the category, as shown in Table 1 below;

[0058] Table 1: Soft edge classification interval

[0059] scope 0-49 50-99 100-149 150-199 200-249 250-255 category 0 1 2 3 4 5

[0060] Step (3-4): Obtain the depth image of the relief: Through the front view projection method, the Z coordinate is linearly transformed to obtain a depth map with a corresponding pixel value of 0-255.

[0061] Step (4): Build and train a multi-task network model, and constrain the depth estimation task through the edge comparison module;

[0062] Step (4-1): Construct an edge category extractor and form an edge comparison module to extract the soft edge map in step (3-3).

[0063] Step (4-1-1): Build an edge category extractor: Build a lightweight dense edge extraction network, such as Figure 2 , used to extract edge labels in step (3-3) from the input image.

[0064] Step (4-1-2): Form an edge comparison module: Use the above-mentioned edge category extractor to extract the edge image in step 3-1 from the input image and the depth estimation image of the multi-task network respectively, and compare them to calculate the error between the two edge images.

[0065] Step (4-2): Use Figure 3 The multi-task neural network framework shown extracts depth, semantic and edge features, and then constrains the depth estimation task through the edge comparison module.

[0066] Step (4-2-1): Use the SAM ViT encoder and the Swin v2 Transformer encoder to extract the underlying features in the input monocular image.

[0067] Step (4-2-2): The depth information decoder gradually restores these features into a depth map, and the semantic information decoder gradually restores these features into a semantic map corresponding to the monocular image. The edge extractor obtains the soft edge classification map from the input image and the output depth image respectively, and calculates the difference between the two through the edge comparison module.

[0068] Step (4-2-3): The above-mentioned depth estimation, semantic segmentation and edge comparison modules are jointly trained and optimized through a multi-task network.

[0069] Step (4-3): Use the relief image dataset obtained in step (3) to train the multi-task network model. By introducing the edge contrast module, the soft edge information extracted from the monocular image and the depth map are compared to help the depth estimation task focus on the details of the edge area more accurately.

[0070] Step (5): Use the above multi-task model to infer different bas-relief monocular images, obtain the corresponding depth map and establish a three-dimensional model.

[0071] This embodiment takes the Borobudur relief as an example:

[0072] Step (1): Collect images of Borobudur relief using a digital camera: Borobudur is 2.7 meters long, 0.92 meters high, and 0.15 meters deep. Photos were taken using a digital camera (RICOH GR III) with a resolution of 6000 × 4000 pixels. The distance from the camera to the temple building was about 2 meters, and each photo overlapped about 60%. Photogrammetry was performed using the structure from motion multi-view stereo method.

[0073] Step (2): Create a 3D point cloud of the Borobudur relief based on photogrammetry: Use the commercial software Agisoft Metashape to generate dense 3D points from the photogrammetry data.

[0074] Step (3): Establish a Borobudur relief image dataset containing monocular images of Borobudur relief and their semantic, edge, and depth labels;

[0075] Step (3-1): Obtain a monocular image of the Borobudur relief: project the 3D point cloud of the relief from the front view into a monocular image with a pixel size of 3072×1024;

[0076] Step (3-2): Obtain the semantic label of the Borobudur relief: Use the semantic segmentation category label and manually annotate the projected monocular image to divide the data into four categories: background (black), characters (red), plants (green) and others (blue), and obtain the semantic image, such as Figure 4 As shown;

[0077] Step (3-3): Obtain edge labels for the Borobudur relief: Use the 3D edge feature extraction algorithm to set the correlation function between the curvature and transparency of each point in the point cloud in step (2), highlight the narrow area around the center line of the edge area with a brighter color, and slowly darken the part where the curvature of the sculpture curve decreases, showing a "soft edge" feature. Through front view projection and dividing the pixel values ​​into intervals, a classified soft edge map is obtained, such as Figure 5 shown.

[0078] The production of edge labels includes the following steps:

[0079] Step (3-3-1): Obtain the projection image of the soft edge in the measured point cloud by orthographic projection, and the pixel value of the edge area depends on the correlation function of curvature and transparency;

[0080] Step (3-3-2): normalize the value range of each pixel value of the above-mentioned projection image to [0-255];

[0081] Step (3-3-3): Divide the soft edges into six categories according to the pixel values, and represent the pixel values ​​belonging to each category with the median value of the category, as shown in Table 2 below.

[0082] Table 2: Soft edge classification interval

[0083] scope 0-49 50-99 100-149 150-199 200-249 250-255 category 0 1 2 3 4 5 Numeric 0 51 102 153 204 255

[0084] Step (3-4): Obtain the depth image of the Borobudur relief: By using the orthographic projection method, convert the Z coordinate to the X-axis to obtain the corresponding depth map with pixel values ​​of 0-255, such as Figure 6 shown.

[0085] Step (4): Use the Borobudur dataset obtained in step (3) to train a multi-task network model, and constrain the depth estimation task through the edge comparison module;

[0086] Step (4-1): Use the edge category extractor to extract the edge image in step (3-1) from the input image and the depth estimation image of the multi-task network respectively, and compare them to calculate the error between the two edge images.

[0087] Step (4-2): Implement the multi-task network model in Pytorch and train the dataset obtained in step (3) on a QuadroRTX 6000 GPU with 24GB GPU memory. The weights of the SAM ViT encoder are initialized by the ViT-BSAM model, and the weights of the SwinV2 Transformer encoder are pre-trained on ImageNet. Use the Adam optimizer and the OneCycleLR learning rate scheduler, with a maximum learning rate of 0.01 and a percentage of 0.2 for a total of 100 epochs.

[0088] Step (4-3): Use a multi-task network model to extract the depth, semantic and edge features of the Borobudur relief monocular image;

[0089] Step (4-4): Use the Borobudur relief image dataset obtained in step (3) to train a multi-task network model. By introducing an edge contrast module, the soft edge information extracted from the monocular image and the depth map are compared to help the depth estimation task focus on the details of the edge area more accurately.

[0090] Step (5): Use the trained multi-task network model to infer the monocular image of the Borobudur relief, obtain the corresponding depth map and establish a three-dimensional model.

[0091] Finally, the best result obtained in this embodiment is compared with the true value. Figure 7 shown.

Claims

1. A method for 3D reconstruction of bas-relief based on monocular images, characterized in that: The following steps are involved: Step (1): using a digital camera to collect bas-relief image information of a specified scene: taking photos with a digital camera, and using a motion multi-view stereo structure method to perform photogrammetry; Step (2): Establishing 3D point cloud data based on photogrammetry: Generate a 3D point cloud from the photo data based on photogrammetry, where the points contain 3D coordinates and RGB value information; Step (3): Establish a small bas-relief image dataset containing monocular images and their semantic, edge, and depth labels; Step (4): Build and train a multi-task network model, and constrain the depth estimation task through the edge comparison module; Step (5): Use the above multi-task model to infer different bas-relief monocular images, obtain the corresponding depth map and establish a three-dimensional model.

2. The method for 3D reconstruction of a bas-relief based on a monocular image according to claim 1, characterized in that: Step (3) specifically includes the following sub-steps: Step (3-1): Obtain a monocular image of the relief: project the three-dimensional point cloud of the relief from the front view into the monocular image; Step (3-2): Obtaining the semantic label of the relief: Establishing the semantic segmentation category label, and manually annotating the projected monocular image to obtain the semantic image; Step (3-3): Obtain the edge label of the relief: Use the 3D edge feature extraction algorithm to set the correlation function between the curvature of each point in the point cloud in step (2) and its transparency, set the transparency of the narrow area around the center line of the edge area to the minimum value to highlight it, and slowly increase the transparency of the part where the curvature of the carving curve decreases so that it slowly darkens visually, showing the characteristics of a "soft edge"; project the front view and divide the pixel values ​​into intervals to obtain a classified soft edge map; specifically, classify and divide the edge information according to the sharpness in the three-dimensional space, transform the edge extraction task from binary classification to multi-classification, and assist in the extraction of high-frequency information in the relief stone carving scene; Step (3-4): Obtain the depth image of the relief: Through the front view projection method, the Z coordinate is linearly transformed to obtain a depth map with a corresponding pixel value of 0-255.

3. The method for 3D reconstruction of a bas-relief based on a monocular image according to claim 2, characterized in that: The preparation of edge labels in step (3-3) includes the following steps: Step (3-3-1): Obtain the projection image of the soft edge in the measured point cloud by orthographic projection, and the pixel value of the edge area depends on the correlation function of curvature and transparency; Step (3-3-2): normalize the value range of each pixel value of the above-mentioned projection image to [0-255]; Step (3-3-3): Divide the soft edges into six categories according to the pixel values, and represent the pixel values ​​belonging to each category with the median value of the category.

4. The method for 3D reconstruction of bas-relief based on monocular image according to claim 2, characterized in that: Step (4) specifically includes the following sub-steps: Step (4-1): construct an edge category extractor and form an edge comparison module to extract the soft edge map in step (3-3); Step (4-2): Use a multi-task neural network framework to extract depth, semantic and edge features, and then constrain the depth estimation task through an edge comparison module; Step (4-3): Use the relief image dataset obtained in step (3) to train the multi-task network model. By introducing the edge contrast module, the soft edge information extracted from the monocular image and the depth map are compared to help the depth estimation task focus on the details of the edge area more accurately.

5. The method for 3D reconstruction of bas-relief based on monocular image according to claim 4, characterized in that: Step (4-1) specifically includes the following sub-steps: Step (4-1-1): Construct an edge category extractor: Build a lightweight dense edge extraction network to extract edge labels in step (3-3) from the input image; Step (4-1-2): Form an edge comparison module: Use the above-mentioned edge category extractor to extract the edge image in step (3-1) from the input image and depth estimation image of the multi-task network respectively, and compare them to calculate the error between the two edge images.

6. The method for 3D reconstruction of bas-relief based on monocular image according to claim 4, characterized in that: Step (4-2) includes the following steps: Step (4-2-1): Use the SAM ViT encoder and the Swin v2 Transformer encoder to extract the underlying features in the input monocular image; Step (4-2-2): The depth information decoder gradually restores these features into a depth map, and the semantic information decoder gradually restores these features into a semantic map corresponding to the monocular image. The edge extractor obtains the soft edge classification map from the input image and the output depth image respectively, and calculates the difference between the two through the edge comparison module; Step (4-2-3): The above-mentioned depth estimation, semantic segmentation and edge comparison modules are jointly trained and optimized through a multi-task network.

Images