A 3D reconstruction method for bas-relief based on monocular images

Abstract

The present invention provides a method for three-dimensional reconstruction of bas-reliefs based on monocular images, which belongs to the field of image processing technology; a digital camera is used to capture relief images, a dense point cloud containing 3D coordinates and RGB values ​​is generated based on a photogrammetry method, and a bas-relief data set containing a monocular image and its semantics, soft edges and depth labels is established. A multi-task neural network framework for soft edge enhancement is trained based on the above-mentioned labeled data, in which semantics, edge and depth features are extracted from the monocular image at the same time, and semantics and edge features are used as effective clues for depth estimation tasks through a multi-task loss function and an edge contrast module, thereby enhancing the model's ability to estimate the depth of details and improving the accuracy of three-dimensional reconstruction. The present invention can directly extract corresponding soft edges and semantic information from a monocular relief photo, assist in depth estimation tasks, enhance its ability to capture details in edge areas, and effectively improve the accuracy of subsequent three-dimensional reconstruction of bas-reliefs.

Description

Technical Field

[0001] The invention provides a bas-relief three-dimensional reconstruction method based on a monocular image, and belongs to the technical field of image processing. Background Art

[0002] Reliefs are a fusion of sculpture and painting, typically attached to a flat surface and viewable from only one or two sides. They are generally categorized as shallow reliefs or high reliefs, depending on how high the carved designs and patterns protrude from the base. Shallow reliefs are more common on buildings and artifacts. For example, some bas-relief cultural heritage is often found at historical sites. However, due to natural or human factors, these precious cultural relics often suffer varying degrees of damage and loss. Traditional restoration processes often require extensive manual intervention and specialized archaeological expertise. While scanning and photogrammetry technologies can permanently preserve 3D digital models of reliefs in their current state, they cannot restore the reliefs' original appearance before damage. Therefore, utilizing the relief information contained in historical photographs to perform 3D reconstruction using monocular depth estimation has become an effective solution. However, due to the complex geometric structure of reliefs and the compression of depth values, it is difficult to accurately extract subtle depth variations in edge regions, thus affecting the accuracy of depth estimation. In summary, existing technologies either require multidimensional prior information beyond the monocular image or are unable to recover sufficient detail from monocular images.

[0003] Pan et al. (J. Pan, L. Li, H. Yamaguchi, K. Hasegawa, F.I. Thufail, Brahmantara, and S. Tanaka. 2020. Fused 3d Transparent Visualization for Large-Scale Cultural Heritage Using Deep Learning-Based Monocular Reconstruction. In ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol. V-2-2020. 989–996. https: / / doi.org / 10 / gs6hng.) proposed a 3D reconstruction method based on a monocular depth estimation network. They reconstructed the buried reliefs of the Borobudur temple. During the reconstruction process, features were extracted from the 3D point cloud data of the reliefs and projected into an edge-binarized image as prior knowledge. This image was then fed into the end of the monocular depth estimation network as auxiliary information, intervening in the decoding process and generating a depth map of the bas-relief with some details. By combining the grayscale image and the depth image and performing coordinate transformation, the 3D reconstruction of the bas-relief is achieved. This technology has the following disadvantages:

[0004] (1) The edge map is extracted from 3D points and used as an additional input, which significantly reduces the accuracy of their proposed model on the test data. This accuracy drop is because during the testing phase, 3D points from old photos are not available, so there is no effective way to obtain an accurate edge map.

[0005] (2) The soft edge map is incorporated into the network in the late part of the decoding stage, which limits its auxiliary ability for the depth estimation task.

[0006] (3) They represent the soft edge map as a binary image, distinguishing edge and non-edge regions as a mask for further processing. However, the binary representation cannot convey the 3D curvature changes inherent in soft edges, which can provide effective clues for depth estimation tasks. Summary of the Invention

[0007] The present invention provides a method for 3D reconstruction of bas-reliefs based on monocular images to solve the following technical problems:

[0008] (1) Insufficient auxiliary effect of additional information: In the existing technology, soft edge information is only introduced in the late decoding stage, and its positive impact on the depth estimation task is limited.

[0009] (2) The binary edge representation as additional information cannot effectively characterize the changes in 3D curvature, resulting in limited guidance and auxiliary effect on the network.

[0010] The specific technical solutions of the present invention are:

[0011] A method for 3D reconstruction of a bas-relief based on a monocular image comprises the following steps:

[0012] Step (1): using a digital camera to collect bas-relief image information of a specified scene: taking photos with a digital camera and using a motion multi-view stereo structure method to perform photogrammetry;

[0013] Step (2): Create a small amount of 3D point cloud data based on photogrammetry: Generate a small amount of 3D points from the photogrammetry data. These scan points contain 3D coordinates and RGB value information.

[0014] Step (3): Create a small bas-relief image dataset containing monocular images and their semantic, edge, and depth labels;

[0015] Step (3-1): Obtain a monocular image of the relief: project the 3D point cloud of the relief from the front view into the monocular image;

[0016] Step (3-2): Obtaining the semantic label of the relief: Using the semantic segmentation category label, manually annotate the projected monocular image to obtain a semantic image;

[0017] Step (3-3): Obtain edge labels of the relief: Use the 3D edge feature extraction algorithm (Kawakami K, Hasegawa K, Li L, et al. Opacity-based edge highlighting for transparent visualization of 3D scanned point clouds [J]. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2020, 2: 373-380.) to set the correlation function of the curvature of each point in the point cloud in step (2) and its transparency (Tanaka S, Hasegawa K, Okamoto N, et al. See-through imaging of laser-scanned 3D cultural heritage objects based on stochastic rendering of large-scale point clouds [J]. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2016, 3: 73-80 and Uchida T, Hasegawa K, Li L, et al. Noise-robust transparent visualization of large-scale point clouds acquired by laser scanning[J].ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 161: 124-134.) The edge highlighting algorithm is applied to highlight the narrow area around the center line of the edge area with a brighter color, and slowly darken the part where the curvature of the engraving curve decreases, showing the characteristics of a "soft edge". This can retain information about the depth changes around the edge area to a certain extent. By projecting the orthographic view and dividing the pixel values ​​into intervals, a classified soft edge map is obtained; specifically, the edge information is classified and divided according to the sharpness in the three-dimensional space, and the edge extraction task is transformed from binary classification to multi-classification, which better assists in the acquisition of high-frequency information in the relief stone carving scene;

[0018] The production of edge labels includes the following steps:

[0019] Step (3-3-1): Obtain the projection image of the soft edge in the measured point cloud by orthographic projection. The pixel value of the edge area depends on the correlation function of curvature and transparency.

[0020] Step (3-3-2): normalize the value range of each pixel value of the above-mentioned projection image to [0-255];

[0021] Step (3-3-3): Soft edges are classified into five categories based on pixel value magnitude, and the pixel values ​​belonging to each category are represented by the median value of that category. Scan points. In this algorithm, the local point distribution is evaluated by measuring changes in curvature, which typically disappears in a planar distribution. The local point distribution is correlated with the opacity of the points. The narrow area around the centerline of the soft edge region exhibits relatively high opacity and is highlighted with a brighter color. This preserves some information about depth variations around the edge region.

[0022] Step (3-4): Obtain the depth image of the relief: linearly transform the Z coordinate through orthographic projection to obtain a depth map with corresponding pixel values ​​of 0-255.

[0023] Step (4): Build and train a multi-task network model and constrain the depth estimation task through the edge comparison module;

[0024] Step (4-1): Construct an edge category extractor and form an edge comparison module to extract the soft edge map in step (3-3).

[0025] Step (4-1-1): Build an edge category extractor: Build a lightweight dense edge extraction network to extract the edge labels of step (3-3) from the input image.

[0026] Step (4-1-2): Construct an edge comparison module: Use the above-mentioned edge category extractor to extract the edge image in step 3-1 from the input image and depth estimation image of the multi-task network, compare them, and calculate the error between the two edge images.

[0027] Step (4-2): Use a multi-task neural network framework to extract depth, semantic and edge features, and then constrain the depth estimation task through the edge comparison module.

[0028] Step (4-2-1): Use the SAM ViT encoder and Swin v2 Transformer encoder to extract the underlying features in the input monocular image.

[0029] Step (4-2-2): The depth information decoder gradually restores these features into a depth map, and the semantic information decoder gradually restores these features into a semantic map corresponding to the monocular image. The edge extractor obtains soft edge classification maps from the input image and output depth image respectively, and calculates the difference between the two through the edge comparison module.

[0030] Step (4-2-3): The above-mentioned depth estimation, semantic segmentation and edge comparison modules are jointly trained and optimized through a multi-task network.

[0031] Step (4-3): Use the relief image dataset obtained in step (3) to train the multi-task network model. By introducing the edge contrast module, the soft edge information extracted from the monocular image and the depth map are compared to help the depth estimation task focus on the details of the edge area more accurately.

[0032] Step (5): Use the above multi-task model to infer different bas-relief monocular images, obtain the corresponding depth map and build a three-dimensional model.

[0033] The invention proposes a multi-task neural network architecture that simultaneously performs depth estimation, semantic segmentation, and edge detection. By integrating these tasks, it can more comprehensively extract multimodal features from a single old photo, such as depth maps, semantic information, and edge information, providing a deeper understanding and analysis of relief scenes.

[0034] This invention redefines the task of soft edge detection for relief. While traditional edge detection primarily relies on brightness changes to determine edges, this invention maps curvature changes in three-dimensional space into a multi-classification problem to identify the "softness" of relief edges. This approach enables more accurate depth estimation in edge regions.

[0035] The present invention introduces an edge matching module, which optimizes the depth estimation task by comparing the edge information in the input image and the depth map, so that the network pays more attention to the details of the edge area and improves the depth estimation accuracy of the edge area.

[0036] In order to further optimize the training of the neural network, the present invention adopts a dynamic edge enhancement loss function, which dynamically adjusts the weight of the edge detection task according to the accuracy of the depth estimation, thereby improving the performance of the network when processing edge areas.

[0037] Compared with the existing technology, the technical effects of this application are:

[0038] By using a small amount of relief image data and multiple labels to train a multi-task neural network, the present invention can directly extract soft-edge images from a single old photo, effectively providing additional information for the depth estimation task. Through the edge matching module, soft-edge information is introduced into the depth estimation task at an early stage of the network, enhancing the ability to capture details in the edge area and effectively improving the accuracy of the overall depth estimation. Through this precise depth estimation task, combined with monocular images, corresponding high-precision point cloud data can be generated to complete the three-dimensional reconstruction task of the bas-relief. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flow chart of the present invention;

[0040] Figure 2 This is a schematic diagram of the multi-classification soft edge extraction network and its results in the preliminary research experiment of the present invention;

[0041] Figure 3 It is a multi-task network framework under edge information enhancement of the present invention;

[0042] Figure 4 The semantic diagram of the Borobudur relief of the embodiment;

[0043] Figure 5 The edge diagram of the Borobudur relief is an embodiment;

[0044] Figure 6 Depth map of the Borobudur relief in the embodiment;

[0045] Figure 7 The figures are compared with the actual values ​​of the embodiment and the best results obtained by the method of the present invention. DETAILED DESCRIPTION

[0046] The specific technical solutions of the present invention are described with reference to the embodiments.

[0047] like Figure 1 As shown, a method for 3D reconstruction of bas-relief based on monocular images includes the following steps:

[0048] Step (1): Use a digital camera to collect bas-relief image information of a specified scene: take photos with a certain pixel resolution using a digital camera, and use the structure of a motion multi-view stereo method to perform photogrammetry.

[0049] Step (2): Create a small amount of 3D point cloud data based on photogrammetry: Use the commercial software Agisoft Metashape to generate dense 3D points from the photogrammetry data. These scan points contain 3D coordinates and RGB value information.

[0050] Step (3): Create a small bas-relief image dataset containing monocular images and their semantic, edge, and depth labels;

[0051] Step (3-1): Obtain a monocular image of the relief: project the 3D point cloud of the relief from the front view into the monocular image;

[0052] Step (3-2): Obtaining the semantic label of the relief: Using the semantic segmentation category label, manually annotate the projected monocular image to obtain a semantic image;

[0053] Step (3-3): Obtain the edge label of the relief: Use the 3D edge feature extraction algorithm to set the correlation function between the curvature of each point in the point cloud in step (2) and its transparency, highlight the narrow area around the center line of the edge area with a brighter color, and slowly darken the part where the curvature of the carving curve decreases, showing the characteristics of a "soft edge". This can retain information about the depth changes around the edge area to a certain extent. Through the front view projection and dividing the pixel values ​​into intervals, a classified soft edge map is obtained; specifically, the edge information is classified and divided according to the sharpness in the three-dimensional space, and the edge extraction task is transformed from binary classification to multi-classification, which better assists in the acquisition of high-frequency information in the relief stone carving scene;

[0054] The production of edge labels includes the following steps:

[0055] Step (3-3-1): Obtain the projection image of the soft edge in the measured point cloud by orthographic projection. The pixel value of the edge area depends on the correlation function of curvature and transparency.

[0056] Step (3-3-2): normalize the value range of each pixel value of the above-mentioned projection image to [0-255];

[0057] Step (3-3-3): Classify the soft edges into the following six categories according to the pixel values, and represent the pixel values ​​belonging to each category with the median value of the category, as shown in Table 1 below;

[0058] Table 1: Soft edge classification interval

[0059] scope 0-49 50-99 100-149 150-199 200-249 250-255 category 0 1 2 3 4 5

[0060] Step (3-4): Obtain the depth image of the relief: linearly transform the Z coordinate through orthographic projection to obtain a depth map with corresponding pixel values ​​of 0-255.

[0061] Step (4): Build and train a multi-task network model and constrain the depth estimation task through the edge comparison module;

[0062] Step (4-1): Construct an edge category extractor and form an edge comparison module to extract the soft edge map in step (3-3).

[0063] Step (4-1-1): Build an edge category extractor: Build a lightweight dense edge extraction network, such as Figure 2 , used to extract edge labels of step (3-3) from the input image.

[0064] Step (4-1-2): Construct an edge comparison module: Use the above-mentioned edge category extractor to extract the edge image in step 3-1 from the input image and depth estimation image of the multi-task network, compare them, and calculate the error between the two edge images.

[0065] Step (4-2): Use Figure 3 The multi-task neural network framework shown extracts depth, semantic and edge features, and then constrains the depth estimation task through the edge comparison module.

[0066] Step (4-2-1): Use the SAM ViT encoder and Swin v2 Transformer encoder to extract the underlying features in the input monocular image.

[0067] Step (4-2-2): The depth information decoder gradually restores these features into a depth map, and the semantic information decoder gradually restores these features into a semantic map corresponding to the monocular image. The edge extractor obtains soft edge classification maps from the input image and output depth image respectively, and calculates the difference between the two through the edge comparison module.

[0068] Step (4-2-3): The above-mentioned depth estimation, semantic segmentation and edge comparison modules are jointly trained and optimized through a multi-task network.

[0069] Step (4-3): Use the relief image dataset obtained in step (3) to train the multi-task network model. By introducing the edge contrast module, the soft edge information extracted from the monocular image and the depth map are compared to help the depth estimation task focus on the details of the edge area more accurately.

[0070] Step (5): Use the above multi-task model to infer different bas-relief monocular images, obtain the corresponding depth map and build a three-dimensional model.

[0071] This embodiment takes the Borobudur relief as an example:

[0072] Step (1): Use a digital camera to capture images of the Borobudur relief: Borobudur is 2.7 meters long, 0.92 meters high, and 0.15 meters deep. Photos were taken with a digital camera (RICOH GR III) at a resolution of 6000 × 4000 pixels. The distance from the camera to the temple building was approximately 2 meters, and each photo overlapped approximately 60%. Photogrammetry was performed using the structure from motion multi-view stereo method.

[0073] Step (2): Create a 3D point cloud of the Borobudur relief based on photogrammetry: Use the commercial software Agisoft Metashape to generate dense 3D points from the photogrammetry data.

[0074] Step (3): Establish a Borobudur relief image dataset containing monocular images of Borobudur relief and their semantic, edge, and depth labels;

[0075] Step (3-1): Obtain a monocular image of the Borobudur relief: Project the 3D point cloud of the relief from the front view into a monocular image with a pixel size of 3072×1024;

[0076] Step (3-2): Obtain the semantic labels of the Borobudur relief: Use semantic segmentation category labels and manually annotate the projected monocular image to divide the data into four categories: background (black), characters (red), plants (green) and others (blue), and obtain semantic images, such as Figure 4 As shown;

[0077] Step (3-3): Obtain edge labels for the Borobudur relief: Use the 3D edge feature extraction algorithm to set the correlation function between the curvature and transparency of each point in the point cloud in step (2), highlight the narrow area around the center line of the edge area with a brighter color, and slowly darken the part where the curvature of the carving curve decreases, showing a "soft edge" feature. By projecting the front view and dividing the pixel values ​​into intervals, a classified soft edge map is obtained, such as Figure 5 shown.

[0078] The production of edge labels includes the following steps:

[0079] Step (3-3-1): Obtain the projection image of the soft edge in the measured point cloud by orthographic projection. The pixel value of the edge area depends on the correlation function of curvature and transparency.

[0080] Step (3-3-2): normalize the value range of each pixel value of the above-mentioned projection image to [0-255];

[0081] Step (3-3-3): Divide the soft edges into six categories according to the pixel values, and represent the pixel values ​​belonging to each category with the median value of the category, as shown in Table 2 below.

[0082] Table 2: Soft edge classification interval

[0083] scope 0-49 50-99 100-149 150-199 200-249 250-255 category 0 1 2 3 4 5 Numerical 0 51 102 153 204 255

[0084] Step (3-4): Obtain the depth image of the Borobudur relief: By using the orthographic projection method, convert the Z coordinate to the X-axis to obtain the corresponding depth map with pixel values ​​of 0-255, as shown in Figure 6 shown.

[0085] Step (4): Use the Borobudur dataset obtained in step (3) to train a multi-task network model and constrain the depth estimation task through the edge comparison module;

[0086] Step (4-1): Use the edge category extractor to extract the edge image in step (3-1) from the input image and depth estimation image of the multi-task network respectively, compare them, and calculate the error between the two edge images.

[0087] Step (4-2): Implement the multi-task network model in Pytorch and train it on the dataset obtained in step (3) on a QuadroRTX 6000 GPU with 24GB of GPU memory. The weights of the SAM ViT encoder are initialized from the ViT-BSAM model, and the weights of the SwinV2 Transformer encoder are pre-trained on ImageNet. Use the Adam optimizer and the OneCycleLR learning rate scheduler with a maximum learning rate of 0.01 and a percentage of 0.2 for 100 epochs.

[0088] Step (4-3): Use a multi-task network model to extract the depth, semantic and edge features of the Borobudur relief monocular image;

[0089] Step (4-4): Use the Borobudur relief image dataset obtained in step (3) to train a multi-task network model. By introducing an edge comparison module, the soft edge information extracted from the monocular image and the depth map are compared to help the depth estimation task focus on the details of the edge area more accurately.

[0090] Step (5): Use the trained multi-task network model to infer the monocular image of the Borobudur relief, obtain the corresponding depth map and build a three-dimensional model.

[0091] Finally, the best result obtained in this embodiment is compared with the true value. Figure 7 shown.

Claims

1. A method for 3D reconstruction of bas-relief based on monocular images, characterized in that: The following steps are involved: Step (1): Use a digital camera to collect bas-relief image information of a specified scene: take photos with a digital camera and use the motion multi-view stereo structure method to perform photogrammetry; Step (2): Create 3D point cloud data based on photogrammetry: Generate 3D point cloud from photo data based on photogrammetry. These points contain 3D coordinates and RGB value information. Step (3): Create a small bas-relief image dataset containing monocular images and their semantic, edge, and depth labels; It includes the following sub-steps: Step (3-1): Obtain a monocular image of the relief: Project the 3D point cloud of the relief from the front view into the monocular image; Step (3-2): Obtain the semantic label of the relief: establish semantic segmentation category labels and manually annotate the projected monocular image to obtain a semantic image; Step (3-3): Obtain the edge label of the relief: Use the 3D edge feature extraction algorithm to set the correlation function between the curvature of each point in the point cloud in step (2) and its transparency, set the transparency of the narrow area around the center line of the edge area to the minimum value to highlight it, and slowly increase the transparency of the part where the curvature of the carving curve decreases so that it slowly darkens visually, showing the characteristics of a "soft edge"; project the front view and divide the pixel values ​​into intervals to obtain a classified soft edge map; specifically, classify and divide the edge information according to the sharpness in the three-dimensional space, transform the edge extraction task from binary classification to multi-classification, and assist in the extraction of high-frequency information in the relief stone carving scene; Step (3-4): Obtain the depth image of the relief: linearly transform the Z coordinate through orthographic projection to obtain the corresponding depth map with pixel values ​​of 0-255; Step (4): Build and train a multi-task network model and constrain the depth estimation task through the edge comparison module; It includes the following sub-steps: Step (4-1): Construct an edge category extractor and form an edge comparison module to extract the soft edge map in step (3-3); Step (4-2): Use a multi-task neural network framework to extract depth, semantic and edge features, and then constrain the depth estimation task through the edge comparison module; Step (4-3): Use the relief image dataset obtained in step (3) to train the multi-task network model. By introducing the edge comparison module, the soft edge information extracted from the monocular image and the depth map are compared to help the depth estimation task focus on the details of the edge area more accurately. Step (5): Use the above multi-task network model to infer different bas-relief monocular images, obtain the corresponding depth map and build a three-dimensional model.

2. The method for 3D reconstruction of bas-relief based on monocular image according to claim 1, characterized in that: The production of edge labels in step (3-3) includes the following steps: Step (3-3-1): Obtain the projection image of the soft edge in the measured point cloud by orthographic projection. The pixel value of the edge area depends on the correlation function of curvature and transparency. Step (3-3-2): normalize the value range of each pixel value of the above-mentioned projection image to [0-255]; Step (3-3-3): Divide the soft edges into six categories according to the pixel value, and represent the pixel values ​​belonging to each category with the median value of the category.

3. The method for 3D reconstruction of bas-relief based on monocular image according to claim 1, characterized in that: Step (4-1) specifically includes the following sub-steps: Step (4-1-1): Build an edge category extractor: Build a lightweight dense edge extraction network to extract the edge labels of step (3-3) from the input image; Step (4-1-2): Construct an edge comparison module: Use the above-mentioned edge category extractor to extract the edge image in step (3-1) from the input image of the multi-task network and the depth estimation image, compare them, and calculate the error between the two edge images.

4. The method for 3D reconstruction of bas-relief based on monocular image according to claim 1, characterized in that: Step (4-2) includes the following steps: Step (4-2-1): Use the SAM ViT encoder and Swin v2 Transformer encoder to extract the underlying features of the input monocular image; Step (4-2-2): The depth information decoder gradually restores these features into a depth map, and the semantic information decoder gradually restores these features into a semantic map corresponding to the monocular image. The edge extractor obtains soft edge classification maps from the input image and output depth image respectively, and calculates the difference between the two through the edge comparison module; Step (4-2-3): The above-mentioned depth estimation, semantic segmentation and edge comparison modules are jointly trained and optimized through a multi-task network.

Images