ISfM-PMVS fluvial sediment digital modeling method based on images collected by smart phone

Abstract

The invention relates to the technical field of fluvial sediment digitization, in particular to an ISfM-PMVS fluvial sediment digitization modeling method based on images collected by a smartphone, high-resolution multi-view images can be obtained in a short time through an ISfM method in combination with the smartphone, and the method is suitable for quickly constructing a three-dimensional digital model. By combining SfM and PMVS technologies, a high-density three-dimensional point cloud and a digital orthoimage (DOM) can be generated, so that data not only contain geometric information, but also contain rich color and texture information; three-dimensional model parameters are optimized through RANSAC and bundle adjustment (BA), and the accuracy of sparse point clouds is ensured. The portable characteristic of the smart phone allows efficient data acquisition in complex terrains and harsh environments, and the generated data model can accurately reflect the actual conditions of geological points.

Description

Technical Field

[0001] The present invention relates to the technical field of river sediment digitization, and in particular to an ISfM-PMVS river sediment digitization modeling method based on images collected by a smart phone. Background Art

[0002] With the development of science and technology, the continuous progress of modern surveying and mapping technologies such as three-dimensional laser scanners (Three Dimensional Laser Scanner) and unmanned aerial vehicles (UAV) for earth observation and spatial information recording can record and track real-life three-dimensional scenes faster, easier and more detailed, which can effectively enrich the geological database.

[0003] Compared with two-dimensional images and text information, three-dimensional digital models make data rich in diversity. Many scholars are working in the fields of earth sciences such as quantitative interpretation of geological outcrops, precise measurement of glacial landform changes, and neotectonic activities and their hazards. Usually, high-overlapping images are obtained through UAVs, and then digital models are constructed in combination with the Structure from Motion (SfM) algorithm to carry out various studies. With the rapid development of computer vision, in order to make it easier to store, share and interpret information such as geological outcrops and geological relics, some scholars have gradually tried to use high-definition photos taken by smartphones to build 3D models. The model can record field information and ensure that researchers can digitally interpret and analyze it. Some scholars have carried out related research work. For example, Stefano et al. compared the accuracy of the model built by smartphones with the 3D model built by UAV in the natural rock cliff of Pietrasecca Village in central Italy. The results showed that smartphones can obtain digital models that meet certain accuracy requirements. Chen et al. built a three-dimensional point cloud model of the rock outcrops in Nanwang Mountain and Guilin based on the image matching algorithm to measure the discontinuous direction of the rock outcrops. An et al. established digital models of mud limestone or mud siltstone in lithologic landslide zones based on smartphones and measured the morphological characteristics of rocks by means of digital images. Tavani et al. established digital models of the “Acquedotto Felice” section in Rome, Italy and three geological outcrops in the Aman Mountains using images taken by smartphones. Through accuracy verification, it was found that high-definition images taken by smartphones can effectively interpret geological information. Tavani et al. used the inertial measurement unit (IMU) and magnetometer available on some smartphones to collect directional data and verified that the error was less than 3.6°, which was basically consistent with the results of traditional compass measurements, indicating that smartphones have important potential in assisting geological field work. To ensure the effectiveness of smartphones in establishing digital models of actual slag piles in mining environments, Tungol et al. first used smartphones to model larger, angular rocks and smaller, round rocks, and then quantitatively evaluated the errors of SfM for three-dimensional reconstruction. The results showed that the accuracy of the digital models established by smartphones was guaranteed, and the more images in the same data set, the smaller the scale error. Corradetti et al. conducted a digital modeling study on a trench about 20 meters long in Enemonzo, Italy. The results showed that although the digital model built by the smartphone had a rotation error of 13°, it could meet the accuracy requirements of three-dimensional digital modeling to a certain extent.

[0004] In summary, many scholars at home and abroad have been conducting research on digital models constructed using the SfM method based on smartphone photos, but many scholars only interpret digital models of a single dimension such as point clouds or DOM. This paper selects three geological points to quickly obtain photos of geological relics through smartphones, and uses intelligent algorithms to interpret the unique geological information of different geological points based on digital models of different dimensions. Summary of the invention

[0005] The purpose of the present invention is to provide an ISfM-PMVS river sediment digital modeling method based on images collected by a smart phone. The ISfM method combined with a smart phone can obtain high-resolution multi-view images in a short time, which is suitable for quickly building a three-dimensional digital model; combining ISfM and PMVS technologies, a high-density three-dimensional point cloud and digital orthophoto (DOM) can be generated, so that the data not only contains geometric information, but also contains rich color and texture information; RANSAC and bundle adjustment (BA) are used to optimize the three-dimensional model parameters to ensure the accuracy of the sparse point cloud.

[0006] In order to achieve the above technical objectives and the above technical effects, the present invention is implemented through the following technical solutions:

[0007] An ISfM-PMVS river sediment digital modeling method based on smartphone-collected images includes the following steps:

[0008] S1: Use a smartphone to capture images from multiple perspectives at a predetermined location;

[0009] S2: Extract feature points from each image and establish the geometric relationship between images by matching these feature points.

[0010] S3: The ISfM method is used to estimate camera parameters and sparse point cloud coordinates, and the initial 3D model is constructed through RANSAC and bundle adjustment optimization.

[0011] S4: Apply the PMVS algorithm and use multi-view consistency verification to generate a high-density 3D point cloud to ensure the accuracy and integrity of the model; transfer the color and detail information of the photo to the 3D model through texture mapping to form a digital orthophoto (DOM).

[0012] S5: Using the generated 3D model and 2D DOM, the geometric morphology of river sediments is identified and analyzed through a convolutional neural network.

[0013] S6: Verify the model accuracy by comparing the digital model with actual measurement data, and use Deep Learning methods to detect the characteristics and distribution of sediments.

[0014] Furthermore, the step S1 specifically includes:

[0015] Use smartphones such as HUAWEIP30 Pro for image acquisition. This phone is equipped with a 6mm focal length lens, which is suitable for high-resolution close-up shooting. Careful planning of the shooting plan is required to ensure sufficient image overlap, which is crucial for subsequent 3D reconstruction. The researchers determined the shooting angle, distance and movement direction on site. Specifically, a smartphone was used to shoot along a semi-circular path in the sediment area at a shooting distance of 3 meters. Starting from one side of the sediment area, move along the predetermined semicircular path and take a photo at a certain distance to ensure full coverage of the perspective and obtain information on different sides of the sediment. During the movement, keep the smartphone stable to ensure that there is a 60% to 80% overlap between adjacent photos for later feature matching and 3D reconstruction. At the same time, choose uniform light conditions for shooting to avoid strong shadows and light contrast from affecting image quality. A total of 81 photos with a resolution of 2736×3648 were obtained at the geological point. The content and shooting position of each photo were designed to obtain complete coverage and optimal image overlap to ensure the accuracy and completeness of subsequent processing.

[0016] Furthermore, step S2 specifically includes: extracting features from adjacent photos and matching feature points can determine the relative position and posture of the camera at different viewing angles, providing an accurate spatial coordinate basis for 3D reconstruction. At the same time, feature points with the same name between different images can also be determined, thereby establishing geometric constraints between images. Figure 2 Show, let m K-1 With m K is a pair of matching points with the same name in two adjacent images, and their coordinates are (X, Y, 1) and (X', Y', 1) respectively. The corresponding points with the same name m satisfy the geometric constraints:

[0017]

[0018] Where F is a 3×3 matrix with 7 degrees of freedom.

[0019] Assume that the camera parameters are C = {C1, C2, C3······C n}, the point cloud coordinates are X={X1,X2,X3······X n}, through perspective projection change, the point cloud coordinates are projected into the image plane coordinate system, and the projection error is assumed to be g(C,X).

[0020] Furthermore, the step S3 specifically includes: by extracting and matching feature points in the image, the ISfM algorithm searches for the camera and scene parameter optimization goals from the image to solve the image matching sparse point cloud. ISfM mainly estimates the parameter information of the camera C and the three-dimensional coordinates of the point cloud X by nonlinearly minimizing g(C,X):

[0021]

[0022] Where: n is the number of images taken by the camera, and m is the number of feature matching points. j On image i, then ω i,j =1, otherwise ω i,j =0,||q i,j -p(C i ,X j )|| 2 is the projection error of the point in the camera.

[0023] The Random Sample Consensus (RANSAC) algorithm and Bundle Adjustment (BA) are used to project the point cloud onto each image and find the corresponding points and pixel coordinates. After the BA iteration, a sparse point cloud of the whole scene is obtained.

[0024] Furthermore, the step S4 specifically includes:

[0025] The sparse point cloud uses the camera's posture information through PMVS to generate a high-density 3D point cloud through multi-view consistency verification. Based on the feature points in the sparse point cloud, patches are generated, and multi-view verification methods such as photometric and parallax consistency are used. Photometric consistency verifies the color consistency of the image block under different viewpoints, and parallax consistency controls the consistency of the geometric position to ensure the matching accuracy of these image blocks under different viewpoints. At the same time, the normal vector optimization is applied to update the normal information of each image block to improve the accuracy of depth estimation, especially in areas with complex textures or morphological mutations. In the final dense point cloud construction process, PMVS integrates the image blocks that meet the consistency requirements into 3D points, and the coordinates f of each point can be obtained by weighted average in the multi-view projection results through formula 6. Then the texture information is mapped to the point cloud to complete the digital construction of the 3D model, image dense matching (DIM) point cloud and digital orthophotomap (DOM).

[0026]

[0027] Where: N represents the number of viewpoints; Ri and ti represent the rotation matrix and translation vector of the i-th viewpoint, pi is the image projection point in the viewpoint, and K is the camera intrinsic parameter matrix.

[0028] Furthermore, the step S5 specifically includes: the constructed three-dimensional model and the two-dimensional DOM generated by it are used to interpret the geometric and morphological information of river sediments. With the help of Darknet-53 convolutional neural network, the model can accurately decompose and record the geometric features of geological points on the digital orthophoto.

[0029] Furthermore, step S6 specifically includes: by comparing and analyzing the results of the digital model measurement with the actual field measurement data, ensuring that the geometric accuracy of the model is within an acceptable range. Using the detection frame of the Darknet-53 network to identify the location of the sediment, and revising the perimeter of the sediment according to the detection results. This process also requires statistics on the distribution of sediments of different sizes, so as to provide data support for the analysis and geological interpretation of river sediments.

[0030] Beneficial effects of the present invention:

[0031] The present invention realizes low-cost and high-efficiency three-dimensional modeling by using a smart phone for image acquisition. Smart phones are common and easily accessible devices with high-definition cameras that can meet the needs of collecting high-resolution images. Combined with the incremental structure from motion (ISfM) algorithm, feature points can be gradually extracted and matched from the image to establish a sparse point cloud and estimate camera parameters, thereby generating an initial three-dimensional model. Compared with traditional professional equipment, such as three-dimensional laser scanners or drones, the equipment cost is high, the operating technology requirements are high, and the use and maintenance costs are high. In the field environment, the limitations of transportation and power support affect the extensiveness and sustainability of digitization, and in areas with complex or narrow terrain, the layout and operation of professional equipment are limited by environmental conditions, and the characteristic details of the sediments cannot be fully covered, resulting in partial loss of the digital model. This method does not require additional hardware investment, which reduces the cost of the project. At the same time, because smart phones are portable and easy to use, the flexibility and convenience of field data acquisition are greatly improved.

[0032] In terms of optimizing modeling accuracy, the present invention uses the random sampling consensus (RANSAC) algorithm and bundle adjustment (Bundle Adjustment, BA) for data optimization. RANSAC selects samples from the data set for model fitting by iteration, and removes outliers to improve the accuracy of point cloud matching. The bundle adjustment further optimizes the camera parameters and the three-dimensional coordinates of the sparse point cloud, improves the accuracy of the model by minimizing the projection error, and forms a basic geometric structure. Next, a multi-view stereo (PMVS) algorithm is used to generate a high-density point cloud through multi-view consistency verification, ensuring the complete presentation of details and the geometric accuracy of the model to the centimeter level. This accuracy can provide reliable basic data support in the morphological analysis of river sediments.

[0033] The present invention applies convolutional neural networks (CNN), such as Darknet-53, to the automatic interpretation of digital models. By generating digital orthophotos (DOM), the image texture information is accurately mapped onto the three-dimensional model, providing detailed data input for CNN. The design of multi-layer convolution and residual connection in the Darknet-53 network structure enhances the depth and breadth of feature extraction, so that it can show good generalization ability when detecting river sediment characteristics, and detect targets through feature maps of different scales, realizing automatic recognition and classification of sediment geometry. Experiments show that its recognition accuracy rate reaches 0.82, which not only improves the efficiency of geological information interpretation, but also reduces the subjective errors of traditional manual interpretation.

[0034] This technical solution provides a method for quickly acquiring and analyzing geological information, which is of great significance for the recording of geological relics and the protection of natural heritage. The portable nature of smartphones allows efficient data collection in complex terrain and harsh environments, and the generated data model can accurately reflect the actual conditions of geological points. The relative accuracy of the model is ensured through measurement and model comparison verification. Even in the absence of geographic coordinates, the geometric error of the model is controlled at the centimeter level. For heritage protection, it can not only assist management and monitoring, but also implement effective protection measures by dynamically tracking geological changes. This method can also be integrated with a variety of modern surveying and mapping technologies to form a multi-scale and multi-information level monitoring system, providing solid data support for the continuous research of geological and ecological environment.

[0035] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0037] Figure 1 It is a schematic diagram of the incremental calculation expression of images;

[0038] Figure 2 This is a schematic diagram of photo feature point matching;

[0039] Figure 3 Schematic diagram of the process of establishing a digital model for SfM;

[0040] Figure 4 This is a schematic diagram of the river sediment detection network structure;

[0041] Figure 5 Digital models and physical survey diagrams of geological points;

[0042] Figure 6 This is a schematic diagram of the river sediment profile;

[0043] Figure 7 Schematic diagram of training and validation loss;

[0044] Figure 8 This is a schematic diagram of river sediments; DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] Example 1

[0047] The ISfM-PMVS river sediment digital modeling method based on smartphone-collected images described in this embodiment includes the following steps:

[0048] S1: Use a smartphone to capture images from multiple perspectives at a predetermined location;

[0049] S2: Extract feature points from each image and establish the geometric relationship between images by matching these feature points.

[0050] S3: The SfM method is used to estimate camera parameters and sparse point cloud coordinates, and the initial 3D model is constructed through RANSAC and bundle adjustment optimization.

[0051] S4: Apply the PMVS algorithm and use multi-view consistency verification to generate a high-density 3D point cloud to ensure the accuracy and integrity of the model; transfer the color and detail information of the photo to the 3D model through texture mapping to form a digital orthophoto (DOM).

[0052] S5: Using the generated 3D model and 2D DOM, the geometric morphology of river sediments is identified and analyzed through a convolutional neural network.

[0053] S6: Verify the model accuracy by comparing the digital model with actual measurement data, and use Deep Learning methods to detect the characteristics and distribution of sediments.

[0054] Example 2

[0055] The study area of ​​this embodiment is the Zhongxing Town-Tiger Leaping Gorge section of the Jinsha River in Yulong Naxi Autonomous County, Lijiang City, and a study on the digital modeling and interpretation of river sediments by smartphones is carried out. The study area is located in the Zhongxing Town-Tiger Leaping Gorge section of the Jinsha River. The left bank belongs to Shangri-La City, Diqing Tibetan Autonomous Prefecture, and the right bank belongs to Yulong Naxi Autonomous County, Lijiang City. Since the Quaternary Period, the river geological action in this area has been very active due to factors such as neotectonic movement and climate warming, forming "V"-shaped river valleys, canyons, diversions, side beaches and heart beaches, river terraces and other landform types. According to the river valley geomorphology of the Zhongxing Town-Tiger Leaping Gorge section and the genetic type of deposits in the river valley, this river section can be divided into the Zhongxing Town-Longpan Township section and the Tiger Leaping Gorge section. The river valley of the Zhongxing Town-Longpan Township section is wide and the slope is gentle. The longitudinal gradient is about 0.67%, the side beaches and heart beaches are developed, and multi-level river terraces are generally developed on both sides. The impact objects in the river valley have a distinct binary structure. The alluvial gravels are well rounded and are commonly known as pebbles. They are various rocks in the Jinsha River and its tributaries that have been weathered and eroded, carried by flowing water over long distances, collided, rubbed, and rounded with each other before being deposited on the riverbed. The gravels have rich colors and complex compositions.

[0056] Smartphone Model Building

[0057] Smartphones are common electronic products today, which can provide data for visual systems at a low cost, effectively recording and protecting cultural heritage. In order to obtain a scene-retaining record of the actual scene without destroying the site, and to accurately measure and analyze different geological relics, this paper determined the theory of close-range photogrammetry technology methods on site, and tried to use mobile phone mobile shooting imaging methods. According to the actual conditions observed on site, a feasible and practical imaging perspective, shooting distance and mobile photography direction were designed, and the geological relics data collection with a certain degree of overlap was carried out and completed. Through HUAWEIP30 Pro, the lens focal length is 6mm, and a total of 81 photos taken by mobile phones from multiple perspectives were obtained at three geological points, with a resolution of 2736*3648 / pixel.

[0058] Incremental SfM iteratively fuses one or more images into the initial image pair model. In this process, the relevant algorithm is repeatedly used to calculate the three-dimensional information of the object and update the existing model until the image sequence is added. Figure 1 Show.

[0059] To build a digital model based on ISfM and smartphone images, it is necessary to first extract the feature points of the photos and match them. This step is the basis for building a digital model and is used to determine the feature points of the same name between different images, thereby establishing geometric constraints between images. Figure 2 Show, let m K-1 With m K is a pair of matching points with the same name in two adjacent images, and their coordinates are (X, Y, 1) and (X', Y', 1) respectively. The corresponding points with the same name m satisfy the geometric constraints:

[0060]

[0061] Where F is a 3×3 matrix with 7 degrees of freedom.

[0062] Assume that the camera parameters are C = {C1, C2, C3······C n}, the point cloud coordinates are X={X1,X2,X3······X n}, through perspective projection change, the point cloud coordinates are projected into the image plane coordinate system, and the projection error is assumed to be g(C,X).

[0063] The SfM algorithm

[25] is used to find the camera and scene parameter optimization targets from the image to solve the image matching sparse point cloud. SfM mainly estimates the parameter information of the camera C and the 3D coordinates of the point cloud X by nonlinearly minimizing g(C,X):

[0064]

[0065] Where: n is the number of images taken by the camera, and m is the number of feature matching points. j On image i, then ω i,j =1, otherwise ω i,j =0,||q i,j -p(C i ,X j )|| 2 is the projection error of the point in the camera.

[0066] At the same time, the Random Sample Consensus (RANSAC) algorithm and Bundle Adjustment (BA) are used to project the point cloud on each image and find the corresponding points and pixel coordinates. After the BA iteration, the sparse point cloud of the panorama is obtained.

[0067] Finally, the sparse point cloud uses the camera's posture information through PMVS to generate a high-density 3D point cloud through multi-view consistency verification. Based on the feature points in the sparse point cloud, patches are generated, and multi-view verification methods such as photometric and parallax consistency are used to ensure the matching accuracy of these image blocks under different viewpoints. The verified image blocks will continue to expand, and the accuracy of depth estimation will be improved through normal vector optimization. In the final dense point cloud construction process, PMVS integrates the image blocks that meet the consistency requirements into 3D points, and the coordinates f of each point can be obtained by weighted average in the multi-view projection results through formula 6. Then map the texture information to the point cloud to complete the digital construction of the 3D model, image dense matching (DIM) point cloud and digital orthophoto map (DOM).

[0068]

[0069] Where: N represents the number of viewpoints; Ri and ti represent the rotation matrix and translation vector of the i-th viewpoint, pi is the image projection point in the viewpoint, and K is the camera intrinsic parameter matrix.

[0070] Geological information interpretation using smartphone models

[0071] ISfM is a cost-effective 3D reconstruction method that not only reconstructs the 3D morphology of the scene, but also maps the color and texture of the image to the surface of the model. It also restores the real texture of the scene, providing a more intuitive and richer visual experience. In addition, this process also derives digital models of multiple dimensions, such as 3D DIM point clouds and 2D DOM. The digital models of different dimensions constructed by ISfM-PMVS, 3D real-life models and 2D DOM can effectively interpret stratigraphic profiles and morphological maps. Based on DOM, the intelligent network structure Darknet-53 is used to accurately decompose and record the geometric morphological information of river sediments at geological points.

[0072] DOM maps the color and texture information of the image to the three-dimensional model through orthorectification, eliminating the geometric errors caused by factors such as terrain undulation, camera tilt and lens distortion, and can effectively analyze the accumulation form, particle size distribution and morphological characteristics of river sediments. Darknet53 is a convolutional neural network model proposed by Joseph Redmon, which consists of 53 convolutional layers and 5 max-pooling layers. The network structure is as follows Figure 5 The idea of ​​pyramid feature map is used for reference. Small-sized feature maps are used to detect large-sized objects, while large-sized feature maps are used to detect small-sized objects. At the same time, multiple convolution and residual connection layers are stacked to improve the effect of feature extraction. A total of 3 feature maps are output. The first feature map is downsampled 32 times, the second feature map is downsampled 16 times, and the third is downsampled 8 times. The concat operation is to splice feature maps according to the channel dimension. For example, two 8*8*16 feature maps are spliced ​​to generate a 8*8*32 feature map.

[0073] The Darknet53 network training river sediments for DOM identification of geological points are used to calculate the accuracy according to Formula 7, that is, the proportion of the results detected as positive samples by the model that are actually positive samples.

[0074]

[0075] Where: TP means that the prediction result matches the actual object, and FP means that the prediction result does not match the actual object, but the algorithm identifies it as the target object.

[0076] Accuracy verification of geological point model established by smartphone

[0077] The high-definition photos taken by smartphones are used to construct the 3D model of geological points through ISfM-PMVS. Figure 6The model has a high degree of texture restoration, clear surface texture and realistic colors, and can delicately show the texture and light and shadow changes of geological relics. Since mobile phone photos do not have geographic coordinates, this paper compares the distances of different entities in the digital model and the distances measured by a tape measure to make an error comparison analysis to ensure that the model can meet the requirements in actual applications.

[0078] In the absence of geographic coordinates, the accuracy verification of the model mainly relies on the assessment of relative accuracy and the check of geometric consistency. The length and width of chopsticks and scale are measured with a tape measure and the difference analysis is performed with the digital model measurement values. The results are shown in Table 1. The maximum error is 0.11 cm and the minimum error is 0.01 cm. The centimeter-level geometric error can reflect the effectiveness of the digital model constructed this time to a certain extent.

[0079] Table 1 Comparative analysis of digital model accuracy of geological points

[0080]

[0081] Smartphones build models to interpret geological information

[0082] River terraces on the right bank of the Jinsha River in Shaba Village Near Shaba Village downstream of Shigu Town, there are 4 levels of river terraces on the right bank of the Jinsha River. River sediments are also called alluvial deposits. When the longitudinal slope ratio of the riverbed decreases and the width becomes wider, the riverbed becomes flat, the flow rate slows down, and the river's carrying capacity decreases, sedimentation occurs. The sediments formed have a distinct binary structure, that is, the upper part is floodplain sediments, and the lower part is riverbed sediments. The cross-section of river sediments on the right bank of the Jinsha River on the north slope of Shaba Village is shown in the figure below. Figure 6 It can be found that the upper part is composed of silt and mud with a thickness of about 50 to 60 cm, which belongs to the floodplain phase sediments; the lower part is composed of riverbed phase sand and gravel (golden sandstone) with a thickness of more than 2.5 m. The gravel composition is complex (all three major types of rocks are present, which should be consistent with the bedrock lithology of its upstream basin), the gravels are of different sizes, highly rounded, well sorted, and have a certain orientation and are arranged in an imbricate manner.

[0083] Based on the calculation of the processor being a 12th Gen Intel Core i7-12700H and the graphics card being an NVIDIA GeForce RTX3070Ti Laptop GPU, the Darknet-53 network is called, where the epoch, batch size, and chip_size are 200, 32, and 224 respectively. Validation is performed with 20% of the data, and the results are as follows Figure 7The blue curve represents the change in loss of the training set, and the orange curve represents the change in loss of the validation set. From the change in the loss function, it can be seen that the model shows good convergence on both the training and validation sets. The loss function continues to decrease and tends to be stable. The similarity between the validation loss and the training loss shows that there is no overfitting phenomenon in the model. The accuracy of the model in detecting sediments on the validation set is 0.82, indicating that the Darknet-53 network has good generalization ability after the loss function converges.

[0084] The Darknet-53 network detection box is a rectangle, and the perimeter needs to be revised at the location where the sediment is identified. A total of 397 sediments were detected. The results are as follows Figure 8 The statistical results are shown in Table 2. Small shapes with Circumference<50 are distributed densely on the left and middle of the figure, indicating that the circumference of the shapes in these areas is relatively small. 50≤Circumference<100 is mainly concentrated in the middle of the figure, with a relatively uniform and dense distribution. It belongs to the shape of small sediments, but accounts for the largest proportion, reaching 50.60% of the area. 100≤Circumference<150 is mainly distributed in the middle left and upper right corners of the figure, indicating that there are more medium-sized sediments in these areas. 150≤Circumference<200 is relatively small in number, accounting for 6.05%, and is more dispersed, mainly concentrated on the right and top of the figure. 200≤Circumference<300 is a larger sediment, distributed in the middle and right of the entire figure, accounting for the smallest proportion, accounting for 3.53% of the entire area.

[0085] Table 2 Statistics of river sediment perimeter

[0086]

[0087]

[0088] There are missing holes in the point cloud in the 3D model built using a smartphone, indicating that in field data collection, the viewing angle, direction, and distance of mobile phone shooting need to be adjusted accordingly according to the actual conditions of the scene targets, so as to improve the missing matching feature point cloud in the SFM-MVS modeling process. Of course, the modeling holes of the tiny gaps between the inlaid rocks are insurmountable and unavoidable due to the mobile phone shooting itself, which requires necessary supplementary measurements during field surveys.

[0089] The processing method based on mobile phone imaging modeling technology used in this embodiment shows the advantages of convenient and fast geological relics recording, but it has its limitations for the observation and modeling of larger scenes. For this, it is possible to consider the application of comprehensive air-ground technology and try to produce results supported by air-ground observation methods to show the multi-granularity characteristics of scene modeling, that is, the multi-scale characteristics of spatial resolution. This can be used to describe the overall and local composition of the site on the landscape scale for the degree of detail of targets such as outcrops and rock masses. In addition, the results produced under the technical support of air-ground methods are not only a description of its spatial position and the overall state of its environment, but also, in terms of time sequence and function, are more conducive to the dynamic tracking description and scientific management of the quality status (change and protection) of rock fossil relics and ruins. In general, the geographical and geological ecological environment where geological relics are located is extremely fragile, and future protection is very challenging and needs to be strengthened.

[0090] In summary, the present invention proposes an ISfM-PMVS method for collecting images with a smartphone to establish a digital model of river sediments on the river terraces on the right bank of the Jinsha River in Shaba Village. The length and width of chopsticks and scales are measured by a tape measure and the difference test is performed with the digital model measurement values. It is found that the geometric error of the object is only at the centimeter level, which can reflect the effectiveness of building a digital model. Different river sediment information is interpreted by digital models of different dimensions. The results show that: based on the analysis of the three-dimensional real-life digital model, it is found that the river sediments in the test area are composed of silt and mud with a thickness of about 50 to 60 cm, which belongs to the floodplain phase sediments; the lower part is composed of riverbed gravel (golden sand stone) with a thickness of more than 2.5 m; based on DOM combined with the Darknet-53 network, the river sediments are detected with an accuracy of 0.82, and a total of 397 sediments are detected. The sediment perimeters of 50≤Circumference<100 are mainly concentrated in the middle of the map, with a relatively uniform and dense distribution, belonging to small sediments, accounting for the largest proportion, reaching 50.60%, and 200≤Circumference<300 are large sediments, accounting for the smallest proportion, accounting for 3.53% of the entire area. Although there will be some distortion in the edge areas of the digital model established by smartphones, this method can not only quickly and comprehensively obtain the digital model of the geological relic site, but also realize the intelligent interpretation of geological information, which is of great significance to the protection of geological open-air and geological heritage.

[0091] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. An ISfM-PMVS river sediment digital modeling method based on smartphone-collected images, characterized in that: The following steps are involved: S1: Use a smartphone to capture images from multiple perspectives at a predetermined location; S2: Extract feature points from each image and establish the geometric relationship between images by matching these feature points; S3: The SfM method is used to estimate camera parameters and sparse point cloud coordinates, and the initial 3D model is constructed through RANSAC and bundle adjustment optimization; S4: Apply PMVS algorithm and use multi-view consistency verification to generate high-density 3D point cloud to ensure the accuracy and integrity of the model; transfer the color and detail information of the photo to the 3D model through texture mapping to form a digital orthophoto (DOM); S5: Using the generated 3D model and 2D DOM, the geometric morphology of river sediments is identified and analyzed through convolutional neural networks; S6: Verify the model accuracy by comparing the digital model with actual measurement data, and use Deep Learning methods to detect the characteristics and distribution of sediments.

2. The ISfM-PMVS river sediment digital modeling method based on smartphone image collection as claimed in claim 1, characterized in that: The step S1 specifically includes: Use a smartphone to collect images; determine the viewing angle, distance, and movement direction of the shooting, and obtain a total of 81 photos with a resolution of 2736x3648 at three different geological points; to obtain complete coverage and optimal image overlap.

3. The ISfM-PMVS river sediment digital modeling method based on smartphone image collection as claimed in claim 1, characterized in that: The step S2 specifically includes: extracting feature points of the photo and matching each feature point to determine the feature points of the same name between different images, thereby establishing geometric constraints between the images; assuming m K-1 With m K is a pair of matching points with the same name in two adjacent images, and their coordinates are (X, Y, 1) and (X', Y', 1) respectively. The corresponding points with the same name m satisfy the geometric constraints: Where F is a 3×3 matrix with 7 degrees of freedom; Set the camera parameters to C = {C1, C2, C3...C n }, the point cloud coordinates are X={X1,X2,X3······X n }, through perspective projection change, the point cloud coordinates are projected into the image plane coordinate system, and the projection error is assumed to be g(C,X).

4. The ISfM-PMVS river sediment digital modeling method based on smartphone image collection as claimed in claim 1, characterized in that: The step S3 specifically includes: by extracting and matching feature points in the image, the SfM algorithm searches for the camera and scene parameter optimization target from the image to solve the image matching sparse point cloud, and the SfM estimates the parameter information of the camera C and the three-dimensional coordinates of the point cloud X by nonlinearly minimizing g(C,X): Where: n is the number of images taken by the camera, m is the number of feature matching points; if X j On image i, then ω i,j =1, otherwise ω i,j =0,||q i,j -p(C i ,X j )|| 2 is the projection error of the point in the camera; The random sampling consensus algorithm and bundle adjustment are used to project the point cloud onto each image and find the corresponding points and pixel coordinates. After the BA iteration, the sparse point cloud of the whole view is obtained.

5. The ISfM-PMVS river sediment digital modeling method based on smartphone image collection as claimed in claim 1, characterized in that: The step S4 specifically includes: the sparse point cloud uses the camera's posture information through PMVS to generate a high-density three-dimensional point cloud through multi-view consistency verification; based on the feature points in the sparse point cloud, a patch is generated, and a multi-view verification method of photometric and parallax consistency is adopted, wherein photometric consistency verifies the color consistency of the image block under different view angles, and parallax consistency controls the consistency of the geometric position to ensure the matching accuracy of these image blocks under different view angles; at the same time, the normal vector optimization is applied to update the normal information of each image block to improve the accuracy of depth estimation, especially in areas with complex textures or morphological mutations; in the final dense point cloud construction process, PMVS integrates the image blocks that meet the consistency requirements into three-dimensional points, and the coordinates f of each point are obtained by weighted average in the multi-view projection results through formula 6; then the texture information is mapped to the point cloud to complete the digital construction of the three-dimensional model, image dense matching point cloud and digital orthophoto; Where: N represents the number of viewpoints; Ri and ti represent the rotation matrix and translation vector of the i-th viewpoint, pi is the image projection point in the viewpoint, and K is the camera intrinsic parameter matrix.

6. The ISfM-PMVS river sediment digital modeling method based on smartphone image collection as claimed in claim 1, characterized in that: The step S5 specifically includes: the constructed three-dimensional model and the two-dimensional DOM generated by it are used to interpret the geometric and morphological information of river sediments; with the help of the Darknet-53 convolutional neural network, the model can accurately decompose and record the geometric features of geological points on the digital orthophoto.

7. The ISfM-PMVS river sediment digital modeling method based on smartphone image collection as claimed in claim 1, characterized in that: The step S6 specifically includes: ensuring that the geometric accuracy of the model is within an acceptable range by comparing and analyzing the results of digital model measurement with the actual on-site measurement data; using the detection frame of the Darknet-53 network to identify the location of the sediment and revising the perimeter of the sediment based on the detection results; this process also requires statistics on the distribution of sediments of different sizes, so as to provide data support for the analysis of river sediments and geological interpretation.

Images