Multi-view three-dimensional reconstruction method based on Deeplab semantic segmentation

Abstract

The invention discloses a multi-view three-dimensional reconstruction method based on Deeplab semantic segmentation. The method comprises the following steps: acquiring a multi-view image sequence of a target scene; performing pixel-level semantic annotation on each view image by using the trained Deeplab semantic segmentation model to obtain a semantic segmentation map containing object category information; based on a semantic segmentation result, carrying out semantic constraint feature matching on regions belonging to the same semantic category in different views; performing optimization estimation on the camera attitude in combination with a semantic constraint feature matching result; differentiated deep optimization strategies are adopted for the regions of different semantic categories; and generating dense point cloud data of the target scene based on the optimized depth information, and finally outputting a high-precision three-dimensional reconstruction model. According to the method, class information of Deeplab semantic segmentation is introduced, so that the robustness of multi-view feature matching is enhanced, and the reconstruction precision of object boundaries and weak texture regions in a complex scene is effectively improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and relates to 3D reconstruction technology, specifically a multi-view method based on Deeplab semantic segmentation. Figure Three Dimensional reconstruction method. Background Technology

[0002] Multiview Figure Three 3D reconstruction technology restores the 3D structure of a scene by performing feature matching, camera pose estimation, and point cloud construction on images from different perspectives, and is widely used in fields such as autonomous driving, cultural relic digitization, and virtual reality. However, existing methods have significant shortcomings when dealing with complex scenes: blurred object boundaries (such as adjacent areas of different types of objects) and regions with similar textures (such as walls and ground with repeated textures) can easily lead to matching ambiguities, resulting in a high feature mismatch rate. This, in turn, causes camera pose estimation errors and uneven point cloud density, ultimately resulting in low accuracy and lack of detail in the reconstructed model, making it difficult to meet the requirements of high-precision scene modeling.

[0003] Semantic segmentation technology can classify image pixels, providing pixel-level semantic priors for multi-view reconstruction. Deeplab models, leveraging techniques such as dilated convolution and spatial pyramid pooling, excel in semantic segmentation tasks, accurately outputting pixel category information. However, existing multi-view reconstruction methods have not fully utilized the category constraint advantages of Deeplab semantic segmentation and lack a design that deeply integrates semantic information with the reconstruction process. Summary of the Invention

[0004] Purpose of the invention: To overcome multiple perspectives Figure Three To address the technical shortcomings of 3D reconstruction, such as low reconstruction accuracy and missing model details caused by blurred object boundaries and ambiguous matching of similar texture regions in the scene, this paper proposes a multi-view reconstruction method based on Deeplab semantic segmentation. Figure Three The 3D reconstruction method enhances the robustness of multi-view feature matching by introducing category information from Deeplab semantic segmentation, effectively improving the reconstruction accuracy of object boundaries and weakly textured regions in complex scenes.

[0005] Technical solution: To achieve the above objectives, this invention provides a multi-view solution based on Deeplab semantic segmentation. Figure Three The 3D reconstruction method includes the following steps:

[0006] S1: Obtain a multi-view image sequence of the target scene;

[0007] S2: Use the trained Deeplab semantic segmentation model to perform pixel-level semantic annotation on each view image to obtain a semantic segmentation map containing object category information;

[0008] S3: Based on the semantic segmentation results, perform semantic constraint feature matching on regions belonging to the same semantic category in different views;

[0009] S4: Combine the semantic constraint feature matching results to optimize and estimate the camera pose;

[0010] S5: Based on the optimized camera pose, a differentiated deep optimization strategy is adopted for regions with different semantic categories;

[0011] S6: Based on the optimized depth information, generate dense point cloud data of the target scene, and finally output a high-precision 3D reconstruction model.

[0012] Furthermore, in step S2, the Deeplab semantic segmentation model uses HRNet (High Resolution Network) as the backbone network to improve the segmentation accuracy of object edges. The model output formula is:

[0013] S(i,j)=argmax c∈{1,2,…,C} P(c|x i,j )

[0014] Where: x i,j P(c)x represents the feature (or original pixel value) of the input image at pixel (i,j); i,j ) represents the probability that the pixel belongs to category c; S(i,j) is the category label of pixel (i,j) in the semantic segmentation map (i.e. the final output segmentation result).

[0015] Furthermore, in step S3, for regions belonging to the same semantic category in different views, the SIFT (Size Invariant Feature Transform) algorithm is used to extract and match feature points. The specific process includes:

[0016] A1: Scale Space Construction

[0017] Scale Space Fundamentals:

[0018] The scale space function L(x,y,σ) is obtained by convolving the original image I(x,y) with a Gaussian function G(x,y,σ) of variable scale:

[0019] L(x,y,σ)=G(x,y,σ)*I(x,y)

[0020] Among them, Gaussian function * indicates a convolution operation, and L(x,y,σ) is the image representation at scale σ;

[0021] Difference of Gaussian Pyramid (DOG):

[0022] Obtained by subtracting Gaussian images from adjacent scales, used to detect scale-space extrema:

[0023] D(x,y,σ)=(G(x,y,kσ)-G(x,y,σ))*I(x,y)

[0024] Where: k is a constant;

[0025] A2: Feature Point Detection

[0026] Scale-space extreme point detection:

[0027] In the DOG pyramid, each pixel needs to be compared with its 8 neighborhood pixels at its own scale and 18 pixels at its adjacent scales. If it is a local extremum, it is considered a candidate feature point.

[0028] D(x,y,σ)>max{D(26 neighborhood pixels)} or D(x,y,σ)>max{D(26 neighborhood pixels)}

[0029] Edge response filtering:

[0030] Calculate the trace and determinant of the Hessian matrix, and filter by curvature ratio:

[0031] Hessian matrix:

[0032] Trace Tr(H) = D xx +D yy Determinant det(H) = D xx D yy -(D xy ) 2 curvature ratio When r < (r+1) 2 When / r, feature points are preserved;

[0033] A3: Direction Assignment

[0034] Gradient calculation:

[0035]

[0036] Where m(x,y) is the gradient magnitude and θ(x,y) is the gradient direction;

[0037] A4: Feature Descriptor

[0038] Regional division:

[0039] Centered on the feature point, a 16×16 area is selected and divided into 4×4 sub-regions;

[0040] Gradient direction:

[0041] The gradient values ​​in 8 directions are counted for each sub-region, resulting in a 4×4×8=128-dimensional vector;

[0042] A5: Feature Matching

[0043] The matching degree between two feature points is measured by the Euclidean distance between their 128-dimensional descriptors:

[0044]

[0045] in: These are the descriptors for feature points A and B, respectively.

[0046] Furthermore, in step S3, a feature matching penalty mechanism is set to reduce the matching weight of non-homologous regions (regions of different semantic categories). The feature matching penalty mechanism includes:

[0047] Let C p C represents the semantic category to which p belongs. q If q belongs to a specific semantic category, then the semantic consistency factor is defined as follows:

[0048]

[0049] Where α∈(0,1) is the penalty coefficient.

[0050] Furthermore, in step S3, feature point matching and motion estimation are preferentially performed within the same semantic region to reduce cross-class mismatches and improve the accuracy and robustness of feature matching.

[0051] Motion estimation: High-quality matching point pairs are selected through "semantic consistency", and robust algorithms such as RANSAC are used to solve the set transformation relationship between views, and finally the camera motion parameters are obtained by decomposition.

[0052] Motion estimation can reduce cross-class mismatches and improve the accuracy and robustness of feature matching.

[0053] Furthermore, the optimization estimation of camera pose in step S4 includes: incorporating the semantically matched feature point pairs as constraints into the bundle adjustment model to correct the camera pose estimation error caused by feature mismatch and obtain more accurate camera pose information.

[0054] The objective function of the bundle adjustment model is:

[0055]

[0056] Where: M is the number of camera views, and N is the number of 3D points; (R i ,t i X represents the pose of the i-th camera (rotation matrix and translation vector, i.e., extrinsic parameters); j =(x j ,y j ,zj ) T Let x be the world coordinates of the j-th 3D point; ij =(u ij ,v ij ) T For a 3D point X j The observed pixel coordinates on the i-th image; π(R i ,t i ,X j ) is the projection function, which projects the three-dimensional point X. j The intrinsic and extrinsic parameters of the camera are projected onto the i-th image; ω ij This is the weighting factor.

[0057] The process of integrating bundle adjustment models includes

[0058] B1: Iterative Optimization

[0059] The solution to semantic constraints (BA) is achieved through iterative optimization using the Levenberg-Marquardt (LM) algorithm, which specifically includes:

[0060] B1-1: Initialization:

[0061] Based on the results of motion estimation and triangulation of triangulation points As initial value;

[0062] B1-2: Constructing the Jacobian matrix:

[0063] For each camera pose parameter (R) i ,t i ) and three-dimensional point (X) j ), calculate the Jacobian matrix J;

[0064] B1-3: Solving the incremental equations:

[0065] In each iteration, the parameter increment Δ is calculated using the LM algorithm, with the following formula:

[0066] (J T WJ+λI)Δ=J T We

[0067] Where W is the diagonal weight matrix, e is the reprojection error vector, λ is the LM damping factor, and I is the identity matrix;

[0068] B1-4: Update parameters:

[0069] If the increment reduces the objective function value, then update the camera pose (R). i =R i +ΔR i ,t i =ti +Δt i ) and three-dimensional point (X) j =X j +ΔX j );

[0070] B1-5: Convergence Judgment

[0071] The iteration stops when the number of iterations reaches a threshold or the parameter increment is less than the threshold, and the optimized camera pose is obtained.

[0072] B2: Camera pose correction, specifically including:

[0073] B2-1: Semantic Consistency Guided Correction:

[0074] If the initial camera pose is deviated due to mismatch, the semantic weight will amplify the reprojection error of the correct matching point, forcing the optimization process to adjust the camera pose to reduce these errors, thereby correcting the pose deviation.

[0075] B2-2: Low weighting of cross-category matching points achieves out-of-point suppression;

[0076] B2-3: Global Consistency Optimization

[0077] The BA algorithm is used to simultaneously optimize the poses of all cameras and 3D points. Semantic constraints ensure that points of the same semantic type are projected consistently in multiple views, ultimately enabling the camera pose to satisfy global geometric constraints.

[0078] Furthermore, the differentiated deep optimization strategy for regions of different semantic categories in step S5 includes:

[0079] For rigid semantic category regions, depth is optimized using geometric constraints, specifically including:

[0080] Let the pixel coordinates of a 3D point X in view i be x. i =(u i ,v i ), with a depth of z i Through camera extrinsic parameters (R k ,t k Projecting the predicted pixels onto view k yields the predicted pixels.

[0081]

[0082] Where: K i π is the camera intrinsic parameter, and π is the projection function;

[0083] The optimization objective is to minimize the difference between the predicted pixel and the actual observed pixel x. k Deviation:

[0084]

[0085] For flexible semantic category regions, depth estimation is performed by combining texture continuity and semantic consistency constraints, specifically including:

[0086] Let the depth of pixel (u,v) be z(u,v), and its image grayscale gradient be... Then depth gradient Negatively correlated with image gradient:

[0087]

[0088] Among them, I u and I v These represent the gradients in the horizontal and vertical directions, respectively; σ is an adjustment parameter used to control the smoothing intensity.

[0089] Beneficial effects: Compared with existing technologies, this invention introduces pixel-level semantic category information to constrain feature matching, camera pose estimation, and depth optimization processes, thereby improving the accuracy and detail integrity of 3D reconstruction in complex scenes (including areas with blurred object boundaries and weak textures). Specifically, it includes the following advantages:

[0090] (1) Introducing pixel-level category constraints of Deeplab semantic segmentation restricts feature matching across semantic category regions, significantly reducing the feature mismatch rate in complex scenes (such as object boundaries and repeated texture regions) and improving feature matching accuracy.

[0091] (2) Differentiated depth optimization strategies are adopted for different semantic category regions to adapt to the reconstruction needs of regions with different characteristics such as rigidity and flexibility, effectively improving the reconstruction details of weak texture regions and complex boundary regions, making the 3D model more in line with the real scene.

[0092] (3) The present invention only requires a simple monocular camera or a regular mobile phone on the market to take pictures and acquire multi-view image sequences, which can greatly reduce the investment of funds. Attached Figure Description

[0093] Figure 1 This is an overall flowchart of the method of the present invention;

[0094] Figure 2 This is a multi-view image sequence according to an embodiment of the method of the present invention;

[0095] Figure 3 This is a schematic diagram of semantic segmentation using the method of the present invention;

[0096] Figure 4 A schematic diagram for camera pose estimation;

[0097] Figure 5 This is a depth image of a multi-view image sequence according to an embodiment of the present invention;

[0098] Figure 6 This is a schematic diagram of the point cloud data generated by the present invention. Detailed Implementation

[0099] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0100] Example 1:

[0101] like Figure 1 As shown, this embodiment provides a multi-view approach based on Deeplab semantic segmentation. Figure Three The 3D reconstruction method includes the following steps:

[0102] S1: Obtain a multi-view image sequence of the target scene;

[0103] This embodiment utilizes a monocular camera to acquire multi-view image sequences of the target scene, ensuring sufficient overlap between different views to provide basic data for subsequent feature matching and reconstruction.

[0104] S2: Use the trained Deeplab semantic segmentation model to perform pixel-level semantic annotation on each view image to obtain a semantic segmentation map containing object category information;

[0105] This embodiment uses a pre-trained Deeplab semantic segmentation model (Deeplab v3+) to perform pixel-level semantic annotation on each view image obtained in step S1;

[0106] S3: Based on the semantic segmentation results, perform semantic constraint feature matching on regions belonging to the same semantic category in different views;

[0107] S4: Combine the semantic constraint feature matching results to optimize and estimate the camera pose;

[0108] S5: Based on the optimized camera pose, a differentiated deep optimization strategy is adopted for regions with different semantic categories;

[0109] S6: Based on the optimized depth information, generate dense point cloud data of the target scene, and finally output a high-precision 3D reconstruction model.

[0110] In step S2, the Deeplab semantic segmentation model uses HRNet (high resolution network) as the backbone network to improve the segmentation accuracy of object edges. The original model used ResNet as the backbone network. In this embodiment, the Deeplab semantic segmentation model has been improved.

[0111] After the input image is processed by the model, the output is a semantic segmentation map containing object categories, providing a basis for subsequent semantic constraints; the output formula is:

[0112] S(i,j)=argmax c∈{1,2,…,C} P(c|x i,j )

[0113] Where: x i,j P(c)x represents the feature (or original pixel value) of the input image at pixel (i,j); i,j ) represents the probability that the pixel belongs to category c; S(i,j) is the category label of pixel (i,j) in the semantic segmentation map (i.e. the final output segmentation result).

[0114] In step S3, for regions belonging to the same semantic category in different views, the SIFT (Size Invariant Feature Transform) algorithm is used to extract and match feature points, as detailed below:

[0115] The SIFT algorithm consists of four core steps: scale-space extremum detection, keypoint localization, orientation assignment, and keypoint descriptor generation. The specific process is as follows:

[0116] 1) Scale Space Construction

[0117] Scale Space Fundamentals:

[0118] The scale space function L(x,y,σ) is obtained by convolving the original image I(x,y) with a Gaussian function G(x,y,σ) of variable scale:

[0119] L(x,y,σ)=G(x,y,σ)*I(x,y)

[0120] Among them, Gaussian function * indicates a convolution operation, and L(x,y,σ) is the image representation at scale σ.

[0121] Difference of Gaussian Pyramid (DOG):

[0122] Obtained by subtracting Gaussian images from adjacent scales, used to detect scale-space extrema:

[0123] D(x,y,σ)=(G(x,y,kσ)-G(x,y,σ))*I(x,y)

[0124] Where: k is a constant (the scaling factor between adjacent scales).

[0125] 2) Feature point detection

[0126] Scale-space extreme point detection:

[0127] In the DOG pyramid, each pixel needs to be compared with its 8 neighborhood pixels at its own scale and 18 pixels at its immediate and adjacent scales (a total of 26 pixels). If it is a local extremum, it is considered a candidate feature point.

[0128] D(x,y,σ)>max{D(26 neighborhood pixels)} or D(x,y,σ)>max{D(26 neighborhood pixels)}

[0129] Edge response filtering (Hessian matrix):

[0130] To eliminate unstable points on the edges, the trace and determinant of the Hessian matrix are calculated, and the points are filtered by curvature ratio:

[0131] Hessian matrix:

[0132] Trace Tr(H) = D xx +D yy Determinant det(H) = D xx D yy -(D xy ) 2 curvature ratio When r < (r+1) 2 When / r, feature points are preserved.

[0133] 3) Direction assignment

[0134] To ensure that feature points have rotation invariance, the principal direction is determined by the gradient direction;

[0135] Gradient calculation (gradient magnitude and direction):

[0136]

[0137] Where m(x,y) is the gradient magnitude and θ(x,y) is the gradient direction.

[0138] 4) Feature descriptor (128-dimensional vector)

[0139] Regional division:

[0140] Centered on the feature point, a 16×16 area is taken and divided into 4×4 sub-regions (each sub-region is 4×4 pixels).

[0141] Gradient direction:

[0142] The gradient values ​​in 8 directions are counted for each sub-region, resulting in a 4×4×8=128-dimensional vector.

[0143] 5) Feature matching (Euclidean distance)

[0144] The matching degree between two feature points is measured by the Euclidean distance between their 128-dimensional descriptors:

[0145]

[0146] in: These are the descriptors for feature points A and B, respectively.

[0147] In step S3, a feature matching penalty mechanism is set for cross-semantic category regions to reduce the matching weight of non-homologous regions (regions of different semantic categories).

[0148] Feature matching penalty mechanisms include:

[0149] Let C p C represents the semantic category to which p belongs. q If q belongs to a specific semantic category, then the semantic consistency factor is defined as follows:

[0150]

[0151] Where: α∈(0,1) is the penalty coefficient (e.g., α=0.3 means that the matching weight of non-homologous regions is reduced by 70%), the smaller the value, the stronger the penalty.

[0152] In step S3, feature point matching and motion estimation are performed preferentially within the same semantic region to reduce cross-class mismatches and improve the accuracy and robustness of feature matching.

[0153] Motion estimation: High-quality matching point pairs are selected through "semantic consistency", and robust algorithms such as RANSAC are used to solve the set transformation relationship between views, and finally the camera motion parameters are obtained by decomposition.

[0154] Motion estimation can reduce cross-class mismatches and improve the accuracy and robustness of feature matching.

[0155] Step S4 involves optimizing the camera pose estimation by incorporating semantically matched feature point pairs as constraints into the bundle adjustment model to correct camera pose estimation errors caused by feature mismatches and obtain more accurate camera pose information.

[0156] The essence of bundle adjustment is to jointly optimize the camera pose and the coordinates of 3D points by minimizing the reprojection of 3D points onto the image. The objective function of the bundle adjustment model is:

[0157]

[0158] Where: M is the number of camera views, and N is the number of 3D points; (R i ,t i X represents the pose of the i-th camera (rotation matrix and translation vector, i.e., extrinsic parameters); j =(xj ,y j ,z j ) T Let x be the world coordinates of the j-th 3D point; ij =(u ij ,v ij ) T For a 3D point X j The observed pixel coordinates on the i-th image; π(R i ,t i ,X j ) is the projection function, which projects the three-dimensional point X. j The intrinsic and extrinsic parameters of the camera are projected onto the i-th image; ω ij This is the weighting factor.

[0159] The process of incorporating the bundle adjustment model in step S4 includes:

[0160] 1. Iterative optimization

[0161] The semantic constraint BA is solved through iterative optimization using the Levenberg-Marquardt (LM) algorithm.

[0162] 1) Initialization:

[0163] Based on the results of motion estimation and triangulation of triangulation points As initial values.

[0164] 2) Construct the Jacobian matrix:

[0165] For each camera pose parameter (R) i ,t i ) and three-dimensional point (X) j ), calculate the Jacobian matrix J.

[0166] 3) Solve the incremental equation:

[0167] In each iteration, the parameter increment Δ is calculated using the LM algorithm, with the following formula:

[0168] (J T WJ+λI)Δ=J T We

[0169] Where W is the diagonal weight matrix, e is the reprojection error vector, λ is the LM damping factor, and I is the identity matrix.

[0170] 4) Update parameters:

[0171] If the increment reduces the objective function value, then update the camera pose (R). i =R i +ΔR i ,ti =t i +Δt i ) and three-dimensional point (X) j =X j +ΔX j ).

[0172] 5) Convergence judgment:

[0173] The iteration stops when the number of iterations reaches a threshold or the parameter increment is less than the threshold, and the optimized camera pose is obtained.

[0174] 2. Camera attitude correction

[0175] 1) Semantic consistency-guided correction:

[0176] If the initial camera pose is deviated due to mismatch, the semantic weight will amplify the reprojection error of the correct matching point, forcing the optimization process to adjust the camera pose to reduce these errors, thereby correcting the pose deviation.

[0177] 2) Out-of-point inhibition:

[0178] The low weight of cross-class matching points makes their influence in the Jacobian matrix weak. Even if their reprojection error is large, they will not dominate the optimization direction, thus avoiding the pose being "biased" by mismatches.

[0179] 3) Global consistency optimization:

[0180] The BA algorithm is used to simultaneously optimize the poses of all cameras and 3D points. Semantic constraints ensure that points of the same semantic type are projected consistently in multiple views, ultimately enabling the camera pose to satisfy global geometric constraints.

[0181] Step S5 employs differentiated deep optimization strategies for regions of different semantic categories, including:

[0182] For regions with rigid semantic categories (such as buildings and vehicles), depth is optimized using geometric constraints (plane fitting, multi-view geometric constraints, etc.).

[0183] Implementation method:

[0184] The projection of 3D points in a rigid region onto different views should satisfy camera pose constraints, and depth consistency should be enhanced using optimized camera extrinsic parameters.

[0185] Mathematical model:

[0186] Let the pixel coordinates of a 3D point X in view i be x. i =(u i ,v i ), with a depth of z i (Using view i as the reference frame), through the camera extrinsic parameters (R) k,t k The predicted pixels can be projected onto view k.

[0187]

[0188] Where: K i π is the camera intrinsic parameter, and π is the projection function.

[0189] The optimization objective is to minimize the difference between the predicted pixel and the actual observed pixel x. k Deviation:

[0190]

[0191] For flexible semantic category regions (such as vegetation and fabric), depth estimation is constrained by combining texture continuity and semantic consistency.

[0192] Implementation method:

[0193] In flexible regions, the depth of adjacent pixels with similar textures should not change abruptly; depth smoothing should be guided by image texture.

[0194] Mathematical model:

[0195] Let the depth of pixel (u,v) be z(u,v), and its image grayscale gradient be... Then depth gradient Negatively correlated with image gradient:

[0196]

[0197] Among them, I u and I v These represent the gradients in the horizontal and vertical directions, respectively; σ is an adjustment parameter used to control the smoothing intensity.

[0198] In step S6, dense point cloud data of the target scene is generated by depth map fusion.

[0199] Example 2:

[0200] In this embodiment, the method of the present invention is applied to the three-dimensional reconstruction of an electric vehicle. The specific process is as follows:

[0201] (1)Reference Figure 2 A monocular camera was used to acquire a sequence of 58 multi-view images of an outdoor scene. The image resolution was set to 960×544, and the overlap rate of adjacent views was ≥80%.

[0202] (2) A pre-trained Deeplab v3+ model (backbone: Xception) was selected. Based on the Cityscapes dataset, semantic segmentation was performed on outdoor scenes. To ensure the accuracy and timeliness of the 3D reconstruction, this embodiment only sets up a semantic segmentation map for one category, namely "electric scooter," as shown below. Figure 3 As shown.

[0203] (3) For each semantic segmentation map, regions are divided based on category labels. Feature points are extracted using the SIFT algorithm, and the cross-category region matching penalty coefficient is set to 0.2 (matching weight reduced by 80%), prioritizing matching within the same category region. Feature points within the "electric scooter" category region are required to only have a high-intensity match with feature points in the "electric scooter" region of other views, rejecting mismatches with other regions.

[0204] (4) The semantically constrained feature point pairs are imported into the bundle adjustment optimization framework to construct an optimization model that includes camera intrinsic parameters (focal length, principal point, etc.) and extrinsic parameters (rotation, translation). With the goal of minimizing reprojection error, the camera pose is iteratively optimized, reducing the average reprojection error from 1.5 pixels in traditional methods to within 0.8 pixels. This yields... Figure 4 The camera pose shown.

[0205] (5) Differentiated depth optimization and point cloud generation

[0206] For the "electric scooter" flexible category region, depth optimization is performed by combining texture continuity constraints and utilizing the texture similarity of adjacent views, specifically as follows: Figure 5 As shown; then, based on the optimized depth map, a dense point cloud is generated, specifically as follows: Figure 6 As shown.

[0207] This invention utilizes DeepLab's semantic segmentation capabilities to segment the differences between different objects, reducing errors in feature extraction and matching. It only requires a simple and inexpensive monocular camera or mobile phone to capture multi-view image sequences, making it easy to operate and inexpensive. It can perform 3D reconstruction for different scenes, making it highly applicable.

Claims

1. A multi-view 3D reconstruction method based on Deeplab semantic segmentation, characterized in that, Includes the following steps: S1: Obtain a multi-view image sequence of the target scene; S2: Use the trained Deeplab semantic segmentation model to perform pixel-level semantic annotation on each view image to obtain a semantic segmentation map containing object category information; S3: Based on the semantic segmentation results, perform semantic constraint feature matching on regions belonging to the same semantic category in different views; S4: Combine the semantic constraint feature matching results to optimize and estimate the camera pose; S5: Based on the optimized camera pose, a differentiated deep optimization strategy is adopted for regions with different semantic categories; S6: Based on the optimized depth information, generate dense point cloud data of the target scene, and finally output a high-precision 3D reconstruction model.

2. The multi-view 3D reconstruction method based on Deeplab semantic segmentation according to claim 1, characterized in that, In step S2, the Deeplab semantic segmentation model uses HRNet as its backbone network, and the model output formula is: S(i,j)=argmax c∈{1,2,…,C} P(c|x i,j ) Where: x i,j P(c)x represents the feature of the input image at pixel (i,j); i,j ) represents the probability that the pixel belongs to category c; S(i,j) is the category label of pixel (i,j) in the semantic segmentation map, which is the final output segmentation result.

3. The multi-view 3D reconstruction method based on Deeplab semantic segmentation according to claim 1, characterized in that, In step S3, the SIFT algorithm is used to extract and match feature points for regions belonging to the same semantic category in different views. The specific process includes: A1: Scale Space Construction Scale Space Fundamentals: The scale space function L(x,y,σ) is obtained by convolving the original image I(x,y) with a Gaussian function G(x,y,σ) of variable scale: L(x,y,σ)=G(x,y,σ)*I(x,y) Among them, Gaussian function * indicates a convolution operation, and L(x,y,σ) is the image representation at scale σ; Difference of Gaussian Pyramid (DOG): Obtained by subtracting Gaussian images from adjacent scales, used to detect scale-space extrema: D(x,y,σ)=(G(x,y,kσ)-G(x,y,σ))*I(x,y) Where: k is a constant; A2: Feature Point Detection Scale-space extreme point detection: In the DOG pyramid, each pixel needs to be compared with its 8 neighborhood pixels at its own scale and 18 pixels at its adjacent scales. If it is a local extremum, it is considered a candidate feature point. D(x,y,σ)>max{D(26 neighborhood pixels)} or D(x,y,σ)>max{D(26 neighborhood pixels)} Edge response filtering: Calculate the trace and determinant of the Hessian matrix, and filter by curvature ratio: Hessian matrix: Trace Tr(H) = D xx +D yy Determinant det(H) = D xx D yy -(D xy ) 2 curvature ratio When r < (r+1) 2 When / r, feature points are preserved; A3: Direction Assignment Gradient calculation: Where m(x,y) is the gradient magnitude and θ(x,y) is the gradient direction; A4: Feature Descriptor Regional division: Centered on the feature point, a 16×16 area is selected and divided into 4×4 sub-regions; Gradient direction: The gradient values ​​in 8 directions are counted for each sub-region, resulting in a 4×4×8=128-dimensional vector; A5: Feature Matching The matching degree between two feature points is measured by the Euclidean distance between their 128-dimensional descriptors: in: These are the descriptors for feature points A and B, respectively.

4. The multi-view 3D reconstruction method based on Deeplab semantic segmentation according to claim 1, characterized in that, In step S3, a feature matching penalty mechanism is set to reduce the matching weight of non-homologous regions. The feature matching penalty mechanism includes: Let C p C represents the semantic category to which p belongs. q If q belongs to a specific semantic category, then the semantic consistency factor is defined as follows: Where α∈(0,1) is the penalty coefficient.

5. The multi-view 3D reconstruction method based on Deeplab semantic segmentation according to claim 1, characterized in that, In step S3, feature point matching and motion estimation are preferentially performed within semantic regions of the same type to reduce cross-class mismatches. Improve the accuracy and robustness of feature matching.

6. The multi-view 3D reconstruction method based on Deeplab semantic segmentation according to claim 1, characterized in that, The optimization estimation of camera pose in step S4 includes: using the semantically matched feature point pairs as constraints, incorporating them into the bundle adjustment model, correcting the camera pose estimation error caused by feature mismatch, and obtaining camera pose information.

7. A multi-view 3D reconstruction method based on Deeplab semantic segmentation according to claim 6, characterized in that, The objective function of the bundle adjustment model in step S4 is: Where: M is the number of camera views, and N is the number of 3D points; (R i ,t i X represents the pose of the i-th camera (rotation matrix and translation vector, i.e., extrinsic parameters); j =(x j ,y j ,z j ) T Let x be the world coordinates of the j-th 3D point; ij =(u ij ,v ij ) T For a 3D point X j The observed pixel coordinates on the i-th image; π(R i ,t i ,X j ) is the projection function, which projects the three-dimensional point X. j The intrinsic and extrinsic parameters of the camera are projected onto the i-th image; ω ij This is the weighting factor.

8. A multi-view 3D reconstruction method based on Deeplab semantic segmentation according to claim 7, characterized in that, The process of incorporating the bundle adjustment model in step S4 includes: B1: Iterative Optimization The solution to semantic constraints (BA) is achieved through iterative optimization using the Levenberg-Marquardt (LM) algorithm, which specifically includes: B1-1: Initialization: Based on the results of motion estimation and triangulation of triangulation points As initial value; B1-2: Constructing the Jacobian matrix: For each camera pose parameter (R) i ,t i ) and three-dimensional point (X) j ), calculate the Jacobian matrix J; B1-3: Solving the incremental equations: In each iteration, the parameter increment Δ is calculated using the LM algorithm, with the following formula: (J T WJ+λI)Δ=J T We Where W is the diagonal weight matrix, e is the reprojection error vector, λ is the LM damping factor, and I is the identity matrix; B1-4: Update parameters: If the increment reduces the objective function value, then update the camera pose (R). i =R i +ΔR i ,t i =t i +Δt i ) and three-dimensional point (X) j =X j +ΔX j ); B1-5: Convergence Judgment The iteration stops when the number of iterations reaches a threshold or the parameter increment is less than the threshold, and the optimized camera pose is obtained. B2: Camera pose correction, specifically including: B2-1: Semantic Consistency Guided Correction: If the initial camera pose is deviated due to mismatch, the semantic weight will amplify the reprojection error of the correct matching point, forcing the optimization process to adjust the camera pose to reduce these errors, thereby correcting the pose deviation. B2-2: Low weighting of cross-category matching points achieves out-of-point suppression; B2-3: Global Consistency Optimization The BA algorithm is used to simultaneously optimize the poses of all cameras and 3D points. Semantic constraints ensure that points of the same semantic type are projected consistently in multiple views, ultimately enabling the camera pose to satisfy global geometric constraints.

9. A multi-view 3D reconstruction method based on Deeplab semantic segmentation according to claim 1, characterized in that, The differentiated deep optimization strategy adopted in step S5 for regions of different semantic categories includes: For rigid semantic category regions, depth is optimized using geometric constraints, specifically including: Let the pixel coordinates of a 3D point X in view i be x. i =(u i ,v i ), with a depth of z i Through camera extrinsic parameters (R k ,t k Projecting the predicted pixels onto view k yields the predicted pixels. Where: K i π is the camera intrinsic parameter, and π is the projection function; The optimization objective is to minimize the difference between the predicted pixel and the actual observed pixel x. k Deviation: For flexible semantic category regions, depth estimation is performed by combining texture continuity and semantic consistency constraints, specifically including: Let the depth of pixel (u,v) be z(u,v), and its image grayscale gradient be... Then depth gradient Negatively correlated with image gradient: Among them, I u and I v These represent the gradients in the horizontal and vertical directions, respectively; σ is an adjustment parameter used to control the smoothing intensity.

Images