A Multi-View Image Semantic Two-Dimensional Mapping Annotation Method Based on 3D Reconstruction

Abstract

This invention relates to a multi-view image semantic 2D mapping annotation method based on 3D reconstruction. It involves performing 3D reconstruction and semantic labeling on highly overlapping multi-view input images to obtain an annotated 3D model with semantic labels. Then, a joint rendering technique is used to render the annotated 3D model from the perspectives of each input image, generating multiple rendered depth maps and patch index maps of the annotated 3D model. Depth difference detection is then used to detect occlusion regions, and pixel-level visibility confidence scores for each pixel in the annotated 3D model are calculated. Based on the pixel occlusion detection results and all obtained pixel-level visibility confidence scores, the semantic labels on the annotated 3D model are dynamically mapped to the 2D image. This achieves accurate mapping from 3D semantics to 2D annotation, ensuring that the target is labeled with the same semantic information from any perspective, achieving consistent annotation covering the entire viewpoint of the target.

Description

Technical Field

[0001] This invention relates to the field of surveying and mapping, and in particular to a two-dimensional semantic mapping annotation method for multi-view images based on three-dimensional reconstruction. Background Technology

[0002] In the field of surveying and mapping, data annotation and processing are core foundational steps in machine learning and AI model training. Current mainstream AI models, such as deep learning, rely on supervised learning, requiring labeled data to establish a mapping relationship between input features and output targets, enabling the model to learn patterns and rules within the data. The scale of high-quality labeled data directly impacts the model's accuracy.

[0003] To label data, existing data labeling methods mainly include manual frame-by-frame labeling, semi-automatic labeling tools, and multi-view joint labeling. Among them, manual frame-by-frame labeling involves manually labeling each frame in a video or continuous image sequence to indicate its target category and location, which is inefficient and costly; semi-automatic labeling uses a pre-trained model to generate preliminary labels, which are then manually corrected, but still requires a lot of manual intervention; multi-view joint labeling cross-validates label consistency through data from multiple cameras, but it does not solve the labeling conflict problem caused by viewpoint occlusion and deformation.

[0004] In general, existing data annotation methods have the following problems: during the data annotation process, the same object may be labeled inconsistently from different viewpoints due to occlusion, lighting or deformation, making it difficult to achieve consistent annotation that covers the target from all viewpoints. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a multi-view image semantic two-dimensional mapping annotation method based on three-dimensional reconstruction, which addresses the aforementioned existing technologies. This multi-view image semantic annotation back-inference method can generate a three-dimensional model based on multi-view images, and then map the semantic annotation of the three-dimensional model onto the two-dimensional image, thereby achieving consistent annotation covering the entire view of the target.

[0006] The technical solution adopted by this invention to solve the above-mentioned technical problems is: a multi-view image semantic two-dimensional mapping annotation method based on three-dimensional reconstruction, characterized by comprising the following steps:

[0007] Step 1: Perform 3D reconstruction on the highly overlapping multi-view input image to generate an original 3D model, and label the original 3D model with semantic tags to obtain a labeled 3D model with semantic tags; wherein, the original 3D model has several triangular facets, and each triangular facet has a semantic tag corresponding to its 3D model.

[0008] Step 2: Using joint rendering technology, the annotated 3D model is rendered from the perspective of each input image in the high-overlapping multi-view input image, and multiple rendering depth maps and multiple facet index maps of the annotated 3D model are generated simultaneously. The number of rendering depth maps and facet index maps is the same as the number of high-overlapping multi-view input images, and the facet index map directly records the triangular facet ID corresponding to each pixel.

[0009] Step 3: Use the multi-view stereo matching method to estimate the scene depth map of each input image in the highly overlapping multi-view input images, and perform pixel-by-pixel subtraction between the rendered depth map of each input image under the same camera view and the scene depth map to obtain the depth difference value for detecting the occluded area of ​​each pixel in each input image.

[0010] Step 4: Calculate the viewing angle weight of each pixel, and based on the preset pixel depth difference attenuation factor and the obtained viewing angle weight of the corresponding pixel, calculate the pixel-level visibility confidence of each pixel in each input image.

[0011] Step 5: Process the data based on the depth difference values ​​of each pixel in the input image and the obtained pixel-level visibility confidence scores to dynamically map the semantic labels on the annotated 3D model to the 2D image.

[0012] Improved, in the multi-view image semantic 2D mapping annotation method based on 3D reconstruction, the 3D reconstruction process of the original 3D model in step 1 includes the following steps:

[0013] Step a1: Calculate the camera pose and sparse point cloud of the highly overlapping multi-view input image using motion reconstruction structure;

[0014] Step a2: Calculate the scene depth map of each input image in the high-overlapping multi-view input image based on the camera pose and sparse point cloud of the obtained high-overlapping multi-view input image using the multi-view stereo matching method, and generate the dense point cloud of the high-overlapping multi-view input image based on the scene depth maps of all the obtained input images.

[0015] Step a3: Based on the obtained dense point cloud, a three-dimensional model is generated using the Poisson reconstruction method; wherein, the three-dimensional model is the original three-dimensional model.

[0016] In a further improvement, the semantic two-dimensional mapping annotation method for multi-view images based on three-dimensional reconstruction, step 1, which involves annotating the original three-dimensional model with semantic labels, includes the following steps:

[0017] The original 3D model is visually interpreted using human-computer interaction or by employing an automatic annotation algorithm to assign category semantic labels to each triangular facet, generating a 3D model with semantic labels. This 3D model is then used as the annotated 3D model; wherein, the annotated 3D model is labeled as... M , M =( V , F , S ), , , ; V For the annotated 3D model M The set of vertices formed by the vertices of all the triangular faces. v i For vertex set V The first i One vertex, N v For vertex set V The total number of vertices within. R 3 Indicates the position of a point on a 3D model; F For the annotated 3D model M The set of triangular facets formed by all the triangular facets on the surface. f j A set of triangular facets F The first j A triangular facet, N f A set of triangular facets F The total number of triangular facets within; S For the annotated 3D model M The semantic set formed by the semantics of all the triangular facets on the surface. s j For semantic set S The first j One semantic; Z Semantics s j The category to which it belongs.

[0018] Improved, in the multi-view image semantic 2D mapping annotation method based on 3D reconstruction, in step 2, the joint rendering process is as follows: Each triangular facet in the annotated 3D model is projected onto the image plane using rasterization technology, and for pixels that pass the z-buffer test, the depth value and triangular facet ID of the triangular facet are written to the corresponding pixel; wherein, the joint rendering function of the joint rendering process is... , C k = ( Kk , R k , t k ), R k ∈ SO (3), t k ∈ R 3 ; D k For the rendered depth map, record the depth value corresponding to each pixel; J k This is a patch index map, recording the triangular patch ID corresponding to each pixel; C k For the first k The camera parameter group corresponding to each image; It is a symbolic function representation of the joint rendering function. The output is a depth map D k Dough sheet index diagram J k ; K k This is the camera's intrinsic parameter matrix, describing the camera's inherent properties such as focal length and principal point. R k This is the rotation matrix, describing the direction of rotation of the camera coordinate system relative to the world coordinate system; t k This is a translation vector, describing the position of the camera center in the world coordinate system; SO (3) is the Special Orthogonal Group in 3 Dimensions, which is the set of all rotation matrices in three-dimensional space.

[0019] In a further improvement, in the multi-view image semantic two-dimensional mapping annotation method based on three-dimensional reconstruction, the pixel occlusion region detection process in step 3 includes the following steps:

[0020] Step b1: Calculate the scene depth map of each pixel in the annotated 3D model, and the pixel depth difference between the rendered depth map of each pixel and the corresponding scene depth map; where, the pixel depth difference in the annotated 3D model... p The corresponding scene depth map is marked as D scene,k ( p ), pixels p Rendering depth map D k ( p ) and scene depth map Dscene,k ( p The pixel depth difference value between ) is labeled as Δ d ( p ), The scene depth map of each pixel in the annotated 3D model and the scene depth map of the corresponding pixel are calculated pixel by pixel under the same camera view.

[0021] Step b2: Make a judgment based on the pixel depth difference value corresponding to each pixel: when the pixel depth difference value exceeds a preset threshold, determine that the pixel corresponding to the pixel depth difference value is a pixel occlusion area, and discard the label mapping of the pixel occlusion area; otherwise, determine that the pixel corresponding to the pixel depth difference value is an unoccluded area.

[0022] Furthermore, in the multi-view image semantic two-dimensional mapping annotation method based on three-dimensional reconstruction, in step 4, the pixel-level visibility confidence is calculated as follows: Conference ( p )= oh depth ×max (0 , n j · v p ); ;in, Conference ( p (pixels) p Pixel-level visibility confidence, oh depth This is the penetration depth coefficient. max (0 , n j · v p ) represents the weight of the viewing angle, n j Triangular facets in a 3D model f j The unit normal vector, v p To move from the camera center to the pixel p The ray direction vector, c This represents the depth difference attenuation coefficient.

[0023] In a further improvement, in the multi-view image semantic 2D mapping annotation method based on 3D reconstruction, step 5, the process of dynamically mapping the semantic labels on the annotated 3D model to the 2D image, includes the following steps:

[0024] Step c1: Determine whether each pixel in the annotated 3D model is detected as an occluded region.

[0025] If a pixel is not detected as an occluded region, proceed to step c2; otherwise, if a pixel is detected as an occluded region, discard the semantic labeling operation for the current pixel and do not perform mapping processing on the pixel.

[0026] Step c2: Make a judgment based on the pixel-level visibility confidence of the current pixel, the preset upper limit value, and the preset lower limit value:

[0027] When the pixel-level visibility confidence of a pixel is greater than or equal to a preset upper limit threshold, the semantic label located on the labeled 3D model and corresponding to the pixel is directly mapped to the corresponding pixel position in the 2D image.

[0028] When the pixel-level visibility confidence of a pixel is between the preset upper limit and the preset lower limit, the pixel is marked as awaiting manual review.

[0029] When the pixel-level visibility confidence of a pixel is less than a preset lower limit, the semantic labeling operation for the current pixel is discarded.

[0030] Improved, in the multi-view image semantic two-dimensional mapping annotation method based on three-dimensional reconstruction, when the high-overlap multi-view input image is a continuous frame video stream data, the depth map of the static region is reused, and the depth map of multi-view stereo matching is recalculated only for the moving region in the high-overlap multi-view input image.

[0031] Furthermore, in the multi-view image semantic two-dimensional mapping annotation method based on three-dimensional reconstruction, optical flow detection is performed on the highly overlapping multi-view input image to detect the motion region, and the motion state is determined by preset motion detection conditions; wherein, the preset motion detection conditions are set as follows: d , Where Δu is the pixel displacement between adjacent frames in a highly overlapping multi-view input image, and ║Δu║ is the magnitude of the pixel displacement Δu; d For the displacement threshold of pixel displacement, u t For a highly overlapping multi-view input image, u represents the pixel position of the previous frame t. t+1 The pixel position of the next frame t+1 in the highly overlapping multi-view input image.

[0032] Further improvements are made to the multi-view image semantic 2D mapping annotation method based on 3D reconstruction. During the reuse of the depth map of a static region, the update rule for this depth map is set as follows:

[0033] ;

[0034] in, D scene,t+1 ( p (pixels)p Updated depth map, D scene,t ( p (pixels) p Depth map before update D MVS This is a depth map obtained by recalculating multi-view stereo matching of moving regions in a highly overlapping multi-view input image.

[0035] Compared with the prior art, the advantages of the present invention are as follows:

[0036] First, the proposed method for semantic two-dimensional mapping annotation of multi-view images based on 3D reconstruction performs 3D reconstruction and semantic labeling on highly overlapping multi-view input images to obtain an annotated 3D model with semantic labels. Then, a joint rendering technique is used to render the annotated 3D model, simultaneously generating pixel rendering depth maps and pixel patch index maps for each pixel in the corresponding image. Depth differences are then used to detect occlusion regions, and pixel-level visibility confidence scores for each pixel in the annotated 3D model are calculated. Based on the pixel occlusion detection results and all obtained pixel-level visibility confidence scores, the semantic labels on the annotated 3D model are dynamically mapped to the two-dimensional image. This achieves accurate mapping from 3D semantics to 2D annotation, solving the consistency problem in conventional multi-view annotation processes, ensuring that the target is labeled with the same semantic information from any viewpoint, and achieving consistent annotation covering the entire viewpoint of the target.

[0037] Secondly, when the input image of a highly overlapping multi-view view is a continuous frame video stream, this invention reuses the depth map of the static region and only recalculates the depth map of the multi-view stereo matching for the moving region in the highly overlapping multi-view input image. This avoids redundant annotation calculations for non-moving regions in the input image, saving computational resources.

[0038] In addition, the invention’s multi-view image semantic two-dimensional mapping annotation method based on three-dimensional reconstruction realizes the unified semantic expression of objects in three-dimensional space, giving them three-dimensional semantics and meeting the needs of tasks that require three-dimensional understanding, such as autonomous driving and robot navigation. Moreover, the invention can also cover rare perspectives such as extreme elevation angles and occluded perspectives, meeting the applicability of the model in edge scenes. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the process of the multi-view image semantic two-dimensional mapping annotation method based on three-dimensional reconstruction in an embodiment of the present invention. Detailed Implementation

[0040] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0041] This embodiment provides a method for semantic two-dimensional mapping annotation of multi-view images based on three-dimensional reconstruction. Specifically, see [link to documentation]. Figure 1 As shown, the multi-view image semantic two-dimensional mapping annotation method based on three-dimensional reconstruction in this embodiment includes the following steps 1 to 5:

[0042] Step 1: Perform 3D reconstruction on the highly overlapping multi-view input image to generate an original 3D model, and label the original 3D model with semantic tags to obtain a labeled 3D model with semantic tags; wherein, the original 3D model has several triangular facets, and each triangular facet has a semantic tag corresponding to its 3D model; that is, for the same 3D model, all triangular facets in the 3D model have the same semantic tag.

[0043] Step 2: Using joint rendering technology, the annotated 3D model is rendered from the perspective of each input image in the highly overlapping multi-view input image, simultaneously generating multiple rendering depth maps and multiple facet index maps of the annotated 3D model. The number of rendering depth maps and facet index maps is consistent with the number of highly overlapping multi-view input images. The facet index map directly records the triangle facet ID corresponding to each pixel. In this embodiment, the triangle facet ID is a unique "ID number" or "number" assigned to each triangle facet in the 3D model.

[0044] Step 3: Use the multi-view stereo matching method to estimate the scene depth map of each input image in the highly overlapping multi-view input images, and perform pixel-by-pixel subtraction between the rendered depth map of each input image under the same camera view and the scene depth map to obtain the depth difference value used to detect the occlusion region of each pixel in each input image; where each input image has several pixels, and the depth difference value is used to detect the pixel occlusion region of the corresponding pixel.

[0045] Step 4: Calculate the viewing angle weight of each pixel, and based on the preset pixel depth difference attenuation factor and the obtained viewing angle weight of the corresponding pixel, calculate the pixel-level visibility confidence of each pixel in each input image.

[0046] Step 5: Based on the depth difference values ​​of each pixel in the input image and the obtained pixel-level visibility confidence scores, process the semantic labels on the annotated 3D model to the 2D image to achieve consistent annotation covering the entire view of the target.

[0047] In this embodiment, the 3D reconstruction process of the original 3D model includes the following steps a1 to a3:

[0048] Step a1: Calculate the camera pose and sparse point cloud of the highly overlapping multi-view input image using Structure-from-Motion (SfM); where, camera pose refers to where each image is taken from and in which direction.

[0049] Step a2: Calculate the scene depth map of each input image in the high-overlapping multi-view input image based on the camera pose and sparse point cloud of the obtained high-overlapping multi-view input image using the multi-view-stereo (MVS) matching method, and generate the dense point cloud of the high-overlapping multi-view input image based on the scene depth maps of all the obtained input images.

[0050] Step a3: Based on the obtained dense point cloud, a three-dimensional model is generated using the Poisson reconstruction method; wherein, the three-dimensional model is the original three-dimensional model.

[0051] Specifically, in this embodiment, in step 1, the process of annotating the original 3D model with semantic labels is as follows: Visual interpretation is performed using human-computer interaction, or an automatic annotation algorithm is used to assign category semantic labels to each triangular facet of the original 3D model, generating a 3D model with semantic labels, and this 3D model is used as the annotated 3D model; wherein, the annotated 3D model is labeled as... M , M =( V , F , S ), , , ; V For the annotated 3D model M The set of vertices formed by the vertices of all the triangular faces. v i For vertex set V The first i One vertex, N v For vertex set V The total number of vertices within. R 3 Indicates the position of a point on a 3D model; F For the annotated 3D model M The set of triangular facets formed by all the triangular facets on the surface. f j A set of triangular facets F The j-th triangular facet inside, N f A set of triangular facets F The total number of triangular facets within; S For the annotated 3D model MThe semantic set formed by the semantics of all the triangular facets on the surface. s j For semantic set S The j-th semantic within; Z Semantics s j Category: Set of triangular facets F The total number of triangular faces within the model is equal to the total number of semantic elements on all triangular faces in the annotated 3D model.

[0052] Furthermore, in step 2 of this embodiment, the aforementioned joint rendering process is as follows: Each triangular facet in the annotated 3D model is projected onto the image plane using rasterization technology, and for pixels that pass the z-buffer test, the depth value and triangular facet ID of the facet are written to the corresponding pixel; wherein, the joint rendering function of the joint rendering process is... , C k = ( K k , R k , t k ), R k ∈ SO (3), t k ∈ R 3 ; D k For the rendered depth map, record the depth value corresponding to each pixel. J k This is a patch index map, recording the triangular patch ID corresponding to each pixel; C k For the first k The camera parameter group corresponding to each image; It is a symbolic function representation of the joint rendering function; joint rendering function The output is a depth map D k Dough sheet index diagram J k ; K k This is the camera's intrinsic parameter matrix, describing the camera's inherent properties such as focal length and principal point. R k This is the rotation matrix, describing the direction of rotation of the camera coordinate system relative to the world coordinate system; t k This is a translation vector, describing the position of the camera center in the world coordinate system; SO(3) is a three-dimensional rotation group (Special Orthogonal Group in 3 Dimensions), which is the set of all rotation matrices in three-dimensional space. SO (3) is common knowledge in this field.

[0053] Specifically, in step 3 of this embodiment, the detection process of the aforementioned pixel occlusion region includes the following steps b1~b2:

[0054] Step b1: Calculate the scene depth map of each pixel in the annotated 3D model, and the pixel depth difference between the rendered depth map of each pixel and the corresponding scene depth map; where, the pixel depth difference in the annotated 3D model... p The corresponding scene depth map is marked as D scene,k ( p ), pixels p Rendering depth map D k ( p ) and scene depth map D scene,k ( p The pixel depth difference value between ) is labeled as Δ d ( p ), The scene depth map of each pixel in the annotated 3D model and the scene depth map of the corresponding pixel are calculated pixel by pixel under the same camera view.

[0055] Step b2: Make a judgment based on the pixel depth difference value corresponding to each pixel: when the pixel depth difference value Δ d ( p When the difference exceeds a preset threshold τ, the pixel depth difference value Δ is determined. d ( p The pixel corresponding to ) p If the pixel is an occluded region, the label mapping for that occluded region is discarded; otherwise, the pixel corresponding to the pixel depth difference value is determined to be an unoccluded region. In this embodiment, the preset threshold τ is set to 0.1 times the diameter of the object's smallest circumcircle.

[0056] More specifically, in step 4 of this embodiment, the aforementioned pixel-level visibility confidence is calculated as follows: Conference ( p )= oh depth ×max (0 , n j · v p ); ;in, Conference ( p (pixels) p Pixel-level visibility confidence, oh depth This is the penetration depth coefficient. max (0 , n j · v p ) represents the weight of the viewing angle, n j Triangular facets in a 3D model f j The unit normal vector, v p To move from the camera center to the pixel p The ray direction vector; c This refers to the depth difference attenuation coefficient. c For example, a depth difference attenuation coefficient is used to control the rate at which pixel-level visibility confidence changes with depth difference. c The range of values ​​for is 0.3 ≤ c ≤0.8; In this embodiment, the depth difference attenuation coefficient c The preferred setting is 0.5; Δ d ( p (pixels) p Rendering depth map D k ( p ) and scene depth map D scene,k ( p The pixel depth difference value between ).

[0057] Regarding the angle of view, it should be noted that the closer the angle between the view and the ray is to 180°, the higher the weight of that angle.

[0058] Specifically, in step 5 of this embodiment, the process of dynamically mapping the semantic labels on the annotated 3D model to the 2D image includes the following steps c1~c2:

[0059] Step c1: Determine whether each pixel in the annotated 3D model is detected as an occluded region.

[0060] If a pixel is not detected as an occluded region, proceed to step c2; otherwise, if a pixel is detected as an occluded region, discard the current pixel. p The semantic labeling operation, that is, not for this pixel p Perform mapping processing;

[0061] Step c2, based on the pixel-level visibility confidence of the current pixel. Conference ( p The system makes judgments and processes based on the preset upper limit value 'a' and the preset lower limit value 'b'.

[0062] When the pixel-level visibility confidence level of a pixel Conference ( p When the value is greater than or equal to the preset upper limit threshold 'a', the pixel located on the annotated 3D model will be the one corresponding to that pixel. p The semantic tags are directly mapped to the corresponding pixel positions in the two-dimensional image;

[0063] When the pixel-level visibility confidence level of a pixel Conference ( p When the pixel is between the preset upper limit value a and the preset lower limit value b, the pixel is... p Marked as pending manual review; for example, the preset upper limit value 'a' is set to 0.9 and the preset lower limit value 'b' is set to 0.7.

[0064] When the pixel-level visibility confidence level of a pixel Conference ( p If the value is less than the preset lower limit b, discard the current pixel. p Semantic tagging operations.

[0065] To reduce the computational cost of the multi-view image semantic 2D mapping annotation method, specifically, in this embodiment, when the high-overlap multi-view input image is a continuous frame video stream, the depth map of the static region is reused, and the depth map of multi-view stereo matching is recalculated only for the moving regions in the high-overlap multi-view input image. Specifically, optical flow detection is performed on the high-overlap multi-view input image to detect the moving regions, and the motion state is determined by a preset motion detection condition; wherein, the preset motion detection condition is set as: ║Δu║> d, Where Δu is the pixel displacement between adjacent frames in a highly overlapping multi-view input image, and ║Δu║ is the magnitude of the pixel displacement Δu; d For the displacement threshold of pixel displacement, u t For a highly overlapping multi-view input image, u represents the pixel position of the previous frame t. t+1 The pixel position of the next frame t+1 in the highly overlapping multi-view input image.

[0066] For example, in this embodiment, during the reuse of the depth map of the static region, the update rule for the depth map is set as follows: ;in, D scene,t+1 ( p (pixels) p Updated depth map, D scene,t ( p (pixels) p Depth map before update DMVS This is a depth map obtained by recalculating multi-view stereo matching of moving regions in a highly overlapping multi-view input image.

[0067] Although preferred embodiments of the present invention have been described in detail above, it should be clearly understood that various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-view image semantic two-dimensional mapping annotation method based on three-dimensional reconstruction, characterized in that, Includes the following steps: Step 1: Perform 3D reconstruction on the highly overlapping multi-view input image to generate an original 3D model, and label the original 3D model with semantic tags to obtain a labeled 3D model with semantic tags; wherein, the original 3D model has several triangular facets, and each triangular facet has a semantic tag corresponding to its 3D model. Step 2: Using joint rendering technology, the annotated 3D model is rendered from the perspective of each input image in the high-overlapping multi-view input image, and multiple rendering depth maps and multiple facet index maps of the annotated 3D model are generated simultaneously. The number of rendering depth maps and facet index maps is the same as the number of high-overlapping multi-view input images, and the facet index map directly records the triangular facet ID corresponding to each pixel. Step 3: Use the multi-view stereo matching method to estimate the scene depth map of each input image in the highly overlapping multi-view input image, and perform pixel-by-pixel subtraction between the rendered depth map of each input image under the same camera view and the scene depth map to obtain the depth difference value for detecting the occluded area of ​​each pixel in each input image. Step 4: Calculate the viewing angle weight of each pixel, and based on the preset pixel depth difference attenuation factor and the obtained viewing angle weight of the corresponding pixel, calculate the pixel-level visibility confidence of each pixel in each input image. Step 5: Process the semantic labels on the annotated 3D model to the 2D image based on the depth difference values ​​of each pixel in the input image and the obtained pixel-level visibility confidence.

2. The multi-view image semantic two-dimensional mapping annotation method based on three-dimensional reconstruction according to claim 1, characterized in that, In step 1, the 3D reconstruction process of the original 3D model includes the following steps: Step a1: Calculate the camera pose and sparse point cloud of the highly overlapping multi-view input image using motion reconstruction structure; Step a2: Calculate the scene depth map of each input image in the high-overlapping multi-view input image based on the camera pose and sparse point cloud of the obtained high-overlapping multi-view input image using the multi-view stereo matching method, and generate the dense point cloud of the high-overlapping multi-view input image based on the scene depth maps of all the obtained input images. Step a3: Based on the obtained dense point cloud, a three-dimensional model is generated using the Poisson reconstruction method; wherein, the three-dimensional model is the original three-dimensional model.

3. The multi-view image semantic two-dimensional mapping annotation method based on three-dimensional reconstruction according to claim 2, characterized in that, In step 1, the process of annotating the original 3D model with semantic labels includes the following steps: The original 3D model is visually interpreted using human-computer interaction or by employing an automatic annotation algorithm to assign category semantic labels to each triangular facet, generating a 3D model with semantic labels. This 3D model is then used as the annotated 3D model; wherein, the annotated 3D model is labeled as... M , M =( V , F , S ), , , ; V For the annotated 3D model M The set of vertices formed by the vertices of all the triangular faces. v i For vertex set V The first i One vertex, N v For vertex set V The total number of vertices within. R 3 Indicates the position of a point on a 3D model; F For the annotated 3D model M The set of triangular facets formed by all the triangular facets on the surface. f j A set of triangular facets F The first j A triangular facet, N f A set of triangular facets F The total number of triangular facets within; S For the annotated 3D model M The semantic set formed by the semantics of all the triangular facets on the surface. s j For semantic set S The first j One semantic; Z Semantics s j The category to which it belongs.

4. The multi-view image semantic two-dimensional mapping annotation method based on three-dimensional reconstruction according to claim 1, characterized in that, In step 2, the joint rendering process is as follows: Each triangular facet in the annotated 3D model is projected onto the image plane using rasterization technology, and for pixels that pass the z-buffer test, the depth value and triangular facet ID are written to the corresponding pixel; wherein, the joint rendering function in the joint rendering process is... , C k = ( K k , R k , t k ), R k ∈ SO (3), t k ∈ R 3 ; D k For the rendered depth map, record the depth value corresponding to each pixel; J k This is a patch index map, recording the triangular patch ID corresponding to each pixel; C k For the camera parameter group corresponding to the k-th image, It is a symbolic function representation of the joint rendering function. The output is a depth map D k Dough sheet index diagram J k ; K k This is the intrinsic parameter matrix of the camera. R k Let be a rotation matrix. t k It is a translation vector.

5. The multi-view image semantic two-dimensional mapping annotation method based on three-dimensional reconstruction according to claim 2, characterized in that, In step 3, the process of detecting pixel occlusion areas includes the following steps: Step b1: Calculate the scene depth map of each pixel in the annotated 3D model, and the pixel depth difference between the rendered depth map of each pixel and the corresponding scene depth map; where, the pixel depth difference in the annotated 3D model... p The corresponding scene depth map is marked as D scene,k ( p ), pixels p Rendering depth map D k ( p ) and scene depth map D scene,k ( p The pixel depth difference value between ) is labeled as Δ d ( p ), The scene depth map of each pixel in the annotated 3D model and the scene depth map of the corresponding pixel are calculated pixel by pixel under the same camera view. Step b2: Make a judgment based on the pixel depth difference value corresponding to each pixel: when the pixel depth difference value exceeds a preset threshold, determine that the pixel corresponding to the pixel depth difference value is a pixel occlusion area, and discard the label mapping of the pixel occlusion area; otherwise, determine that the pixel corresponding to the pixel depth difference value is an unoccluded area.

6. The multi-view image semantic two-dimensional mapping annotation method based on three-dimensional reconstruction according to claim 5, characterized in that, In step 4, the pixel-level visibility confidence score is calculated as follows: Conf ( p )= ω depth ×max (0 , n j · v p ); ;in, Conf ( p (pixels) p Pixel-level visibility confidence, ω depth This is the penetration depth coefficient. max (0 , n j · v p ) represents the weight of the viewing angle, n j Triangular facets in a 3D model f j The unit normal vector, v p To move from the camera center to the pixel p The ray direction vector, γ This represents the depth difference attenuation coefficient.

Images