A sparse multi-view image plane-level consistency segmentation method without depth dependency

Abstract

The application discloses a depth-independent sparse multi-view image plane-level consistency segmentation method, which comprises the following steps: step 1, acquiring sparse multi-view RGB images and their internal and external parameters, and extracting two-dimensional instance masks of each view; step 2, generating a global proxy point cloud skeleton and a surface normal vector based on the multi-view two-dimensional masks; step 3, strictly mapping each two-dimensional mask to a three-dimensional space through forward projection of a 3D point cloud to a 2D mask and double visibility screening; step 4, taking a two-dimensional mask as a node, taking corresponding three-dimensional point set geometric features and view co-visibility relations as edge weights, and constructing a multi-dimensional semantic-geometric index polarization force field graph; step 5, performing segmentation and clustering on the force field graph based on a polarization energy threshold, and obtaining three-dimensional consistency plane instance clusters; step 6, performing unsupervised consistency evaluation on each instance cluster and outputting a globally consistent segmentation result, and generating a high-quality multi-view consistency segmentation result.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, 3D reconstruction and pattern recognition, and in particular to a depth-independent sparse multi-view image plane-level consistency segmentation method. Background Technology

[0002] In response to the needs of industrial component digitization and CAD reverse engineering, 3D image reconstruction technology

[0003] Playing a central role. In recent years, with the development of neural rendering technologies such as Neural Radiation Field (NeRF) and 3D Gaussian sputtering...

[0004] With the emergence of 3DGS, multi-view 3D reconstruction has received increasing attention from both academia and industry. These methods are able to...

[0005] High-fidelity reconstruction of object geometry and appearance directly from sparse multi-view images. However, to ensure accurate reconstruction...

[0006] The structure is clean and free of redundancy, and it has independent multi-component editability. These advanced methods contribute to high-quality "multi-component" editing.

[0007] Multi-View Consistent Masks have very high prerequisite requirements.

[0008] Existing mainstream segmentation models typically only solve instance segmentation in a single, isolated two-dimensional image in their underlying logic.

[0009] Few systems are specifically designed for "multi-view consistent segmentation" that spans three-dimensional perspective transitions. Traditional serial methods encounter the following technical bottlenecks when facing real-world non-contact inspection scenarios:

[0010] 1. Deep Hardware Dependencies and Missing Components: Current cross-view 3D consistent connectivity systems with deep multiprocessing...

[0011] Continuous video streams (such as RGB-D data) heavily rely on hardware-level depth sensors. However, industrial production lines are often equipped only with pure RGB monocular camera arrays, which cannot provide physical depth, causing conventional algorithms that rely on rigid depth for occlusion detection to fail.

[0012] 2. Feature matching failure caused by solid color smoothing: As a traditional remedy in depthless scenarios, the industry typically...

[0013] Depth is measured using a multi-view stereo vision (MVS) algorithm based on SIFT / SURF image corner feature matching.

[0014] However, the parts to be inspected are usually large areas of solid metal / plastic color, smooth and lack significant local texture features, which causes the image mapping matching framework that relies on feature point extraction to fail.

[0015] 3. Three-dimensional cross-plane connectivity based on two-dimensional extensive correlation: Existing pure visual segmentation and concatenation schemes project the viewpoint

[0016] The "two-dimensional mask overlap area ratio" after image intersection is used as an attractive index for three-dimensional patch stitching. This method is used in processing...

[0017] When dealing with smooth geometries containing continuous structures such as three-dimensional corners and stepped differences, perspective distortion can cause variations in the appearance of different physical directions.

[0018] The mask has slight perspective overlap at adjacent edges, which can be incorrectly merged by the connected components of the graph, causing different faces to be merged.

[0019] And make it a single instance.

[0020] Therefore, developing a multi-view consistent segmentation system is necessary to solve the problem of reconstructing three-dimensional multi-view images from two-dimensional single-view segmentation.

[0021] The technology gap is a problem that urgently needs to be addressed. Summary of the Invention

[0022] Purpose of the invention: The technical problem to be solved by the present invention is to provide a depth-independent, sparse multi-view image planar consistency segmentation method to address the shortcomings of the prior art.

[0023] To address the aforementioned technical problems, this invention discloses a depth-independent sparse multi-view image planar level consistency segmentation method, comprising the following steps:

[0024] Step 1: Obtain sparse multi-view RGB image sequences and their corresponding camera intrinsic and extrinsic parameters, and use a 2D segmentation model to extract two-dimensional instance masks for each viewpoint as initial node primitives;

[0025] Step 2: Based on multi-view 2D masks, generate full-view masks using 3D symbolic distance field fusion and surface reconstruction techniques.

[0026] Local agent point cloud skeleton and surface normal vectors;

[0027] Step 3: Through forward projection of 3D point cloud to 2D mask and dual visibility filtering, each 2D mask is strictly mapped to 3D space to obtain the set of visible 3D points corresponding to each mask;

[0028] Step 4: Using the two-dimensional mask as nodes and its corresponding three-dimensional point set geometric features and view co-view relationship as edge weights, construct a multi-dimensional semantic-geometric exponential polarization force field diagram.

[0030] Step 5: Segment and cluster the force field map based on the polarization energy threshold to obtain a three-dimensional uniform plane instance cluster;

[0032] Step 6: Perform unsupervised consistency evaluation on each instance cluster and output a globally consistent partitioning result.

[0033] The 2D segmentation model mentioned in step 1 uses a pre-trained instance segmentation model (such as SAM, SegmentAnything Model).

[0034] Furthermore, the step 1 of obtaining the sparse multi-view RGB image sequence and its corresponding camera intrinsic and extrinsic parameters includes:

[0035] Step 1-1: Obtain a set of depthless, sparse, multi-view RGB camera image sequences around the non-contact target object. and the corresponding set of intrinsic and extrinsic parameter matrices. ;

[0036] Steps 1-2: Use a pre-trained 2D segmentation model to process discrete viewpoint image sets. Perform independent segmentation and extract the set of two-dimensional instance masks for each viewpoint. N is the number of instances, and each two-dimensional mask Serves as the initial node primitive for subsequent graph network structures.

[0037] Step 2, which describes generating a global proxy point cloud and surface normal vectors using 3D symbolic distance field fusion and surface reconstruction techniques, specifically includes:

[0038] Step 2-1: Perform a distance transformation on the two-dimensional mask for each viewpoint to generate a two-dimensional symbolic distance field. :

[0039]

[0040] in, Indicates the first All areas covered by the two-dimensional mask in each viewpoint This represents the pixel coordinates of the region. This indicates the pixel coordinates that do not belong to this region. This means taking all pixels that do not belong to this region and comparing them with the current point. The minimum Euclidean distance between them;

[0041] Step 2-2: Set a discrete virtual space exploration grid V with a fixed resolution (e.g., 128*128*128). For any grid point P = (X, Y, Z), fuse the two-dimensional distance fields from each viewpoint to obtain a three-dimensional symbolic distance field:

[0042]

[0043] in, This represents the standard perspective projection transformation function that projects a 3D point P onto the k-th viewpoint image plane using the camera's intrinsic and extrinsic parameter matrices.

[0044] Steps 2-3 involve using the Moving Cubes algorithm to extract... The isosurface, obtained

[0045] Obtain a smooth envelope mesh:

[0046]

[0047] Steps 2-4 involve generating a point cloud based on uniform area sampling from the mesh to obtain uniformly distributed proxy points.

[0048] cloud :

[0049]

[0050] in, Indicates uniform sampling;

[0051] Steps 2-5, using Gradient calculation of the precise normal vector of each point in the proxy point cloud

[0052]

[0053] in, For proxy point cloud The i-th point in This represents the global surface normal vector at that point. Represents the three-dimensional symbolic distance field At point The gradient vector at that point, The magnitude of the gradient vector is given by the negative sign, which is used to adjust the gradient direction to point outward from the surface.

[0054] Step 3, which involves forward projection from 3D point cloud to 2D mask and dual visibility filtering to strictly map each 2D mask to 3D space, specifically includes:

[0055] Step 3-1, global proxy point cloud The 3D points in the image are forward-projected onto each viewpoint through the camera's intrinsic and extrinsic parameter matrices.

[0056] From the angular image plane, obtain the two-dimensional projected pixel coordinates of each point. The same two-dimensional pixel coordinate It may correspond to multiple three-dimensional points, forming a candidate point set;

[0057] Step 3-2, for two-dimensional pixel coordinates For each projected candidate point in the corresponding candidate point set Calculate the direction of light from the camera Its global normal vector dot product ;like ( If the threshold is set to a preset value, then the point is determined to be a cutting surface point and is removed from the candidate set.

[0058] Step 3-3: For the pixels retained in the two-dimensional pixel coordinates after filtering in step 3-2... For multiple points in the candidate point set, construct a depth buffer at that pixel. Only retain depth values ​​and The difference does not exceed the tolerance The point, i.e., removing the back points;

[0059] Step 3-4, denote the set of visible points obtained in step 3-3 as... Based on its pixel coordinates The mask label is queried on the 2D mask image at the corresponding viewpoint, and the label is assigned to each 3D point in the set of visible points. Finally, the set of all visible points corresponding to the 2D pixel coordinates covered by each mask is summarized to obtain each 2D mask. The corresponding 3D point set .

[0060] The construction of the multidimensional semantic-geometric exponential polarization force field map described in step 4 includes:

[0061] Step 4-1, for each two-dimensional mask Create graph nodes And based on its set of visible points Calculate the three-dimensional centroid and fitting principal normal vector (By performing principal component analysis on the point set, the eigenvector corresponding to the smallest eigenvalue is obtained) );

[0062] Step 4-2, for any two sections Calculate the union of the 3D point sets corresponding to all viewpoints. Based on observational feedback, the total number of commonly visible viewpoints is defined as... The number of viewpoints on which the two masks are projected into the same 2D region is . Then the fundamental common gravity is:

[0063]

[0064] Step 4-3, calculate the spatial geometric deviation error:

[0065] Normal deviation error ,

[0066] Spatial centroid difference intercept error ;

[0067] Step 4-4, define the edge weights of the exponentially polarized force field:

[0068]

[0069] in, is the exponential penalty coefficient, with a value range of [0, 100].

[0070] Step 5, which involves segmenting and clustering the force field map based on the polarization energy threshold, includes:

[0071] Step 5-1, Set the cutting threshold ,like If the edge between the two nodes is broken, then the edge between the two nodes will be disconnected.

[0072] Step 5-2, in the remaining graph structure The above process involves connected component extraction, using a graph traversal algorithm (such as breadth-first search) to obtain all connected components and thus a node cluster. Each cluster corresponds to a three-dimensional consistency plane instance.

[0073] The unsupervised intra-cluster consistency evaluation and fully connected dimensionality reduction output described in step 6 include:

[0074] Step 6-1, for each instance cluster Aggregate all its visible points to obtain a point set. Calculate the centroid of the cluster. and covariance matrix

[0075] This represents the m-th instance cluster. This is the set of all visible points contained in the cluster. Let be the centroid of the cluster.

[0076] Let p be the covariance matrix of the cluster, and p denote the set. The point in the middle, | | indicates the number of points;

[0077] Step 6-2, for Perform eigenvalue decomposition to obtain eigenvalues. Calculate the expansion evaluation ratio ;like If the threshold is set to 0, then the cluster is determined to be a certain value. The memory is located on an unseparated heterogeneous plane, triggering the adjustment of graph structure parameters;

[0078] Step 6-3, for each instance cluster that passes the evaluation Assign a globally unique ID, and based on the original image position of each mask within the cluster, assign the ID to the two-dimensional region corresponding to each viewpoint, and output the multi-view consistent segmentation result.

[0079] A depth-independent sparse multi-view image plane-level consistent segmentation system, comprising:

[0080] The first module is used to acquire sparse multi-view RGB images and their intrinsic and extrinsic parameters.

[0081] The second module is used to extract two-dimensional instance masks from each viewpoint using a pre-trained 2D segmentation model;

[0082] The third module is used to generate a global array based on a two-dimensional mask using three-dimensional symbolic distance field fusion and surface reconstruction techniques.

[0083] proxy point cloud and normal vector;

[0084] The fourth module is used to accurately map a 2D mask to a 3D point set through forward projection of the 3D point cloud to the 2D mask and dual visibility filtering.

[0085] The fifth module is used to construct the exponential polarization force field map and perform segmentation and clustering based on the polarization energy threshold;

[0086] The sixth module is used to perform unsupervised evaluation and output globally consistent two-dimensional segmentation results.

[0087] Beneficial effects:

[0088] This invention can directly recover planar consistency segmentation results of 3D entities from 2D images, eliminating the dependence of previous technologies on depth hardware, dense point clouds, and manual annotation. Compared with traditional methods that use the same multi-view images but rely on depth sensors or texture features, this invention has the following significant advantages: it requires no physical depth sensor and can complete 3D planar instance segmentation using only pure RGB images; it does not rely on object surface textures and can still work stably in smooth, textureless scenes such as metal and plastic; it solves the cross-plane connectivity collapse problem caused by perspective projection through dual visibility filtering, ensuring that the segmentation results conform to the physical 3D structure; the final output multi-view consistency mask can be directly used for downstream tasks such as 3D reconstruction, reverse engineering, and CAD modeling, greatly improving the degree of automation. Attached Figure Description

[0089] Figure 1 This is a schematic diagram of the overall process of the present invention.

[0090] Figure 2 This is a schematic diagram of the final multi-view consistent segmentation result in one embodiment. Detailed Implementation

[0091] This invention proposes a depth-independent, plane-level consistent segmentation method for sparse multi-view images. The method includes the following steps:

[0092] Step 1: Obtain sparse multi-view RGB image sequences and their corresponding camera intrinsic and extrinsic parameters, and use a 2D segmentation model to extract two-dimensional instance masks for each viewpoint as initial node primitives;

[0093] Step 2: Based on multi-view 2D masks, generate full-view masks using 3D symbolic distance field fusion and surface reconstruction techniques.

[0094] Local proxy point cloud and surface normal vector;

[0095] Step 3: Through forward projection of 3D point cloud to 2D mask and dual visibility filtering, each 2D mask is strictly mapped to 3D space to obtain the set of visible 3D points corresponding to each mask;

[0096] Step 4: Using the two-dimensional mask as nodes and its corresponding three-dimensional point set geometric features and view co-view relationship as edge weights, construct a multi-dimensional semantic-geometric exponential polarization force field diagram.

[0098] Step 5: Segment and cluster the force field map based on the polarization energy threshold to obtain a three-dimensional uniform plane instance cluster;

[0100] Step 6: Perform unsupervised consistency evaluation on each instance cluster and output the globally unique label from all perspectives.

[0101] A two-dimensional segmentation mask for recognition.

[0102] Example 1:

[0103] Step 1: Obtain a sparse multi-view RGB image sequence and its corresponding camera intrinsic and extrinsic parameters. Simultaneously acquire images of an industrial machine part using 20 industrial cameras with different viewpoints to obtain an image set. The intrinsic and extrinsic parameter matrices of each image are obtained using the Zhang Zhengyou calibration method or pose estimation method. A pre-trained SAM model is then used to perform independent instance segmentation on each image, resulting in a set of two-dimensional instance masks for each viewpoint. ;

[0104] Step 2: Generate a global proxy point cloud. For each 2D mask... Calculate its two-dimensional distance field In three-dimensional space, the resolution is defined as For each grid point V, ,calculate Extraction using the moving cube algorithm The isosurface is used to obtain a smooth envelope mesh. The mesh is then uniformly sampled to generate a point cloud of 20,000 uniformly distributed points. ,use Gradient calculation of the normal vector for each point cloud ;

[0105] Step 3: Map each 2D mask to 3D space through forward projection from 3D point cloud to 2D mask and dual visibility filtering. All points are forward-projected onto the image planes of each viewpoint using the camera's intrinsic and extrinsic parameter matrices to obtain the pixel coordinates of each point. For each pixel of each viewpoint, perform a double filtering: first, calculate the viewing direction. With normal vector The dot product, if If a point is identified as a cut surface point, it is removed. Then, for the remaining points projected to the same pixel, the depth value is calculated. A depth buffer is constructed, retaining points with the minimum depth and a depth difference of no more than 0.08 from the minimum depth, i.e., removing back-facing points. The set of points that passes the filter is denoted as... Based on its pixel coordinates, the mask label is queried on the 2D mask image of the corresponding viewpoint, and the label is assigned to the 3D point to obtain the 3D point set corresponding to each 2D mask. .

[0106] Step 4: Construct the exponential polarization force field map. Calculate the centroid for each mask node. and fitted normal vector (right Perform PCA and extract the eigenvector corresponding to the smallest eigenvalue. Calculate the fundamental common-view gravity for all node pairs. The number of shared observation perspectives and the number of supporters were counted. The normal deviation error was calculated. Centroid difference intercept error .set up , The edge weight W is calculated according to the formula.

[0107] Step 5: Clustering based on polarization energy threshold. Set the cutting threshold. Then disconnect the edges, and then extract the connected components using a graph traversal algorithm to obtain the node clusters. Each cluster corresponds to a three-dimensional plane instance.

[0108] Step 6: Unsupervised intra-cluster consistency evaluation and output. For each instance cluster... Aggregate all its visible points to obtain a point set. Calculate the centroid and covariance matrix .right Perform eigenvalue decomposition to obtain eigenvalues. Calculate the expansion evaluation ratio .like Then the parameter adjustment is triggered (increase respectively). Re-cluster (up to 20, 80). Assign a globally unique ID to each cluster that passes the evaluation, and label the two-dimensional mask regions belonging to that cluster from each viewpoint with that ID, outputting the multi-view consistent segmentation results.

[0109] Through the above steps, this method can automatically obtain planar instance segmentation results with consistent multi-view perspectives from sparse multi-view RGB images without the need for depth sensors and manual annotation, thus verifying the feasibility and robustness of the method.

[0110] Example 2:

[0111] The validation was performed using a typical industrial part from the ABC-NEF dataset. This part is a complex structure made of metal.

[0112] The part has a smooth surface without obvious texture and includes multiple planar, cylindrical, and corner features. The method of the present invention is performed on this part according to the following steps, with specific parameters for each step as follows:

[0113] Step 1: The camera is evenly circled around the part to acquire 20 multi-view RGB images of the part, and the precise intrinsic and extrinsic parameters of each image are obtained. A pre-trained SAM model is used to perform independent instance segmentation on each image, resulting in a set of two-dimensional instance masks for each viewpoint. Approximately 180 initial masks were extracted to obtain sparse multi-view RGB image sequences and their corresponding camera intrinsic and extrinsic parameters.

[0114] Step 2: Generate the global proxy point cloud skeleton. For each 2D mask... Calculate its two-dimensional distance field In three-dimensional space, the resolution is defined as For each grid point V, ,Pass

[0115] Calculation of camera intrinsic and extrinsic parameters and projection point calculation The minimum value of the distance field from each viewpoint is taken as the three-dimensional symbolic distance field value of that point: Extraction using the moving cube algorithm The isosurface is used to obtain a smooth envelope mesh. The mesh is then uniformly sampled to generate a point cloud of 20,000 uniformly distributed points. ,use Gradient calculation of the normal vector for each point cloud The gradient calculation uses the central difference method, and the direction of the normal vector is uniformly adjusted to point outwards from the object.

[0116] Step 3: Map each 2D mask to 3D space through forward projection from 3D point cloud to 2D mask and dual visibility filtering. All points are forward-projected onto the image plane of each viewpoint using the camera's intrinsic and extrinsic parameter matrices to obtain the pixel coordinates (u, v) of each point. For each pixel in each viewpoint, a double filtering process is performed: - First, a double filtering process is then performed.

[0117] Polar angle back-facing elimination: Calculate the line-of-sight direction With normal vector dot product Set the threshold γ = −0.15, if If the point is not found to be a cutting surface point, it is removed from the candidate set. - Then proceed...

[0118] Row Z-Buffer Occlusion Filtering: For the remaining points projected to the same pixel, calculate the depth value of each point. (i.e., the distance from the point to the camera's optical center), construct the depth buffer at that pixel. Set tolerance = 0.08, only retain depth values ​​and Points with a difference of no more than 0.08 are removed, i.e., points facing away from the occluded area. The path will be...

[0119] The set of points that has passed the double filtering is denoted as Svisible. The mask is then queried on the corresponding two-dimensional mask image based on its pixel coordinates.

[0120] Code labels are assigned to 3D points to obtain the 3D point set corresponding to each 2D mask. .

[0121] Step 4: Construct the exponential polarization force field map. Create graph nodes for each 2D mask. According to its set of visible points Calculate the three-dimensional centroid Principal component analysis is performed on the point set, and the eigenvector corresponding to the smallest eigenvalue is taken as the fitting principal normal vector. For any two nodes Calculate the union of the sets of points corresponding to all 20 viewpoints. Based on the observation data, the total number of commonly visible viewpoints is defined as... The number of viewpoints on which the two masks are projected into the same two-dimensional region is . Then the fundamental common gravity Calculate the spatial geometric deviation error:

[0122] Normal deviation error ,

[0123] Spatial centroid difference intercept error ;

[0124] Set the exponential penalty coefficient , Calculate the edge weights using the formula:

[0125]

[0126] Step 5: Clustering based on polarization energy threshold. Set the cutting threshold. Then disconnect the edge, in the remaining graph structure Above, a breadth-first search algorithm is used to extract connected components, resulting in node clusters. Each cluster corresponds to a 3D planar instance. A total of 10 planar instances were obtained.

[0127] Step 6: Unsupervised intra-cluster consistency evaluation and output. For each instance cluster... Aggregate all its visible points to obtain a point set. Calculate the centroid and covariance matrix .right Perform eigenvalue decomposition to obtain eigenvalues. Calculate the expansion evaluation ratio Set a threshold. ,like If this is the case, it is determined that there are unseparated heteroplanks within the cluster, triggering graph structure parameter adjustments (e.g., adjusting the graph structure parameters). Increase to 30, Increase the number to 100 and repeat step 5 of clustering. After one iteration, all clusters... All values ​​are less than 0.12, so the evaluation is passed. A globally unique ID is assigned to each instance cluster, and the two-dimensional region corresponding to each viewpoint is assigned to the ID based on the original image position of each mask within the cluster. Finally, the multi-view consistent segmentation results of all 50 images are output.

[0128] Figure 2 The experimental results of the method of this invention on parts of the ABC-NEF dataset are presented, and are divided into two columns: First

[0129] Examples of facet instance masks obtained by segmentation using the pre-trained SAM model in step 1 are listed below (four typical viewpoints are selected).

[0130] Different colors represent different 2D patch instances; the second column shows the final multi-view consistent segmentation result; each patch instance cluster is identified by a unique color, and the node number (such as N0, N1, N2...) is marked on the instance.

[0131] The same number represents the same cluster of three-dimensional facet instances from different perspectives, which intuitively reflects the unified segmentation and correspondence achieved by the present invention across multiple perspectives.

[0132] This embodiment further verifies the superiority of the present invention in scenes with no depth and weak texture. First, this method

[0133] It requires no deep learning hardware and can complete 3D planar segmentation using only pure RGB images, overcoming the limitations of traditional methods on objects.

[0134] The reliance on depth sensors. Secondly, for the textureless surfaces widely present in industrial parts, SDF fusion and...

[0135] Dual visibility filtering ensures stable generation of geometric proxies and accurate attachment of mask labels, avoiding feature matching failure issues.

[0136] Finally, the segmentation results can be directly used for reverse engineering and CAD modeling, providing a high-quality data foundation for industrial automation and digitalization.

[0137] This invention provides a depth-independent, sparse multi-view image planar consistency segmentation method. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A depth-independent, sparse multi-view image planar consistency segmentation method, characterized in that, include The following steps are required: Step 1: Obtain sparse multi-view RGB image sequences and their corresponding camera intrinsic and extrinsic parameters, and extract the set of two-dimensional instance masks for each view, with each two-dimensional mask serving as the initial node primitive; Step 2: Based on the set of two-dimensional instance masks from each viewpoint, generate the global proxy point cloud skeleton and surface normal vectors; Step 3: Through forward projection of 3D point cloud to 2D mask and dual visibility filtering, each 2D mask is strictly mapped to 3D space to obtain the set of visible 3D points corresponding to each mask; Step 4: Using the 2D mask as nodes, and the geometric features of its corresponding 3D point set and the view co-view relationship as edge weights. Re-construct a multidimensional semantic-geometric exponential polarization force field map; Step 5: Segment and cluster the force field map based on the polarization energy threshold to obtain three-dimensional uniform plane instances. cluster; Step 6: Perform unsupervised consistency evaluation on each instance cluster and output a globally consistent partitioning result.

2. The depth-independent sparse multi-view image planar consistency segmentation method according to claim 1, characterized in that, Step 2 specifically generates a global proxy point cloud and global surface normal vectors through 3D symbolic distance field fusion and surface reconstruction technology, including: Step 2-1: Perform a distance transformation on the two-dimensional mask for each viewpoint to generate a two-dimensional symbolic distance field. ; Step 2-2: At the discrete grid point P = (X, Y, Z) in three-dimensional space, fuse the two-dimensional symbolic distance fields from each viewpoint to obtain the three-dimensional symbolic distance field. ; Steps 2-3: Extract the three-dimensional symbolic distance field using the moving cube algorithm. The isosurfaces are used to obtain a smooth envelope mesh; Steps 2-4 involve generating a point cloud based on uniform area sampling from the envelope mesh to obtain a uniformly distributed proxy point cloud. ; Steps 2-5 utilize the three-dimensional symbolic distance field Gradient calculation of the global surface normal vector of each point in the proxy point cloud .

3. The depth-independent sparse multi-view image planar consistency segmentation method according to claim 2, characterized in that, The two-dimensional distance field mentioned in step 2-1 is defined as follows: in, Indicates the first All areas covered by the two-dimensional mask in each viewpoint This represents the pixel coordinates of the region. This indicates the pixel coordinates that do not belong to this region. This means taking all pixels that do not belong to this region and comparing them with the current point. The minimum Euclidean distance between them.

4. The depth-independent sparse multi-view image planar consistency segmentation method according to claim 2, characterized in that, The normal vector mentioned in steps 2-5 is calculated using the following formula: in, For proxy point cloud The i-th point in This represents the global surface normal vector at that point. Represents the three-dimensional symbolic distance field At point The gradient vector at that point, The magnitude of the gradient vector is given by the negative sign, which is used to adjust the gradient direction to point outward from the surface.

5. The depth-independent sparse multi-view image planar consistency segmentation method according to claim 3, characterized in that, Step 3 specifically includes: Step 3-1, global proxy point cloud The 3D points in the image are forward-projected onto each viewpoint through the camera's intrinsic and extrinsic parameter matrices. From the angular image plane, obtain the two-dimensional projected pixel coordinates of each point. The same two-dimensional pixel coordinate When there are multiple corresponding 3D points, a candidate point set is formed. Step 3-2, for two-dimensional pixel coordinates For each projected candidate point in the corresponding candidate point set Calculate the direction of light from the camera Its global normal vector dot product ;like , If the threshold is set, the point is determined to be a cutting surface point and is removed from the candidate set; Step 3-3: For the pixels retained in the two-dimensional pixel coordinates after filtering in step 3-2... For multiple points in the candidate point set, construct a depth buffer at that pixel, retaining only the depth value and the depth buffer. The difference does not exceed the tolerance The point, i.e., removing the back points; Step 3-4, denote the set of visible points obtained in step 3-3 as... Based on its pixel coordinates The mask label is queried on the 2D mask image at the corresponding viewpoint, and the label is assigned to each 3D point in the set of visible points. Finally, the set of all visible points corresponding to the 2D pixel coordinates covered by each mask is summarized to obtain each 2D mask. The corresponding 3D point set .

6. The depth-independent sparse multi-view image planar consistency segmentation method according to claim 5, characterized in that, Step 4, constructing the exponential polarization force field map, includes: Step 4-1, for each two-dimensional mask Create graph nodes And based on its set of visible points Calculate the three-dimensional centroid and fitting principal normal vector ; Step 4-2, calculate any two nodes Basic co-perceived gravity ; Step 4-3, calculate spatial geometric deviation error: normal deviation error Spatial centroid difference intercept error , For nodes The fitted principal normal vector, For nodes The fitted principal normal vector; Step 4-4, define the edge weights of the exponentially polarized force field: in, is the exponential penalty coefficient, with a value range of [0, 100].

7. The depth-independent sparse multi-view image planar consistency segmentation method according to claim 6, characterized in that, The basic co-visible gravity described in step 4-2 ,in To obtain the number of viewpoints for jointly observing the union of two node sets, The number of views in which two masks are projected onto the same two-dimensional region.

8. The depth-independent sparse multi-view image planar consistency segmentation method according to claim 7, characterized in that, The segmentation and clustering described in step 5 include: Step 5-1, Set the cutting threshold ,like If the edge between the two nodes is broken, then the edge between the two nodes will be disconnected. Step 5-2: Perform connected component extraction on the remaining graph structure to obtain several node clusters. Each cluster corresponds to a three-dimensional consistency plane instance.

9. The depth-independent sparse multi-view image planar consistency segmentation method according to claim 8, characterized in that, The unsupervised intra-cluster consensus assessment in step 6 includes: Step 6-1, for each instance cluster Aggregate all its visible points to obtain a point set. Calculate the centroid of the cluster. and covariance matrix ; Step 6-2, for the covariance matrix Perform eigenvalue decomposition to obtain eigenvalues. Calculate the expansion evaluation ratio ;like If the preset threshold is used, then the cluster is determined. The memory is located on an unseparated heterogeneous plane, triggering the adjustment of graph structure parameters; Step 6-3, for each instance cluster that passes the evaluation Assign a globally unique ID, and based on the original image position of each mask within the cluster, assign the ID to the two-dimensional region corresponding to each viewpoint, and output the multi-view consistent segmentation result.

10. The planar consistency segmentation method for sparse multi-view images without depth dependence according to claim 8, wherein the extraction of the two-dimensional instance mask for each viewpoint in step 1 is performed using a pre-trained instance segmentation model.

Images