A 3D shape segmentation method and system based on text driving and attention mechanism

Abstract

The application discloses a kind of 3D shape segmentation method and system based on text driving and attention mechanism, it is related to computer vision technical field, including: obtaining the 3D model to be handled and its corresponding component description text;Based on component description text input to prefix tuning module, obtain global sentence feature and multiple lexical features;Based on 3D model input to grid attention module, obtain multiple patch geometry features;Based on global sentence feature, lexical feature and patch geometry feature input to text-driven fusion module, obtain geometric text fusion feature;Based on geometric text fusion feature input to Laplacian adaptive attention module, obtain Laplace transform feature;Based on Laplace transform feature input to output module, obtain the predicted semantic label of each patch of 3D model, realize 3D model segmentation. Improve the segmentation accuracy of 3D shape.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and more particularly to a 3D shape segmentation method and system based on text driving and attention mechanism. BACKGROUND

[0002] At present, 3D shape segmentation is a basic task in computer graphics, and is widely used in medical image analysis, computer-aided design, robot technology and digital content creation fields. Traditional 3D shape segmentation methods mainly rely on geometric feature descriptors, and distinguish different semantic parts by extracting surface normal vectors, curvature and other geometric information. These methods usually adopt a three-stage process: feature extraction, clustering classification and label projection.

[0003] The traditional 3D shape segmentation method has the following main problems: (1) geometric ambiguity problem: when different parts of the 3D model have similar geometric features, methods that rely only on geometric information are difficult to accurately distinguish, resulting in ambiguous segmentation results, for example, the five fingers of the human hand have similar geometric features, and traditional methods are difficult to accurately segment; (2) local feature limitation: hand-designed geometric descriptors mainly capture local shape features, and are difficult to effectively encode global context information, and the performance decreases when dealing with complex topological structures, noise or incomplete data; (3) lack of semantic understanding: pure geometric methods lack semantic understanding ability and cannot utilize the functionality and semantic meaning of parts to guide the segmentation process; (4) difficulty in integrating multi-scale features: existing methods have challenges in integrating local and global features, and it is difficult to capture discriminative information at different scales.

[0004] Therefore, how to improve the segmentation accuracy of 3D shapes is a problem that those skilled in the art need to solve. SUMMARY

[0005] Therefore, the present application provides a 3D shape segmentation method and system based on text driving and attention mechanism, which improves the segmentation accuracy of 3D shapes.

[0006] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0007] A 3D shape segmentation method based on text driving and attention mechanism, comprising:

[0008] obtaining a 3D model to be processed and its corresponding part description text;

[0009] inputting the part description text into a prefix tuning module based on the part description text, obtaining global sentence features and a plurality of word features;

[0010] inputting the 3D model into a mesh attention module based on the 3D model, obtaining a plurality of face sheet geometric features;

[0011] inputting the global sentence feature, the vocabulary feature, and the patch geometry feature into a text-driven fusion module to obtain a geometry-text fusion feature;

[0012] inputting the geometry-text fusion feature into a Laplacian adaptive attention module to obtain a Laplacian transform feature;

[0013] inputting the Laplacian transform feature into an output module to obtain a predicted semantic label of each patch of the 3D model, and realizing segmentation of the 3D model.

[0014] In one embodiment, the prefix tuning module comprises an embedding layer and a plurality of feature extraction units connected in sequence;

[0015] obtaining a learnable prefix vector;

[0016] inputting the component description text into the embedding layer to obtain a text embedding vector;

[0017] concatenating the text embedding vector and the prefix vector and inputting them into a plurality of feature extraction units in sequence to obtain the global sentence feature and a plurality of vocabulary features.

[0018] In one embodiment, the plurality of feature extraction units are of the same structure, each comprising a first self-attention mechanism layer, a first residual normalization layer, a first feedforward neural network layer, and a second residual normalization layer;

[0019] inputting the input feature into the first self-attention mechanism layer in sequence to obtain a first processed feature;

[0020] inputting the first processed feature and the input feature into the first residual normalization layer to obtain a fusion feature;

[0021] inputting the fusion feature into the first feedforward neural network layer in sequence to obtain a second processed feature;

[0022] inputting the second processed feature and the fusion feature into the second residual normalization layer to obtain an output feature.

[0023] In one embodiment, the mesh attention module comprises a multi-dimensional feature extraction unit, a second self-attention mechanism layer, a second feedforward neural network layer, a third self-attention mechanism layer, and a global average pooling layer;

[0024] inputting the 3D model into the multi-dimensional feature extraction unit to obtain a neighborhood feature matrix;

[0025] The face sheet geometric feature is obtained by sequentially inputting the neighborhood feature matrix into the second self-attention mechanism layer, the second feedforward neural network layer, the third self-attention mechanism layer and the global average pooling layer.

[0026] In one embodiment, the neighborhood feature matrix acquisition method comprises:

[0027] Based on the multi-dimensional feature extraction unit, the local geometric neighborhood composed of each face sheet and its neighbor face sheet sharing a boundary in the 3D model is determined.

[0028] Based on each face sheet in the local geometric neighborhood, a plurality of different types of geometric descriptors are obtained respectively and spliced to obtain a comprehensive geometric vector of the face sheet.

[0029] Based on all the comprehensive geometric vectors in the local geometric neighborhood, the neighborhood feature matrix is formed.

[0030] In one embodiment, the geometric descriptor comprises: average geodesic distance, Gaussian curvature, shape diameter function, scale-invariant heat kernel feature and wave kernel feature.

[0031] The average geodesic distance measures the average shortest path of the face sheet to other face sheets on the model surface, reflecting the global position information.

[0032] The Gaussian curvature describes the local surface bending degree and distinguishes different geometric shapes.

[0033] The shape diameter function measures the local thickness information.

[0034] The scale-invariant heat kernel feature provides multi-scale geometric information.

[0035] The wave kernel feature captures different frequency geometric patterns and enhances the shape discrimination ability.

[0036] In one embodiment, the text-driven fusion module comprises: a first multi-layer perception, a first fusion layer, a splicing layer, a second multi-layer perception, a third multi-layer perception, a second fusion layer and a third fusion layer.

[0037] Based on all the word features and the face sheet geometric features, the first multi-layer perception is inputted to obtain the correlation weight of each word feature and the face sheet geometric feature.

[0038] Based on the correlation weight and all the word features, the first fusion layer is inputted for weighted fusion to obtain a dynamic word feature.

[0039] Based on the dynamic word feature, the global sentence feature and the face sheet geometric feature, the splicing layer is inputted to obtain a spliced text feature.

[0040] inputting the spliced text features into the second multi-layer perception and the third multi-layer perception respectively based on the spliced text features, to obtain scaling parameters and offset parameters correspondingly;

[0041] inputting the patch geometry features and the scaling parameters into the second fusion layer, to obtain preliminary fusion features;

[0042] inputting the preliminary fusion features and the offset parameters into the third fusion layer, to obtain the geometry-text fusion features.

[0043] In an embodiment, the Laplace adaptive attention module comprises a flattening layer, a first calculation unit, a second calculation unit, a Laplace calculation unit, and a fourth fusion layer;

[0044] inputting the geometry-text fusion features into the flattening layer for flattening operation, to obtain a geometry-text one-dimensional vector;

[0045] inputting the geometry-text one-dimensional vector into the first calculation unit and the second calculation unit respectively, to obtain vector mean and vector variance correspondingly;

[0046] inputting the vector mean and the vector variance into the Laplace calculation unit, to obtain a Laplace distribution;

[0047] inputting the Laplace distribution and the geometry-text fusion features into the fourth fusion layer, to obtain the Laplace transform features.

[0048] In an embodiment, the output module comprises a third feedforward neural network layer and an activation layer;

[0049] inputting the Laplace transform features into the third feedforward neural network layer and the activation layer in sequence, to obtain the predicted semantic label of each patch in the 3D model to be processed, and complete the segmentation of the 3D model to be processed.

[0050] A 3D shape segmentation system based on text driving and attention mechanism, comprising a data acquisition module, a text feature extraction module, a geometry feature extraction module, a feature fusion module, a Laplace transform module, and a segmentation result output module;

[0051] The data acquisition module is configured to acquire a 3D model to be processed and a corresponding component description text thereof.

[0052] The text feature extraction module is configured to input the component description text into a prefix tuning module, to obtain global sentence features and a plurality of word features.

[0053] The geometry feature extraction module is configured to input the 3D model into a mesh attention module to obtain a plurality of patch geometry features;

[0054] The feature fusion module is configured to input the global sentence feature, the vocabulary feature, and the patch geometry feature into a text-driven fusion module to obtain a geometry-text fusion feature;

[0055] The Laplace transform module is configured to input the geometry-text fusion feature into a Laplace adaptive attention module to obtain a Laplace transform feature.

[0056] The segmentation result output module is configured to input the Laplace transform feature into an output module to obtain a predicted semantic label of each patch of the 3D model, thereby realizing segmentation of the 3D model.

[0057] Compared with the prior art, the 3D shape segmentation method and system based on text driving and attention mechanism provided by the present application has the following beneficial effects:

[0058] 1. Fundamental solution of geometric ambiguity: The present application completely solves the geometric ambiguity problem faced by traditional methods by introducing a text semantic dimension. Specifically, when the five fingers of a human model or the four legs of a chair have extremely similar geometric features, traditional methods cannot effectively distinguish them completely relying on geometric information. The present application provides a unique semantic identifier for each component through precise text descriptions such as "thumb of a human body" and "index finger of a human body", thereby fundamentally eliminating the segmentation ambiguity caused by geometric similarity.

[0059] 2. Innovative solution to multi-granularity information mismatch: Traditional 3D segmentation faces the granular gap between component-level labeling and patch-level processing. The present application realizes cross-granularity information bridging through a prefix optimization mechanism. The learnable prefix vector acts as a semantic refiner, which can expand coarse-grained component semantics into fine-grained patch-level guidance information. This design enables a "hand" label to provide differentiated semantic guidance for thousands of arm patches, realizing a smooth transition from component semantics to patch geometry.

[0060] 3. Systematic reinforcement of missing spatial context information: The isolated processing mode of traditional geometric descriptors ignores the important spatial correlation between patches. The present application systematically solves this problem through a neighborhood modeling mechanism. The final representation of each patch not only contains its own geometric information, but also fuses the context information of neighboring patches, forming a spatially aware geometric representation suitable for handling ambiguous boundary segmentation tasks and enabling more accurate judgments based on neighborhood consistency.

[0061] 4. Technological breakthrough in heterogeneous feature fusion: Geometric features and text features belong to completely different modalities, and direct fusion often yields poor results; this invention achieves deep fusion of heterogeneous features through an adaptive modulation mechanism. Text features are not simply spliced ​​with geometric features, but rather used as a modulation signal to dynamically control the activation mode of geometric features, thereby realizing a deeper guidance of semantics for geometric understanding.

[0062] 5. Theoretical innovation in feature distribution modeling: Traditional attention mechanisms are based on the Gaussian distribution assumption, but geometric features often exhibit heavy-tailed and sparse characteristics; the adaptive attention module based on the Laplace distribution in this invention more accurately models the true distribution characteristics of geometric features, which can better identify and highlight discriminative features while suppressing noise interference.

[0063] 6. Forward-looking value of technical scalability: The core idea and technical architecture of this invention have good scalability and can be extended to other 3D understanding tasks such as 3D object detection, 3D semantic segmentation, and 3D instance segmentation, which has broad application prospects and technical extension value. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0065] Figure 1 The flowchart of a 3D shape segmentation method based on text-driven and attention mechanisms provided by the present invention is shown.

[0066] Figure 2 This is a schematic diagram of the prefix tuning module structure provided by the present invention.

[0067] Figure 3 This is a schematic diagram of the text-driven fusion module structure provided by the present invention.

[0068] Figure 4 This is a schematic diagram of the Laplace adaptive attention module structure provided by the present invention. Detailed Implementation

[0069] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] Example 1

[0071] like Figure 1 As shown, this embodiment of the invention discloses a 3D shape segmentation method based on text-driven and attention mechanisms, including the following steps, which are numbered S1 to S6 for ease of description. These numbers are not used to limit the sequential relationship between the various steps of this invention:

[0072] S1 retrieves the 3D model to be processed and its corresponding component description text.

[0073] Traditional methods rely solely on geometric features, which can easily lead to confusion when encountering geometrically similar components such as fingers or chair legs. This invention introduces component description text such as "finger of a human body" and "leg of a chair" to provide semantic identifiers for each component, thereby adding semantic discrimination capabilities on top of geometric features.

[0074] Furthermore, the component description text is a description of the components of the 3D model to be processed. For example, for a 3D model of a teddy bear to be processed, its components include ears, head, hands, etc. This invention uses simple text to describe each component.

[0075] S2 takes the component description text as input to the prefix tuning module and obtains global sentence features and multiple lexical features.

[0076] Furthermore, such as Figure 2 As shown, the prefix tuning module includes an embedding layer and multiple sequentially connected feature extraction units;

[0077] Obtain learnable prefix vectors;

[0078] The component description text is input into the embedding layer to obtain the text embedding vector;

[0079] The text embedding vector and prefix vector are concatenated and then sequentially input into multiple feature extraction units to obtain global sentence features and multiple lexical features.

[0080] Furthermore, the multiple feature extraction units have the same structure, each including: a first self-attention mechanism layer, a first residual normalization layer, a first feedforward neural network layer, and a second residual normalization layer;

[0081] The input features are sequentially fed into the first self-attention mechanism layer to obtain the first processed features;

[0082] Based on the first processed features and the input features, the first residual normalization layer is fed into the first residual normalization layer for sequential fusion and normalization operations to obtain the fused features;

[0083] Based on the fused features, they are sequentially input into the first feedforward neural network layer to obtain the second processed features;

[0084] The second processed feature and the fused feature are input into the second residual normalization layer for sequential fusion and normalization operations to obtain the output feature.

[0085] Furthermore, the input features first pass through an attention layer, where the self-attention mechanism comes into play, enabling the model to learn the rich information contained in the text. Subsequently, they pass through a residual and normalization layer, which avoids gradient vanishing through residual connections and normalizes the data distribution through a normalization layer, allowing the model to converge quickly. Then, they pass through a feedforward neural network (FFN) layer, which enhances the model's learning ability through a fully connected layer with a large number of parameters, followed by the same residual and normalization layer.

[0086] Furthermore, in this embodiment, multiple feature extraction units together form the RoBERTa model. RoBERTa is used as a text encoder, keeping its pre-trained parameters unchanged to ensure that the learned language knowledge is not destroyed.

[0087] The text embedding vector and prefix vector are concatenated and input into the frozen RoBERTa model. The model's pre-training parameters remain unchanged to ensure that existing language knowledge is preserved. Through the processing of multiple self-attention layers, the model can model the complex dependencies between words and between prefixes and words. At the same time, the prefix vector learning is associated with different geometric transformation patterns.

[0088] Furthermore, the present invention employs a learnable prefix vector mechanism: a set of learnable special label vectors are added before each text input. These vectors learn during training how to distinguish different facets of the same component in a fine-grained manner. In addition, to avoid inconsistencies in positional information between the training and inference stages, a fixed positional index is assigned to each text sequence while keeping the paragraph embedding unchanged.

[0089] Furthermore, the description text of each component, such as "ear of a teddy bear", is concatenated with a learnable prefix vector and input into the RoBERTa model. The prefix vector is designed as a continuous vector group with adjustable length. In the final output layer, global sentence representations are extracted from special markers as global sentence features, and word-level representations are extracted from each word position as lexical features, providing rich semantic information for subsequent modules.

[0090] S3 inputs a 3D model into the mesh attention module to obtain multiple patch geometric features.

[0091] Traditional geometric descriptors process each face independently, ignoring the spatial relationships between adjacent faces. In reality, the semantic label of a face is often closely related to its neighbors. Based on this, the present invention sets up a mesh attention module to obtain the face geometric features of the 3D model.

[0092] Furthermore, the grid attention module includes: a multi-dimensional feature extraction unit, a second self-attention mechanism layer, a second feedforward neural network layer, a third self-attention mechanism layer, and a global average pooling layer;

[0093] The neighborhood feature matrix is ​​obtained by inputting the 3D model into the multidimensional feature extraction unit.

[0094] The neighborhood feature matrix is ​​sequentially input into the second self-attention mechanism layer, the second feedforward neural network layer, the third self-attention mechanism layer, and the global average pooling layer to obtain the patch geometric features that fuse neighborhood context information.

[0095] Furthermore, the method for obtaining the neighborhood feature matrix is ​​as follows:

[0096] Based on the multi-dimensional feature extraction unit, each facet in the 3D model and its neighboring faces sharing the boundary together form a local geometric neighborhood;

[0097] Based on each patch in the local geometric neighborhood, multiple different types of geometric descriptors are obtained and concatenated to obtain the comprehensive geometric vector of the patch;

[0098] The neighborhood feature matrix is ​​formed by combining all the comprehensive geometric vectors in the local geometric neighborhood.

[0099] Furthermore, based on each facet in the 3D model, three adjacent faces that share an edge with it are selected to form a local geometric neighborhood. This one-ring neighborhood design can capture important local context information while maintaining computational efficiency.

[0100] Furthermore, the geometric descriptors include: mean geodesic distance, Gaussian curvature, shape diameter function, scale-invariant thermal kernel features, and wave kernel features;

[0101] Average Geodesic Distance (AGD): Measures the average shortest path from a patch to other patches on the model surface, reflecting global location information;

[0102] Gaussian Curvature (GC): Describes the degree of curvature of a local surface and distinguishes different geometric shapes such as convex, concave, and saddle-shaped.

[0103] Shape Diameter Function (SDF): Measures local thickness information to help identify thin protrusions (such as fingers, antennas, etc.).

[0104] Scale-Invariant Heat Kernel Signature (SIHKS): Provides multi-scale geometric information and is robust to shape deformation;

[0105] Wave Kernel Signature (WKS): Captures geometric patterns at different frequencies, enhancing shape discrimination capabilities.

[0106] Furthermore, for the face f on the mesh i Its average geodetic distance is:

[0107] ;

[0108] Where N represents the total number of faces in the mesh, d g (f i , f j ) represents the face f i to f j The geodesic distance (calculated using Dijkstra's or Fast Marching algorithm);

[0109] Gaussian curvature K(v) i ):

[0110] ;

[0111] Among them, v i θ represents a vertex on a triangular mesh. j Represents vertex v i Surrounding angles, A mixed Represents vertex v i The area of ​​the mixed Voronoi region;

[0112] Shape diameter function SDF(p):

[0113] ;

[0114] Where p represents a point on the grid surface, r∩M represents the intersection of ray r and grid M, and C(p) represents the set of conical rays originating from point p. The median is taken to obtain a stable diameter estimate.

[0115] Scale-invariant thermonuclear characteristics:

[0116] SIHKS(x,t) is a scale-invariant form based on thermonuclear cores.

[0117] SIHKS(x,t) = t·HKS(x,t);

[0118] The thermonuclear characteristic HKS(x,t) is defined as follows:

[0119] ;

[0120] Among them, a i and b iThese are the eigenvalues ​​and eigenfunctions of the Laplace-Beltrami operator, respectively, where t represents the time parameter;

[0121] Wave kernel feature calculation method:

[0122] Based on the Schrödinger equation of quantum mechanics:

[0123] ;

[0124] The energy window function is as follows:

[0125] f E (λ i ) represents a Gaussian window function with energy E, λ i and Let represent the eigenvalues ​​and eigenfunctions of the Laplace operator, respectively, and σ represent the width of the control energy window.

[0126] Furthermore, the geometric descriptors characterize the geometric properties of the image from different perspectives: the average geodesic distance reflects the global position, the Gaussian curvature describes the local shape, the shape diameter function reveals the thickness information, and the scale-invariant heat kernel feature and wave kernel feature provide multi-scale features; the five descriptors AGD, GC, SDF, SIHKS and WKS have dimensions of 1D, 1D, 1D, 19D (multi-scale time parameters) and 100D (multiple energy levels), respectively, and are finally spliced ​​together to form a 122-dimensional comprehensive geometric vector.

[0127] Furthermore, each facet in the 3D model serves as a central facet. A 4×122 neighborhood feature matrix is ​​formed based on the combined geometric vectors of the central facet and its three neighboring faces. This neighborhood feature matrix is ​​input to the second self-attention mechanism layer, where a linear transformation generates three sets of vectors: query, key, and value. The relevance scores between the central facet and each neighboring facet are calculated, and softmax normalization is used to obtain attention weights, reflecting the influence of different neighboring faces on the central facet. Weighted aggregation of all neighboring faces based on these attention weights is then input to the second feedforward neural network layer to obtain deeper geometric features. These deeper geometric features are processed by the second self-attention mechanism layer to obtain enhanced features, which are then processed through global average pooling to obtain facet geometric features that incorporate neighborhood contextual information. This design ensures that each facet's features not only contain its own geometric information but also incorporate the contextual information of its neighbors.

[0128] S4 inputs global sentence features, lexical features, and patch geometric features into the text-driven fusion module to obtain geometric text fusion features.

[0129] Furthermore, such as Figure 3As shown, the text-driven fusion module includes: a first multilayer perceptron, a first fusion layer, a stitching layer, a second multilayer perceptron, a third multilayer perceptron, a second fusion layer, and a third fusion layer;

[0130] Based on all the lexical features and patch geometric features, the first multilayer perceptron is input to obtain the correlation weight between each lexical feature and the patch geometric feature.

[0131] Based on the relevance weights and all lexical features, the first fusion layer is used for weighted fusion to obtain dynamic word features;

[0132] The concatenated text features are obtained by inputting dynamic word features, global sentence features, and patch geometric features into the concatenation layer.

[0133] Based on the concatenated text features, the inputs are respectively fed into the second and third multilayer perceptrons to obtain the scaling and offset parameters.

[0134] Based on the geometric features of the patches and the scaling parameters, the input is given to the second fusion layer to obtain the preliminary fusion features;

[0135] Based on the initial fusion features and offset parameters, the geometric text fusion features are obtained by inputting them into the third fusion layer.

[0136] Furthermore, the first multilayer perceptron measures the semantic relevance between each word feature and the geometric features of the facet by calculating cosine similarity, which serves as the relevance weight. For example, when processing the facet of the finger region, the relevance weight of the word "finger" is significantly higher than that of "head".

[0137] Based on the relevance weights and all lexical features, the first fusion layer is used for weighted fusion to obtain dynamic word features for the current geometric context. This design allows the same text description to produce different semantic emphases when processing different geometric regions.

[0138] Furthermore, the scaling parameter controls the activation intensity of the geometric features, while the offset parameter adjusts the baseline level of the geometric features. The patch geometric features are first batch normalized to eliminate distributional differences (corresponding to the Hadamard product operation between the patch geometric features and the scaling parameter), and then modulated through a text-controlled affine transformation (corresponding to the cumulative summation operation between the Hadamard product result and the offset parameter). This design allows textual information to directly influence the representation of geometric features, enabling semantic guidance for geometric understanding.

[0139] Furthermore, the fusion module can automatically adjust the fusion weights based on the specific geometric-semantic pairing situation; when geometric features are sufficient for discrimination, it reduces the influence of text features; when geometric features are ambiguous, it enhances the guiding role of text features.

[0140] S5 inputs geometric text fusion features into the Laplacian adaptive attention module to obtain Laplacian transform features.

[0141] Traditional attention mechanisms are based on the Gaussian distribution assumption, but geometric features often exhibit heavy-tailed distribution and sparse activation patterns, which are more consistent with the Laplace distribution. Based on this, this invention designs a Laplace adaptive attention module.

[0142] Furthermore, such as Figure 4 As shown, the Laplacian adaptive attention module includes: a flattening layer, a first computational unit, a second computational unit, a Laplacian computational unit, and a fourth fusion layer;

[0143] Based on the geometric text fusion feature input to the flattening layer, a one-dimensional vector of geometric text is obtained;

[0144] The one-dimensional vector of the geometric text is input into the first and second computing units respectively, and the vector mean and vector variance are obtained accordingly, which are used to capture the distribution characteristics of the current one-dimensional vector of the geometric text.

[0145] The Laplace distribution is obtained by inputting the vector mean and vector variance into the Laplace calculation unit.

[0146] Based on the Laplace distribution and geometric text fusion features, the input is given to the fourth fusion layer to obtain the Laplace transform features.

[0147] Furthermore, the Laplace probability density function is obtained based on the Laplace distribution, and the Laplace attention weights are calculated using the Laplace probability density function. The geometric text fusion features are combined with the Laplace attention weights by element-wise multiplication to finally obtain the Laplace transform features. This design can automatically highlight the feature dimensions with strong discriminative power, while suppressing noise and irrelevant information, thus improving the quality of feature representation.

[0148] Compared to the Gaussian distribution, the Laplace distribution has sharper peaks and heavier tails, making it more suitable for modeling sparse activation patterns commonly found in geometric features.

[0149] Furthermore, the Laplace probability density function is applied in several ways: it uses absolute value deviation instead of squared deviation to better handle outliers and noise; it controls the width of the distribution through a learnable scaling parameter to automatically adapt to the distribution characteristics of different features; and the weight distribution generated by the exponential decay function has stronger sparsity, highlighting important features.

[0150] S6 uses Laplace transform features as input to the output module to obtain the predicted semantic labels for each facet of the 3D model, thus achieving 3D model segmentation.

[0151] Furthermore, the output module includes: a third feedforward neural network layer and an activation layer;

[0152] Based on the Laplace transform features, the data is sequentially input into the third feedforward neural network layer and the activation layer to obtain the predicted semantic labels for each facet in the 3D model to be processed, thus completing the segmentation of the 3D model to be processed.

[0153] The 3D shape segmentation method of the present invention can be widely applied in the following fields:

[0154] 1. Medical image analysis: Organ segmentation of medical scan data to assist in disease diagnosis and surgical planning;

[0155] 2. Computer-aided design: Achieving intelligent model component recognition and editing within CAD software;

[0156] 3. Robot vision: Helps robots understand and manipulate different parts of 3D objects;

[0157] 4. Virtual Reality and Game Development: Achieving detailed modeling and animation of 3D characters and scenes;

[0158] 5. Autonomous driving: Semantic segmentation of point cloud data to identify targets such as vehicles and pedestrians;

[0159] 6. Industrial quality inspection: Defect detection and component identification for industrial products;

[0160] 7. Archaeological and cultural relic research: Detailed analysis and restoration guidance of 3D models of cultural relics.

[0161] Example 2

[0162] Verification of the superiority of the method of this invention:

[0163] A comprehensive improvement in segmentation accuracy: Table 1 shows the segmentation accuracy comparison between this invention and other supervised methods on the authoritative PSB dataset.

[0164]

[0165] As shown in Table 1, the present invention achieves an average segmentation accuracy of 96.27%, which is 1.69 percentage points higher than the best baseline method, PartSLIP, which achieves 94.58%. This represents a significant performance leap. Particularly noteworthy is the improvement in the geometrically ambiguous "human" category, where the accuracy increased from 92.78% to 94.38%, a gain of 1.6 percentage points, fully demonstrating the significant advantages of text-driven methods in resolving geometric ambiguity.

[0166] Cross-dataset generalization capability verification: This invention achieved state-of-the-art performance on four different public datasets: 96.27% for the PSB dataset, 95.50% for the small-scale COSEG dataset, 98.90% for the large-scale COSEG dataset, and 91.2% for the ShapeNetCore dataset. This consistent performance leadership demonstrates the strong generalization capability of the method of this invention, which is not limited to specific data distributions or object categories.

[0167] Practical guarantee of computational efficiency: Despite the introduction of multimodal processing and complex attention mechanisms, the system maintains good computational efficiency: single model segmentation takes approximately 10 seconds, and single-class training takes approximately 15 minutes, fully meeting the efficiency requirements of practical applications. This efficiency advantage stems from the carefully designed network architecture and optimized computational process.

[0168] This invention introduces a text-driven multimodal learning paradigm for the first time in the field of 3D shape segmentation, pioneering a completely new research direction. This cross-modal thinking breaks through the limitations of traditional methods that are confined to a single geometric modality, providing a new technical path for 3D understanding tasks.

[0169] The four-module collaborative framework design is not a simple stacking of technologies, but a systematic technological innovation aimed at addressing the core challenges of 3D segmentation tasks. Each module has a clear technical goal and innovation point, and the synergistic effect between the modules produces an overall performance improvement that is greater than the sum of its parts.

[0170] Example 3

[0171] Based on the same inventive concept, the present invention also provides a 3D shape segmentation system based on text-driven and attention mechanisms, including: a data acquisition module, a text feature extraction module, a geometric feature extraction module, a feature fusion module, a Laplace transform module, and a segmentation result output module;

[0172] The data acquisition module is used to acquire the 3D model to be processed and its corresponding component description text;

[0173] The text feature extraction module is used to input component description text into the prefix optimization module to obtain global sentence features and multiple lexical features;

[0174] The geometric feature extraction module is used to obtain multiple patch geometric features based on the 3D model input to the mesh attention module;

[0175] The feature fusion module is used to input global sentence features, lexical features and patch geometric features into the text-driven fusion module to obtain geometric text fusion features;

[0176] The Laplacian Transform module is used to input geometric text fusion features into the Laplacian Adaptive Attention module to obtain Laplacian Transform features.

[0177] The segmentation result output module is used to input the Laplacian transform features into the output module to obtain the predicted semantic labels for each facet of the 3D model, thereby achieving 3D model segmentation.

[0178] Furthermore, in this embodiment, the functional implementation process of each module corresponds one-to-one with the above-described method, and will not be described in detail here.

[0179] Example 4

[0180] Based on the same inventive concept, the present invention also provides a computer device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0181] Memory, used to store computer programs;

[0182] When the processor executes a program stored in memory, it is able to implement a 3D shape segmentation method based on text-driven and attention mechanisms, as shown in Example 1.

[0183] The electronic device may include a processor, a communications interface, memory, and a communication bus, wherein the processor, communications interface, and memory communicate with each other via the communication bus. The processor can invoke logical instructions in the memory to execute a 3D shape segmentation method based on text-driven and attention mechanisms as described in Embodiment 1.

[0184] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention.

[0185] The aforementioned storage media include: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media that can store program code.

[0186] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0187] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for 3D shape segmentation based on text-driven and attention mechanism, characterized in that, The method comprises the following steps: obtaining a 3D model to be processed and its corresponding component description text; inputting the component description text into a prefix optimization module to obtain global sentence features and multiple word features; inputting the 3D model into a mesh attention module to obtain multiple patch geometry features; the mesh attention module comprises a multi-dimensional feature extraction unit, a second self-attention mechanism layer, a second feedforward neural network layer, a third self-attention mechanism layer, and a global average pooling layer; inputting the 3D model into the multi-dimensional feature extraction unit to obtain a neighborhood feature matrix; inputting the neighborhood feature matrix into the second self-attention mechanism layer, the second feedforward neural network layer, the third self-attention mechanism layer, and the global average pooling layer in sequence to obtain the patch geometry features fused with neighborhood context information; inputting the global sentence features, the word features, and the patch geometry features into a text-driven fusion module to obtain geometry-text fusion features; the text-driven fusion module comprises a first multi-layer perceptron, a first fusion layer, a concatenation layer, a second multi-layer perceptron, a third multi-layer perceptron, a second fusion layer, and a third fusion layer; inputting all the word features and the patch geometry features into the first multi-layer perceptron to obtain the relevance weight of each word feature and the patch geometry feature; inputting the relevance weight and all the word features into the first fusion layer for weighted fusion to obtain dynamic word features; inputting the dynamic word features, the global sentence features, and the patch geometry features into the concatenation layer to obtain concatenated text features; inputting the concatenated text features into the second multi-layer perceptron and the third multi-layer perceptron respectively to obtain scaling parameters and offset parameters correspondingly; inputting the patch geometry features and the scaling parameters into the second fusion layer to obtain preliminary fusion features; inputting the preliminary fusion features and the offset parameters into the third fusion layer to obtain the geometry-text fusion features; inputting the geometry-text fusion features into a Laplacian adaptive attention module to obtain Laplacian transform features; inputting the Laplacian transform features into an output module to obtain the predicted semantic label of each patch of the 3D model, thereby realizing the segmentation of the 3D model.

2. The 3D shape segmentation method based on text-driven and attention mechanism according to claim 1, characterized in that, The prefix optimization module comprises an embedding layer and multiple feature extraction units connected in sequence; obtaining a learnable prefix vector; inputting the component description text into the embedding layer to obtain a text embedding vector; concatenating the text embedding vector and the prefix vector and then inputting them into multiple feature extraction units in sequence to obtain the global sentence features and multiple word features.

3. The 3D shape segmentation method based on text-driven and attention mechanism according to claim 2, characterized in that, The multiple feature extraction units are structurally identical and each comprises a first self-attention mechanism layer, a first residual normalization layer, a first feedforward neural network layer, and a second residual normalization layer; inputting a feature into the first self-attention mechanism layer to obtain a first processed feature; inputting the first processed feature and the input feature into the first residual normalization layer to obtain a fusion feature; The second processing feature is obtained by inputting the fusion feature into the first feedforward neural network layer in sequence. The output feature is obtained by inputting the second processing feature and the fusion feature into the second residual normalization layer.

4. The 3D shape segmentation method based on text-driven and attention mechanism according to claim 1, characterized in that, The neighborhood feature matrix acquisition method is: Based on the multi-dimensional feature extraction unit, it is determined that each face sheet in the 3D model and its neighbor face sheet sharing the boundary together form a local geometric neighborhood; Based on each of the face sheets in the local geometric neighborhood, a plurality of different types of geometric descriptors are obtained and spliced to obtain a comprehensive geometric vector of the face sheet; Based on all the comprehensive geometric vectors in the local geometric neighborhood, the neighborhood feature matrix is formed.

5. The text-driven and attention mechanism based 3D shape segmentation method of claim 4, wherein, The geometric descriptors include: average geodesic distance, Gaussian curvature, shape diameter function, scale-invariant heat kernel feature, and wave kernel feature; The average geodesic distance measures the average shortest path of the face sheet to other face sheets on the model surface, reflecting the global position information; The Gaussian curvature describes the local surface curvature and distinguishes different geometric shapes; The shape diameter function measures the local thickness information; The scale-invariant heat kernel feature provides multi-scale geometric information; The wave kernel feature captures different frequency geometric patterns and enhances shape discrimination ability.

6. The 3D shape segmentation method based on text-driven and attention mechanism according to claim 1, characterized in that, The Laplace adaptive attention module includes a flattening layer, a first calculation unit, a second calculation unit, a Laplace calculation unit, and a fourth fusion layer; The geometric text one-dimensional vector is obtained by inputting the geometric text fusion feature into the flattening layer for flattening operation; The vector mean and vector variance are obtained by inputting the geometric text one-dimensional vector into the first calculation unit and the second calculation unit, respectively; The Laplace distribution is obtained by inputting the vector mean and the vector variance into the Laplace calculation unit; The Laplace transform feature is obtained by inputting the Laplace distribution and the geometric text fusion feature into the fourth fusion layer.

7. The text-driven and attention mechanism based 3D shape segmentation method of claim 1, wherein, The output module includes a third feedforward neural network layer and an activation layer; The predicted semantic label of each face sheet in the 3D model to be processed is obtained by inputting the Laplace transform feature into the third feedforward neural network layer and the activation layer in sequence, and the segmentation of the 3D model to be processed is completed.

8. A text-driven and attention mechanism based 3D shape segmentation system for performing a text-driven and attention mechanism based 3D shape segmentation method according to any one of claims 1-7, characterized in that, It includes: Data acquisition module, text feature extraction module, geometric feature extraction module, feature fusion module, Laplace transform module, and segmentation result output module; The data acquisition module is used to acquire a 3D model to be processed and its corresponding component description text; The text feature extraction module is used to input the component description text into a prefix optimization module to obtain global sentence features and a plurality of word features; The geometric feature extraction module is used to input the 3D model into a grid attention module to obtain a plurality of face sheet geometric features; The feature fusion module is used to input the global sentence features, the word features, and the face sheet geometric features into a text-driven fusion module to obtain geometric text fusion features; The Laplace transform module is configured to input the geometric text fusion feature into a Laplace adaptive attention module to obtain a Laplace transform feature. The segmentation result output module is configured to input the Laplace transform feature into an output module to obtain a predicted semantic label of each face sheet of the 3D model, and realize segmentation of the 3D model.

Images