A three-dimensional part assembly method and system based on hierarchical text prior

Abstract

The application discloses a three-dimensional part assembly method and system based on hierarchical text prior, and feature coding is performed on multiple unordered three-dimensional part point clouds of an object to be assembled, global geometric features are extracted, and a hierarchical text library is constructed; a semantic affinity soft grouping mechanism and a set context attention mechanism are used to distinguish the functions and roles of three-dimensional parts, and a topologically consistent global rough posture is obtained; local contact surface geometric features are extracted by using the global rough posture, functional text semantics are injected into the contact surface geometric features and are aggregated; the aggregated contact surface geometric features are spliced and fused with global context features, and an accurate six-degree-of-freedom matching posture of the three-dimensional part is inferred. The application can introduce semantic prior information capable of representing part identity, functional attributes and spatial context, and make the semantic prior simultaneously act on global topology initialization and local contact alignment process, thereby greatly reducing assembly ambiguity caused by geometric similar parts and improving local connection precision.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional part assembly technology, specifically to a three-dimensional part assembly method, system, electronic device, and medium based on hierarchical text priors. Background Technology

[0002] 3D parts assembly is a fundamental research area in 3D computer vision, intelligent manufacturing, industrial design automation, and robotics. Its core task is to automatically predict the relative positions and assembly postures of a set of disordered components within the overall structure, thereby reconstructing a complete 3D object with a sound structure and reasonable connections. This technology has wide applications in furniture assembly, mechanical parts assembly, digital twins, assembly process planning, and autonomous robot operation.

[0003] Traditional 3D part assembly methods primarily rely on geometric matching and optimization strategies. Early studies typically employed point cloud registration algorithms such as Iterative Closest Point (ICP) and Go-ICP, estimating the assembly pose of each part by minimizing the geometric distance between the part's point cloud and the target shape. However, since parts do not precisely match local regions of the target shape, relying solely on spatial coordinates or geometric features to find correspondences presents significant challenges. Furthermore, optimization-based methods are prone to getting trapped in local optima; while the predicted assembly results may match the target shape in contour, they often lack semantic plausibility.

[0004] With the development of 3D point cloud processing technology and deep learning methods, learning-based methods for 3D part assembly tasks have gradually become an important research direction in this field. Existing deep learning methods typically rely on the geometric information of parts as the main basis, and predict the topological relationships between parts and their final assembly posture by extracting features and modeling relationships from point clouds, meshes, or part graph structures.

[0005] This type of method can achieve certain results when dealing with assembly objects with obvious shape differences and relatively simple structural relationships. However, in actual assembly objects, multiple parts often have strong geometric similarities, especially in furniture and industrial supports. Different parts may have similar geometric features such as length and cross-section, making it difficult to accurately distinguish their true functional roles based solely on geometric shapes.

[0006] For example, in assembly objects such as chairs, tables, and support frames, parts such as chair legs, crossbars, columns, connecting rods, and frames are often very similar in local shape, but their functions in the overall structure are not the same, and their corresponding assembly positions, connection objects, and directional constraints also differ significantly. Existing methods, lacking explicit modeling of part identities, functional attributes, and structural context, are prone to incorrectly matching geometrically similar parts to similar positions during the global assembly reasoning stage, leading to problems such as part confusion, topological errors, and unreasonable overall structures.

[0007] Therefore, there is an urgent need for a completely new three-dimensional parts assembly technology solution. Summary of the Invention

[0008] To achieve the above objectives, this invention discloses a three-dimensional part assembly method based on hierarchical text priors, comprising the following steps: The feature extraction step involves encoding the features of each part point cloud through multiple disordered 3D part point clouds of the object to be assembled, extracting the global geometric features of the disordered 3D parts, and constructing a hierarchical text library. The hierarchical text library covers multi-dimensional functional text features including part identity information, functional attribute information, and spatial context information. The semantically aware global topology initialization step distinguishes the functional roles of the three-dimensional parts through a semantic affinity soft grouping mechanism and a set context attention mechanism, thereby obtaining a topologically consistent global coarse pose. The text-guided fine contact alignment step uses the global coarse pose to extract local contact surface geometric features, and uses a word-level cross-modal cross attention mechanism to inject the functional text semantics in the functional text features into the contact surface geometric features, and performs aggregation processing to eliminate local physical conflicts caused by geometric ambiguity. The attitude regression step combines the aggregated contact surface geometric features with global context features, performs global interaction, and infers the precise six-degree-of-freedom mating attitude of the 3D part.

[0009] Preferably, the part identity information in the functional text features is used to describe the part's category name and geometric details, the functional attribute information is used to describe the part's support connection relationship, and the spatial context information is used to describe the part's relative positional relationship in the overall assembly structure; The text feature vector for each part is obtained by encoding the functional text features separately and averaging the encoding results to eliminate single expression bias.

[0010] Preferably, the semantic affinity soft grouping mechanism specifically comprises: A linear transformation is performed on the feature encoding of each part point cloud by the OpenShape encoder to obtain the adaptation features. Then, the cosine similarity between the adaptation features corresponding to each part point cloud and each text feature sequence is calculated. All the cosine similarities are then normalized to generate a semantic affinity probability distribution.

[0011] Preferably, the set context attention mechanism specifically includes: The semantic affinity probability distribution is concatenated with the global geometric features to obtain the semantic enhancement features of each part. The semantic enhancement features of each part are input into the attention network to model the structural co-occurrence relationship and semantic compatibility relationship among all parts, thereby obtaining the context-aware features of each part. The context-aware features are input into the multilayer perceptron regression network to obtain the three-dimensional translation vector and rotation quaternion of each part, forming the global coarse pose.

[0012] Preferably, the step of extracting local contact surface geometric features using the global coarse pose specifically involves: Based on the coarse translation vector and rotation quaternion in the global coarse pose, the original point cloud of each part is transformed into the coarse assembly coordinate system to obtain the coarse assembly point cloud. Calculate the adjacent distance from each point on each part to the point clouds of other parts, and form a set of contact points by selecting each point whose adjacent distance is less than a preset threshold as candidate contact points. Several candidate contact points are selected from the set of contact points to form a local contact point set, and the local contact surface geometric features are extracted from the local contact point set using the PointNet++ model.

[0013] Preferably, the use of a word-level cross-modal cross-attention mechanism specifically involves: The local contact surface geometric features are projected into a query matrix, and a text instruction sequence is formed by concatenating them from the hierarchical text library. The text sequence instructions are projected into a key matrix and a value matrix. The attention weight of each contact node to different text terms is calculated by the similarity between the query matrix and the key matrix. Then, the attention weights are calculated and the value matrix is ​​weighted and summed to obtain text enhancement features related to functional semantics. The text enhancement features are then aggregated to obtain contact summary features.

[0014] Preferably, the six-degree-of-freedom mate attitude includes the three-dimensional translation parameters and rotation quaternions of the part, wherein the rotation quaternions are used to convert to obtain a rotation matrix for rotating the part point cloud.

[0015] This invention also discloses a three-dimensional part assembly system based on hierarchical text priors, comprising the following modules: The feature extraction module encodes the features of each part point cloud through multiple disordered 3D part point clouds of the object to be assembled, extracts the global geometric features of the disordered 3D parts, and constructs a hierarchical text library. The hierarchical text library covers multi-dimensional functional text features including part identity information, functional attribute information, and spatial context information. The semantic-aware global topology initialization module distinguishes the functional roles of the three-dimensional parts through a semantic affinity soft grouping mechanism and a set context attention mechanism, thereby obtaining a topologically consistent global coarse pose. The text-guided fine contact alignment module extracts local contact surface geometric features using the global coarse pose, and uses a word-level cross-modal cross attention mechanism to inject the functional text semantics in the functional text features into the contact surface geometric features, and performs aggregation processing to eliminate local physical conflicts caused by geometric ambiguity. The attitude regression module fuses the aggregated contact surface geometric features with global context features, and after global interaction, infers the precise six-degree-of-freedom mating attitude of the 3D part.

[0016] The present invention also discloses an electronic device, including one or more processors, a memory, and a computer program, which, when executed by the processor, is used to implement the above-described hierarchical text prior-based three-dimensional part assembly method.

[0017] The present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, is used to implement the above-described three-dimensional part assembly method based on hierarchical text priors.

[0018] Compared with the prior art, the present invention has the following beneficial effects: The 3D part assembly method based on hierarchical text priors provided in this invention first extracts the geometric features of unordered 3D parts and textual features containing functional descriptions of the parts. Then, it infers the precise final assembly posture of the 3D parts through a novel two-stage semantic-aware framework. Feature extraction involves extracting global geometric features of the 3D parts and multi-dimensional functional textual features covering identity, functional attributes, and spatial context. Semantic awareness combines open-world semantic priors, distinguishing the functional roles of parts through semantic affinity soft grouping and ensemble contextual attention mechanisms, thereby establishing a topologically consistent global coarse posture, significantly improving the semantic accuracy and global topological rationality of multi-part assembly. Semantic injection is used during text-guided fine contact alignment. A word-level cross-modal cross-attention mechanism injects explicit functional textual semantics into the local geometric contact surface, eliminating local physical conflicts caused by geometric ambiguity and greatly enhancing the physical connectivity and alignment accuracy between adjacent parts. Finally, posture regression fuses and interacts with the aggregated contact surface features and global contextual features to infer a precise and reliable six-degree-of-freedom assembly posture.

[0019] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the three-dimensional part assembly method based on hierarchical text priors in this invention.

[0021] Figure 2 This is a schematic diagram of the framework principle of three-dimensional part assembly based on hierarchical text prior in this invention. Detailed Implementation

[0022] To make the technical means, inventive features, objectives, and effects of the invention readily understandable, the invention is further illustrated below with reference to specific figures. However, the invention is not limited to the embodiments described below.

[0023] It should be noted that the structures, proportions, sizes, etc., illustrated in the accompanying drawings of this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0024] Existing learning-based methods for 3D part assembly tasks typically rely primarily on the geometric information of the parts. They predict the topological relationships between parts and their final assembly posture by extracting features and modeling relationships from point clouds, meshes, or part graph structures. In terms of technical architecture, these methods mainly include graph neural network-based methods, Transformer-based methods, and pairwise shape matching-based methods.

[0025] However, while these methods are effective in handling assembled objects with significant shape differences and relatively simple structural relationships, they still have the following shortcomings in practical applications: First, geometrically similar parts can easily lead to assembly confusion. In actual assembly objects, multiple parts often exhibit strong geometric similarities—especially in scenarios such as furniture and industrial supports. Different parts may have similar geometric features such as length and cross-section, making it difficult to accurately distinguish their true functional roles based solely on geometric shape. For example, in assembly objects such as chairs, tables, and support frames, parts such as chair legs, crossbars, columns, connecting rods, and frames are often very similar in local shape, but their functions in the overall structure are not the same, and their corresponding assembly positions, connection objects, and directional constraints also differ significantly. Existing methods lack explicit modeling of part identity, functional attributes, and structural context, which can easily lead to incorrect matching of geometrically similar parts to similar positions during the global assembly reasoning stage, resulting in problems such as part confusion, topological errors, and unreasonable overall structures.

[0026] Secondly, the precision of fine alignment in local contact areas is insufficient. 3D part assembly involves not only reasoning about global topological relationships but also fine alignment of local contact areas. For assembly objects with mortise and tenon joints, interlocking joints, snap-fits, fittings, supports, or adjacency relationships, relying solely on rough global attitude prediction is often insufficient to guarantee accurate alignment of local connection areas. In existing technologies, many methods primarily rely on local geometry for contact area matching and alignment, lacking further guidance on connection relationships, functional constraints, and spatial semantics. Therefore, in complex connection areas, phenomena such as contact surface misalignment, excessive connection gaps, local interference, directional deviations, or unstable connections easily occur, thus affecting the overall assembly quality.

[0027] Third, the utilization of semantic prior information is insufficient. Most existing methods treat assembly tasks as purely geometric reasoning problems, failing to effectively introduce semantic prior information that can characterize part identity, functional attributes, and spatial context. Although some methods attempt to utilize basic structural knowledge such as part category labels, they lack a mechanism to simultaneously apply semantic priors to global topology initialization and local contact alignment processes, resulting in insufficient integration between semantic information and geometric reasoning.

[0028] How to effectively introduce semantic prior information that can characterize the identity, functional attributes and spatial context of parts during the assembly process of 3D parts, and make such semantic prior information apply to both global topology initialization and local contact alignment processes, so as to reduce assembly ambiguity caused by geometrically similar parts and improve local connection accuracy, has become a technical problem that urgently needs to be solved in this field.

[0029] The first embodiment of the present invention provides a three-dimensional part assembly method based on hierarchical text priors, such as... Figure 1 As shown, it is implemented through the following steps.

[0030] Step S1: Feature extraction step. Through multiple disordered 3D part point clouds of the object to be assembled, feature encoding is performed on each part point cloud to extract the global geometric features of the disordered 3D parts and construct a hierarchical text library. The hierarchical text library covers multi-dimensional functional text features including part identity information, functional attribute information and spatial context information.

[0031] Specifically, firstly, N unordered 3D point clouds of the object to be assembled are acquired. Feature encoding is then performed on each point cloud to extract global geometric features that characterize the overall shape and structure of the part, independent of single-view image segmentation results. In this embodiment, the OpenShape 3D vision-language joint encoder is used to extract these global geometric features.

[0032] The hierarchical text library includes functional priors for 3D parts, for each part category. The corresponding set of prompt words The semantic description includes three dimensions: (1) identity, describing the category name and geometric details; (2) functional attributes, used to describe the connection and support relationships of the parts; and (3) spatial context, used to describe the relative position or adjacency relationship of the parts in the overall assembly structure. The final text feature vector... By analyzing the set The average of all prompt word features is used to calculate the result, thus eliminating the bias caused by a single expression.

[0033] For example, taking the component category "chair leg" as an example, its identity description could include "one chair leg" or "cylindrical support rod," its functional attribute description could include "used to support the seat" or "bear weight," and its spatial context description could include "located at the bottom" or "vertical structure." These multiple descriptions are input into a text encoder to obtain multiple cue word features within the same category. Then, these cue word features are averaged to obtain the text prior feature vector corresponding to the "chair leg" category. Similarly, for the category "chair back crossbar," its identity description could include "chair back crossbar" or "slender vertical rod," its functional attribute description could include "used for leaning back" or "supports the back," and its spatial context description could include "located at the back" or "located above the seat."

[0034] Step S2: Semantic-aware global topology initialization step, which uses semantic affinity soft grouping mechanism and set context attention mechanism to distinguish the functional roles of the three-dimensional parts and obtain a topologically consistent global coarse pose.

[0035] Specifically, the semantic affinity soft grouping mechanism is as follows: A linear transformation is performed on the feature encoding of each part point cloud by the OpenShape encoder to obtain the adaptation features. Then, the cosine similarity between the adaptation features corresponding to each part point cloud and each text feature sequence is calculated. All the cosine similarities are then normalized to generate a semantic affinity probability distribution.

[0036] The specific set context attention mechanism is as follows: The semantic affinity probability distribution is concatenated with the global geometric features to obtain the semantic enhancement features of each part. The semantic enhancement features of each part are input into the attention network to model the structural co-occurrence relationship and semantic compatibility relationship among all parts, thereby obtaining the context-aware features of each part. The context-aware features are input into the multilayer perceptron regression network to obtain the three-dimensional translation vector and rotation quaternion of each part, forming the global coarse pose.

[0037] Specifically, to decouple the functional roles of parts from global geometric features, the OpenShape encoder is first used to encode the point cloud of the i-th part, obtaining the global geometric feature Fob,i. Then, a semantic adaptation layer is used to linearly transform Fob,i to obtain the adaptation feature Fob,i'. The linear transformation can be expressed as Fob,i' = WsFob,i + bs, where Ws is the learnable weight matrix and bs is the learnable bias term. Subsequently, the cosine similarity between the adaptation feature Fob,i' of the i-th part and the prior text feature Ftext,j of the j-th category is calculated, and the similarity scores corresponding to all categories are normalized to generate a semantic affinity probability distribution matrix S.

[0038] The calculation process is shown in the following formula: Where i represents the part number, j represents the text prior category number, Fob,i represents the global geometric feature of the i-th part, Fob,i' represents the adapted feature after semantic adaptation layer processing, and Ftext,j represents the text prior feature of the j-th category; τ is a temperature parameter used to adjust the sharpness of the similarity distribution. τ can be preset with an initial value and updated as a learnable parameter during model training; the larger τ is, the more the model tends to highlight categories with higher similarity, and the smaller τ is, the smoother the semantic affinity distribution. The key to feature adaptation lies in subsequently using cosine similarity, the learnable temperature parameter, and Softmax to preserve the soft probability distribution, then fusing it with the original geometric features, and finally correcting it by combining ensemble self-attention with the contextual relationships of all parts.

[0039] Furthermore, for a fixed i-th part, its similarity vector relative to all prior text categories is first obtained, and then normalized along the category dimension using the Softmax function to obtain the semantic affinity probability distribution Si={Si,j}. Here, Si,j represents the matching probability between the i-th part and the j-th category prior text. This probability distribution Si is then concatenated with the global geometric features Fob,i extracted by OpenShape to form the semantically enhanced feature representation Fin,i.

[0040] By using a self-attention mechanism to model the global dependencies between all parts, each part can refer to the structural co-occurrence relationship and semantic compatibility relationship of other parts when updating its own features, thereby correcting local misclassification or confusion of symmetrical parts and outputting a refined context-aware feature Fctx.

[0041] The refined context-aware feature Fctx,i is input into a multilayer perceptron regression network consisting of several fully connected layers and nonlinear activation functions. The network outputs the three-dimensional translation vector tic and rotation quaternion qic of the i-th part, thereby obtaining the global coarse six-degree-of-freedom pose of the part.

[0042] It should be noted that in step S2, after implementing semantic affinity soft grouping, the affinity probability distribution of each part relative to different functional categories is concatenated with the global geometric features extracted by OpenShape to obtain semantically enhanced features. Then, a contextual attention mechanism is used to model the structural co-occurrence and semantic compatibility relationships among multiple parts to refine the features and eliminate local misclassification noise. Subsequently, the global coarse pose of each part is obtained by regression based on the refined contextually aware features. Specifically, after semantic adaptation of the global geometric features, the cross-modal similarity between the semantically adapted geometric features and text features is calculated to obtain the semantic affinity probability distribution. Then, semantically enhanced features are generated based on the semantic affinity probability distribution, and the contextual features are refined to obtain the contextual features of each part. Finally, regression is performed to obtain the coarse six-DOF pose of each part.

[0043] In one example, based on the cross-modal similarity between the semantically adapted global geometric features and text features, the semantic affinity probability distribution of each part relative to the prior text of each category is obtained; then the semantic affinity probability distribution is fused with the global geometric features to obtain semantically enhanced features; the contextual relationship between multiple parts is modeled using a set contextual attention mechanism to obtain refined contextual features, and the rough pose of each part is obtained by regression based on these features.

[0044] Step S3: Text-guided fine contact alignment step, using the global coarse pose to extract local contact surface geometric features, using a word-level cross-modal cross attention mechanism to inject the functional text semantics in the functional text features into the contact surface geometric features, and performing aggregation processing to eliminate local physical conflicts caused by geometric ambiguity.

[0045] The extraction of local contact surface geometric features using the aforementioned global coarse pose specifically involves: Based on the coarse translation vector and rotation quaternion in the global coarse pose, the original point cloud of each part is transformed into the coarse assembly coordinate system to obtain the coarse assembly point cloud. Calculate the adjacent distance from each point on each part to the point clouds of other parts, and form a set of contact points by selecting each point whose adjacent distance is less than a preset threshold as candidate contact points. Several candidate contact points are selected from the set of contact points to form a local contact point set, and the local contact surface geometric features are extracted from the local contact point set using the PointNet++ model.

[0046] The PointNet++ model extracts features from the point cloud from local to global perspectives in a hierarchical manner. First, the entire point cloud is divided into multiple overlapping local regions. For each local region, its fine geometric features are extracted. Then, these local features are combined to form a larger region, and higher-level features are extracted again. This process is repeated until the global features of the entire point cloud are obtained.

[0047] The specific implementation of the word-level cross-modal cross-attention mechanism is as follows: The local contact surface geometric features are projected into a query matrix, and a text instruction sequence is formed by concatenating them from the hierarchical text library. The text sequence instructions are projected into a key matrix and a value matrix. The attention weight of each contact node to different text terms is calculated by the similarity between the query matrix and the key matrix. Then, the attention weights are calculated and the value matrix is ​​weighted and summed to obtain text enhancement features related to functional semantics. The text enhancement features are then aggregated to obtain contact summary features.

[0048] The core purpose of this mechanism is to enable each "geometric contact point" to actively "read" the text instructions and extract the functional semantics relevant to itself. The query matrix mentioned above comes from geometric features, which are input into a linear layer by inputting the contact node features. Each row in the matrix represents a "contact node" asking the question of "which functional semantics to match". The key matrix and value matrix come from functional text features in a hierarchical text library. The text instruction sequence is input into two independent linear layers. The key matrix is ​​equivalent to the question index, and the value matrix is ​​equivalent to the question answer. Calculating the attention weights involves calculating the relevance score between each contact node and each text word. The subsequent extraction of text enhancement features involves weighted summation of the value matrix using the calculated attention weights.

[0049] In step S3, to extract and fuse local contact node features, the original point cloud Pi of the i-th part is first transformed into a coarse assembly coordinate system based on the previously predicted coarse translation vector tic and rotation quaternion qic, resulting in a coarsely assembled point cloud. Then, for each point on the i-th part, the nearest neighbor distance to the point cloud sets of other parts is calculated, and points with a distance less than a preset threshold ε are selected as candidate contact points. If the number of candidate contact points is insufficient, several nearest points are selected to supplement the number. K points are then sampled from the candidate contact points to form a local contact point set Ci={ci,k}. The input to the PointNet++ network is the 3D coordinates in this local contact point set Ci, not the coarse pose itself; the PointNet++ network extracts local geometric features Fcont,i,k from these local contact point sets. To avoid incorrect judgments based solely on local shape, the local geometric features Fcont,i,k are concatenated with the global context features Fctx,i of the corresponding part obtained in step S2, and then processed by a fusion network to obtain the contact node features Fnode,i,k. The calculation process is as follows: To perform cross-modal semantic injection, explicit functional attribute semantics are injected into the contact surface features through word-level cross-modal attention. Specifically, the contact node features Fnode are projected as a query matrix Q, and the text instruction sequence Finstr is projected as a key matrix K and a value matrix V. The attention weight Across for each contact node to different text words is calculated based on the similarity between Q and K. Then, V is weighted and summed using this attention weight to obtain the text enhancement features related to functional semantics. Finally, the text enhancement features are residually fused and normalized with the original contact node features to obtain the semantically injected contact surface features Finjected. The calculation process of attention weights and semantically injected features is shown in the following formula: Finally, in order to aggregate the contact surface features, max pooling is performed on the K semantically injected contact surface features of the same part to retain the most significant functional contact information, thus obtaining the contact summary feature Fsummary,i of the part.

[0050] In one example, the part point cloud is transformed to a coarse assembly coordinate system based on the coarse pose, and candidate contact regions are determined according to the nearest neighbor distance between different part point clouds. Local contact surface geometric features of the candidate contact regions are extracted, and the local contact surface geometric features are fused with the context features of the corresponding parts to obtain contact node features. Using a word-level cross-modal cross-attention mechanism, the functional attribute semantics in the text feature sequence are injected into the local contact surface features, and the semantic enhancement features of multiple candidate contact regions are aggregated to obtain contact summary features for the final pose regression.

[0051] Step S4: The attitude regression step involves splicing and fusing the aggregated contact surface geometric features with the global context features, performing global interaction, and then inferring the precise six-degree-of-freedom mating attitude of the 3D part.

[0052] Specifically, the global context feature Fctx,i of the i-th part is concatenated with the aggregated contact summary feature Fsummary,i to form the final enhanced representation Ffinal,i: The final augmented representation Ffinal,i of all parts undergoes self-attention-based global interaction processing, enabling each part to simultaneously perceive the contact relationships, functional compatibility relationships, and global topological relationships of other parts when predicting its final pose. The interactive features are then input into the pose regression network, which outputs the final translation vector tifinal and rotation quaternion qifinal, thereby obtaining the final six-DOF assembly pose of each part.

[0053] like Figure 2 As shown, in this embodiment, semantically aware global topology initialization introduces open-world semantic priors to generate semantic affinity soft grouping, thereby breaking the limitations of purely geometric methods. This effectively distinguishes visually indistinguishable but functionally different similar parts, resolving global logical errors caused by geometric similarity, and significantly improving the semantic correctness and global topological rationality of multi-part assembly. Text-guided fine contact alignment introduces cross-modal semantics, incorporating functional instruction text into local contact surface optimization. This not only effectively eliminates local physical conflicts but also significantly enhances the physical connectivity and alignment accuracy between adjacent parts. By designing assembly steps in two stages—global topology initialization and local contact alignment—both global structural rationality and local connection accuracy are considered, thereby improving the overall reliability of the 3D part assembly results.

[0054] To demonstrate the comprehensive performance of this embodiment in 3D part assembly, the following verification is performed on the PartNet 3D part assembly benchmark dataset.

[0055] First, acquire a set of unordered 3D part point cloud data, as well as a hierarchical set of text prompts containing part category identity, functional attributes, and spatial context descriptions.

[0056] Then, a feature encoder is used to extract the global geometric features and textual prior features of the 3D part, respectively.

[0057] Next, semantic adaptation is performed on the global set features, cross-modal similarity between them and prior text features is calculated, and normalization is performed on the text category dimension to obtain semantic affinity soft grouping; then, the structural co-occurrence relationship and semantic compatibility relationship between multiple parts are modeled through the set context attention mechanism, and refined context-aware features and global coarse pose are output.

[0058] The point clouds of each part are transformed to a coarse assembly coordinate system using the obtained coarse pose. A set of candidate contact points is determined based on the distance between points, and local contact surface geometric features are extracted from the set of candidate contact points. Then, word-level cross-modal cross-attention calculation is performed on the local contact surface geometric features and text features, and semantic injection is performed to obtain aggregated micro-contact surface features.

[0059] The aggregated contact surface features are concatenated and fused with global context features, and global interactive processing is performed on the fused features. During the training phase, the model parameters are optimized through translation loss, rotation loss, and overall assembly shape constraints. During the inference phase, the accurate six-degree-of-freedom assembly posture is predicted and output based on the fused features.

[0060] Repeat the above steps until the assembly posture reasoning of all 3D part objects in the dataset is completed.

[0061] In the above experimental verification, shape chamfer distance, part accuracy, and connectivity accuracy were used as evaluation indicators. Shape chamfer distance measures the global geometric quality of the overall assembly result; a smaller value indicates a better assembly result. Part accuracy evaluates the semantic correctness of part placement, and connectivity accuracy evaluates the quality of physical connections between adjacent parts; a larger value indicates a better assembly effect. Verification results show that the method of this invention performs well in terms of global topology rationality (SCD), part semantic correctness (PA), and local connectivity accuracy (CA). Experimental results are shown in Table 1.

[0062] Table 1. Performance of different methods on the PartNet dataset In the first embodiment of this invention, the input is a point cloud of multiple unordered 3D parts, independent of single-view image segmentation results. Text priors are directly matched cross-modal with the 3D point cloud features to distinguish parts with similar geometric appearances but different functional roles. During assembly, instead of using only part category names or assembly order, a text library containing three levels—identity information, functional attributes, and spatial context—is constructed for each part category. Identity information describes what the part is, functional attributes describe what the part connects to or supports, and spatial context describes the typical location of the part within the assembly. The text priors operate simultaneously in two stages: in the coarse assembly stage, semantic affinity soft grouping and set context refinement eliminate confusion regarding part functional roles; in the fine assembly stage, word-level cross-modal cross-attention injects functional and spatial semantics into local contact surface features, directly participating in pose estimation at the feature level and contact node level. Therefore, this application can solve global topological errors caused by geometrically similar parts, as well as local connectivity problems such as contact surface misalignment, gaps, and interference. This application requires simultaneous prediction of the 6-DOF pose of multiple parts in a unified assembly coordinate system, rather than the pose and size of a single observed object relative to the camera. In the first stage of coarse pose regression, a topologically consistent coarse pose is obtained based on semantic affinity and self-attention across the entire part set. In the second stage of fine pose regression, local contact regions where connections may occur between parts are searched based on the coarse pose. The final input features of this application are formed by concatenating global context features and local contact summary features. The former represents the functional role of a part in the entire assembly, and the latter represents the local contact relationships injected with textual semantics. All concatenated part features are then input into a Transformer for cross-part global interaction, enabling each part to simultaneously perceive the contact relationships, functional compatibility, and global topological relationships of other parts when regressing the final pose. This application first determines candidate contact regions based on the coarse assembly pose, sampling only local points where connections may occur and inputting them into PointNet++; then, the local geometric features are fused with global semantics and functional text. Experiments show that sampling 50 contact points yields the best results, reducing interference from irrelevant points and improving connection accuracy.

[0063] The hierarchical text library in this embodiment is not a simple collection of category labels, but mainly includes the following processing: ① The semantic description of each part category is decomposed into three complementary dimensions: Identity dimension: category name and typical geometric features; Functional attribute dimensions: functional relationships such as support, connection, and carrying; Spatial context dimension: the typical location and adjacency relationship of a part in the overall assembly structure.

[0064] ② We adopt manually defined assembly semantic templates and use GPT-4o to expand natural language according to the constraints of "identity-connection / support relationship-typical spatial location" to form multiple equivalent but different prompt words, thereby reducing the matching bias caused by a single word.

[0065] ③ A frozen OpenShape text encoder is used to encode multiple cue words separately, and the category-level text prior is formed by averaging the cue words, so that the semantic representation of the same category is more stable for different language expressions.

[0066] ④ The text library adopts a two-granularity approach: in the coarse stage, semantic affinity is calculated using category-level average vectors; in the fine stage, the lexical sequences of functional attributes and spatial context are preserved, so that different contact areas can selectively focus on the functional semantics related to themselves.

[0067] The second embodiment of the present invention provides a three-dimensional part assembly system based on hierarchical text priors, including a feature extraction module, a semantically aware global topology initialization module, a text-guided fine contact alignment module, and a pose regression module.

[0068] The feature extraction module is used to encode the features of each part point cloud from multiple disordered 3D part point clouds of the object to be assembled, extract the global geometric features of the disordered 3D parts, and construct a hierarchical text library. The hierarchical text library covers multi-dimensional functional text features including part identity information, functional attribute information, and spatial context information.

[0069] The semantic-aware global topology initialization module is used to distinguish the functional roles of the three-dimensional parts through a semantic affinity soft grouping mechanism and a set context attention mechanism, so as to obtain a topologically consistent global coarse pose.

[0070] The text-guided fine contact alignment module is used to extract local contact surface geometric features using the global coarse pose. It uses a word-level cross-modal cross attention mechanism to inject the functional text semantics in the functional text features into the contact surface geometric features and performs aggregation processing to eliminate local physical conflicts caused by geometric ambiguity.

[0071] The attitude regression module is used to fuse the aggregated contact surface geometric features with global context features, and after global interaction, infer the precise six-degree-of-freedom mating attitude of the 3D part.

[0072] Since the first embodiment corresponds to this embodiment, this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment remain valid in this embodiment, and the technical effects achievable in the first embodiment can also be achieved in this embodiment. To reduce repetition, they will not be repeated here. Correspondingly, the relevant technical details mentioned in this embodiment can also be applied to the first embodiment.

[0073] A third embodiment of the present invention provides an electronic device, including a processor, a memory, and a computer program. When the computer program is executed by the processor, it is used to implement the three-dimensional part assembly method based on hierarchical text prior as described in the first embodiment.

[0074] A fourth embodiment of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, is used to implement the three-dimensional part assembly method based on hierarchical text prior as described in the first embodiment.

[0075] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A three-dimensional part assembly method based on hierarchical text priors, characterized in that, Includes the following steps: The feature extraction step involves encoding the features of each part point cloud through multiple disordered 3D part point clouds of the object to be assembled, extracting the global geometric features of the disordered 3D parts, and constructing a hierarchical text library. The hierarchical text library covers multi-dimensional functional text features including part identity information, functional attribute information, and spatial context information. The semantically aware global topology initialization step distinguishes the functional roles of the three-dimensional parts through a semantic affinity soft grouping mechanism and a set context attention mechanism, thereby obtaining a topologically consistent global coarse pose. The text-guided fine contact alignment step uses the global coarse pose to extract local contact surface geometric features, and uses a word-level cross-modal cross attention mechanism to inject the functional text semantics in the functional text features into the contact surface geometric features, and performs aggregation processing to eliminate local physical conflicts caused by geometric ambiguity. The attitude regression step combines the aggregated contact surface geometric features with global context features, performs global interaction, and infers the precise six-degree-of-freedom mating attitude of the 3D part.

2. The three-dimensional part assembly method based on hierarchical text priors according to claim 1, characterized in that, The part identity information in the functional text features is used to describe the category name and geometric details of the part, the functional attribute information is used to describe the support connection relationship of the part, and the spatial context information is used to describe the relative positional relationship of the part in the overall assembly structure. The text feature vector for each part is obtained by encoding the functional text features separately and averaging the encoding results to eliminate single expression bias.

3. The three-dimensional part assembly method based on hierarchical text priors according to claim 1, characterized in that, The semantic affinity soft grouping mechanism is specifically as follows: A linear transformation is performed on the feature encoding of each part point cloud by the OpenShape encoder to obtain the adaptation features. Then, the cosine similarity between the adaptation features corresponding to each part point cloud and each text feature sequence is calculated. All the cosine similarities are then normalized to generate a semantic affinity probability distribution.

4. The three-dimensional part assembly method based on hierarchical text priors according to claim 3, characterized in that, The specific set context attention mechanism is as follows: The semantic affinity probability distribution is concatenated with the global geometric features to obtain the semantic enhancement features of each part. The semantic enhancement features of each part are input into the attention network to model the structural co-occurrence relationship and semantic compatibility relationship among all parts, thereby obtaining the context-aware features of each part. The context-aware features are input into the multilayer perceptron regression network to obtain the three-dimensional translation vector and rotation quaternion of each part, forming the global coarse pose.

5. The three-dimensional part assembly method based on hierarchical text priors according to claim 1, characterized in that, The specific steps for extracting local contact surface geometric features using the global coarse pose are as follows: Based on the coarse translation vector and rotation quaternion in the global coarse pose, the original point cloud of each part is transformed into the coarse assembly coordinate system to obtain the coarse assembly point cloud. Calculate the adjacent distance from each point on each part to the point clouds of other parts, and form a set of contact points by selecting each point whose adjacent distance is less than a preset threshold as candidate contact points. Several candidate contact points are selected from the set of contact points to form a local contact point set, and the local contact surface geometric features are extracted from the local contact point set using the PointNet++ model.

6. The three-dimensional part assembly method based on hierarchical text priors according to claim 5, characterized in that, The specific implementation of the word-level cross-modal cross-attention mechanism is as follows: The local contact surface geometric features are projected into a query matrix, and a text instruction sequence is formed by concatenating them from the hierarchical text library. The text sequence instructions are projected into a key matrix and a value matrix. The attention weight of each contact node to different text terms is calculated by the similarity between the query matrix and the key matrix. Then, the attention weights are calculated and the value matrix is ​​weighted and summed to obtain text enhancement features related to functional semantics. The text enhancement features are then aggregated to obtain contact summary features.

7. The three-dimensional part assembly method based on hierarchical text priors according to claim 1, characterized in that, The six-degree-of-freedom mate posture includes the three-dimensional translation parameters and rotation quaternions of the part. The rotation quaternions are used to convert and obtain a rotation matrix to perform rotation transformation on the part's point cloud.

8. A three-dimensional parts assembly system based on hierarchical text priors, characterized in that, Includes the following modules: The feature extraction module encodes the features of each part point cloud through multiple disordered 3D part point clouds of the object to be assembled, extracts the global geometric features of the disordered 3D parts, and constructs a hierarchical text library. The hierarchical text library covers multi-dimensional functional text features including part identity information, functional attribute information, and spatial context information. The semantic-aware global topology initialization module distinguishes the functional roles of the three-dimensional parts through a semantic affinity soft grouping mechanism and a set context attention mechanism, thereby obtaining a topologically consistent global coarse pose. The text-guided fine contact alignment module extracts local contact surface geometric features using the global coarse pose, and uses a word-level cross-modal cross attention mechanism to inject the functional text semantics in the functional text features into the contact surface geometric features, and performs aggregation processing to eliminate local physical conflicts caused by geometric ambiguity. The attitude regression module fuses the aggregated contact surface geometric features with global context features, and after global interaction, infers the precise six-degree-of-freedom mating attitude of the 3D part.

9. An electronic device comprising one or more processors, a memory, and a computer program, wherein the computer program, when executed by the processor, is used to implement the three-dimensional part assembly method based on hierarchical text prior as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, is used to implement the three-dimensional part assembly method based on hierarchical text priors as described in any one of claims 1 to 7.

Images