Three-mode refined retrieval method, system and equipment for industrial parts

Abstract

The invention discloses an industrial part three-mode refined retrieval method, system and equipment, the industrial part three-mode refined retrieval method comprises the following steps: obtaining a point cloud of a part, extracting a global feature vector from the point cloud, the point cloud being obtained from a CAD model or 3D scanning data of the part; obtaining a 2D image of the part, and extracting a 2D visual feature vector from the 2D image; obtaining character information of the part, and extracting a semantic embedding vector from the character information; fusing the global feature vector, the 2D visual feature vector and the semantic embedding vector to obtain a fused embedding vector; managing the fusion embedded vector and the unique identifier of the part so as to construct a vector database; and extracting a query vector from the to-be-retrieved part, and searching in the vector database based on the query vector to obtain a part list similar to the to-be-retrieved part.

Description

Technical Field

[0001] This invention belongs to the field of parts retrieval technology, specifically relating to a three-modal refined retrieval method, system and equipment for industrial parts. Background Technology

[0002] Automatic identification and retrieval of industrial parts is an extremely challenging task, its complexity stemming primarily from the following aspects. First, industrial parts typically exhibit high levels of "intra-class variance" and "inter-class similarity." For example, screws of different lengths, materials, or thread specifications may appear visually very similar, yet their functions and applications are drastically different; conversely, bearings or gears with vastly different functions may be difficult to distinguish macroscopically. Second, accurate identification of parts often relies on minute geometric features, precise dimensional parameters, and specific technical attributes—subtle differences that are difficult to capture effectively using traditional computer vision methods.

[0003] Existing technologies typically employ 2D image-based retrieval systems, which often use convolutional neural networks (CNNs) such as ResNet to extract features. However, these methods suffer from fundamental limitations when dealing with industrial parts. CNN models focus more on local information and lack awareness of global context, making it difficult to effectively extract discriminative features. They perform poorly when handling small features or scenes with significant variations in target size; more importantly, 2D image retrieval systems are almost indistinguishable from visually similar parts with completely different 3D geometries. Their performance is extremely sensitive to factors such as shooting angle, lighting conditions, and background occlusion, resulting in poor robustness in real-world industrial environments. Summary of the Invention

[0004] To overcome the above-mentioned technical defects, the present invention provides a three-modal refined retrieval method, system and equipment for industrial parts, which can improve the accuracy of retrieval.

[0005] A refined three-modal retrieval method for industrial parts includes: The point cloud of the component is obtained, and the global feature vector is extracted from the point cloud. The point cloud is obtained from the CAD model or 3D scan data of the component. Acquire 2D images of the components and extract 2D visual feature vectors from the 2D images; Obtain the textual information of the components and extract semantic embedding vectors from the textual information; The global feature vector, 2D visual feature vector, and semantic embedding vector are fused to obtain the fused embedding vector; The embedded vectors are associated with the unique identifiers of the components to build a vector database; Extract query vectors from the parts to be retrieved, and search the vector database based on the query vectors to obtain a list of parts similar to the parts to be retrieved.

[0006] As a further improvement of the present invention, the step of obtaining the point cloud of the component and extracting the global feature vector from the point cloud includes: The farthest point sampling algorithm is used to select a subset of the acquired point cloud as the centroid of the local region; Using each centroid as the center, search for all neighboring points in a spherical space with a predetermined radius around the center using a spherical query method, and construct multiple local region point sets using all neighboring points; The left side of the points in the local region is normalized relative to the centroid, and the global feature vector is extracted using the PointNet network.

[0007] As a further improvement of the present invention, the step of acquiring a 2D image of a component and extracting a 2D visual feature vector from the 2D image includes: Divide the 2D image into several non-overlapping local windows; Perform self-attention calculation on a local window; Shift the local window multiple times; Self-attention calculation is performed on the local window for each shift; Feature transformation is performed on the self-attention calculation results before and after the shift to obtain a 2D visual feature vector.

[0008] As a further improvement of the present invention, the step of fusing the global feature vector, the 2D visual feature vector, and the semantic embedding vector to obtain the fused embedding vector includes: Calculate the cross-attention of the global feature vector, 2D visual feature vector, and semantic embedding vector respectively; Enhancement processing is performed on global feature vectors, 2D visual feature vectors, and semantic embedding vectors based on cross-attention scores; The enhanced global feature vector, 2D visual feature vector, and semantic embedding vector are concatenated to obtain the context-aware vector. Calculate the gating vector; The fused embedding vector is calculated based on the context-aware vector and the gating vector.

[0009] This invention also provides a three-modal refined retrieval system for industrial parts, characterized in that, for implementing the above-mentioned three-modal refined retrieval method for industrial parts, it includes: A hierarchical 3D geometric feature extractor is used to acquire point clouds of parts and extract global feature vectors from the point clouds. A global context-aware 2D visual feature extractor is used to acquire 2D images of parts and extract 2D visual feature vectors from the 2D images. A domain-adaptive text feature extractor is used to fuse global feature vectors, 2D visual feature vectors, and semantic embedding vectors to obtain a fused embedding vector. The gated cross-attention fusion module is used to fuse global feature vectors, 2D visual feature vectors, and semantic embedding vectors to obtain a fused embedding vector. A vector database is used to store fusion embedding vectors, which are associated with unique identifiers of components; The hierarchical 3D geometric feature extractor, the global context-aware 2D visual feature extractor, and the domain-adaptive text feature extractor are also used to extract the global feature vector, 2D visual feature vector, and semantic embedding vector of the component to be retrieved. The gated cross-attention fusion module is also used to fuse the global feature vector, and / or 2D visual feature vector, and / or 2D semantic embedding vector of the component to be retrieved to obtain the query vector. The query module is used to search the vector database based on the query vector to obtain a list of parts similar to the part to be retrieved.

[0010] As a further improvement of the present invention, the hierarchical 3D geometric feature extractor includes several set abstraction layers, wherein the set abstraction layers include: The sampling layer uses the farthest point sampling algorithm to select a subset of the acquired point cloud as the centroid of the local region. The grouping layer, with each centroid as the center, searches for all neighboring points in a spherical space with a predetermined radius around the center using a spherical query method, and constructs multiple local region point sets with all neighboring points; The PointNet layer normalizes the left side of the points in the local region relative to the centroid, and then uses the PointNet network to extract global feature vectors.

[0011] As a further improvement of the present invention, the domain-adaptive text feature extractor is obtained by training a pre-trained BERT model.

[0012] As a further improvement of the present invention, during the training of the pre-trained BERT model, a frequency truncation method is used to add new lexical units.

[0013] As a further improvement of the present invention, it also includes: jointly training a hierarchical 3D geometric feature extractor, a global context-aware 2D visual feature extractor, and a domain-adaptive text feature extractor.

[0014] The present invention also provides an electronic device, characterized in that it includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to realize the above-described three-modal refined retrieval method for industrial parts.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention integrates the 3D information, 2D information and text information of the parts to obtain a multimodal fusion embedding vector, associates the fusion embedding vector with the unique identifier of the parts, and constructs a vector database. Compared with the traditional parts retrieval that relies on a single modality (2D image or text), it can more comprehensively represent the essential features of the parts and provide a foundation for realizing refined retrieval. Attached Figure Description

[0016] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a flowchart of the three-modal refined retrieval method for industrial parts described in this invention. Detailed Implementation

[0017] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0018] Example 1 This embodiment presents a three-modal refined retrieval method for industrial parts, such as... Figure 1 As shown, it includes: S1. Obtain the point cloud of the component and extract the global feature vector from the point cloud. The point cloud is obtained from the CAD model or 3D scan data of the component.

[0019] First, the Farthest Point Sampling (FPS) algorithm is used to select a subset of the point cloud as the centroid of the local region. Then, with each centroid as the center, all neighboring points are searched in a spherical space with a predetermined radius around the center using a ball query method, and multiple local region point sets are constructed with all neighboring points. Finally, the left side of the points in the local region point sets is normalized relative to the centroid, and the global feature vector is extracted using the PointNet network.

[0020] Employing the farthest point sampling algorithm ensures that sampling points are evenly distributed across the component surface, thus achieving effective coverage of the entire geometry. The ball query approach captures local geometric context with fixed dimensions. The PointNet network can learn the relationships between points within a local region, thereby extracting feature vectors of local geometric patterns such as edges, corners, and curvature.

[0021] S2. Obtain 2D images of the components and extract 2D visual feature vectors from the 2D images.

[0022] Although 3D geometry is the primary basis for identification, 2D images still provide indispensable information, such as color (e.g., galvanized layer), texture (e.g., knurling), material reflection (e.g., the difference between stainless steel and aluminum alloy), and surface markings (e.g., model markings, rust, etc.). Therefore, in this embodiment, 2D images are also used to provide the basis for retrieval.

[0023] In step S2 above, the core is a shifted window-based self-attention mechanism. First, the 2D image is divided into several non-overlapping local windows. The self-attention calculation is restricted to each window, which greatly reduces the amount of computation. Self-attention calculation is performed on the local windows. Next, the local windows are shifted multiple times, and self-attention calculation is performed on each shifted local window. This allows image patches that originally belonged to different windows in the previous layer to participate in the attention calculation together in the new window. Feature transformation is performed on the self-attention calculation results before and after the shift to obtain a 2D visual feature vector.

[0024] Through the above steps, cross-window information interaction is achieved, thereby constructing a hierarchical feature representation that can simultaneously capture multi-scale visual features from fine textures to overall shapes.

[0025] S3. Obtain the text information of the parts and extract semantic embedding vectors from the text information; in this embodiment, the vector extraction can be achieved by a BERT model that has been trained and finely tuned.

[0026] S4. The global feature vector, 2D visual feature vector, and semantic embedding vector are fused to obtain a fused embedding vector. Fusing multiple vectors aims to address the problem of intelligently combining multimodal information of varying importance.

[0027] Step S4 above is mainly divided into two stages: 1. Cross-attention stage: First, the cross-attention of the global feature vector, 2D visual feature vector, and semantic embedding vector is calculated separately. Based on the cross-attention scores, the global feature vector, 2D visual feature vector, and semantic embedding vector are enhanced. Taking the enhancement of the semantic embedding vector as an example, the semantic embedding vector... As a query, the global feature vector and 2D visual feature vectors These serve as the key and value, respectively. Attention scores are calculated.

[0028] Based on this, it is possible to learn which part of 3D geometry (hexagonal outline) or which region of 2D vision (the hexagonal top surface with specular reflection) is most strongly correlated with a certain word in the text (such as "hexagonal head"). This process is performed on all modal pairs, thereby enhancing the global feature vector, 2D visual feature vector, and semantic embedding vector.

[0029] 2. Gating fusion stage: First, the enhanced global feature vector, 2D visual feature vector, and semantic embedding vector are concatenated to obtain the context-aware vector. ).

[0030] Subsequently, the gate vector is computed through a fully connected layer and a sigmoid activation function. Gating vector It includes three weights , respectively, correspond to the weights of the global feature vector, the 2D visual feature vector, and the semantic embedding vector. The sum of these three factors is 1. These weights are dynamically generated based on the current input and reflect the importance judgment of each modality in this specific embodiment. The importance judgment of each modality in this specific instance is calculated based on the context-aware vector and the gating vector.

[0031] Fusion Embedded Vectors By analyzing context-aware feature vectors ( We obtain the weighted summation as follows:

[0032] The two-stage design of attention followed by gating in this embodiment is key to achieving performance improvements, and its logic has a profound causal relationship. The first stage, cross-attention, allows different modalities to exchange opinions and share knowledge, enabling each modality's representation to absorb contextual information from other modalities, becoming more comprehensive. For example, the word "stainless steel" in text can be associated with the metallic luster in a 2D image and the smooth surface in 3D geometry through the attention mechanism, thereby enhancing the representation capabilities of all modalities.

[0033] The second stage of gated fusion is a final decision made after sufficient information exchange. Based on the enriched information, it determines which modality's information is most reliable and discriminative for the specific component instance. For example, for a standard bolt with a simple shape and no special visual features, the gating mechanism might learn to assign higher weights to 3D geometry and textual descriptions; while for a part with unique visual markers, such as color coding or surface defects, the gating mechanism might dynamically increase the weights of 2D visual features. This is a completely data-driven, adaptive fusion strategy dependent on specific inputs.

[0034] S5. Associate the fused embedded vectors with the unique identifiers of the components to build a vector database.

[0035] For each industrial part in the vector database, multi-source data is collected, including at least one 2D high-resolution image, its corresponding 3D CAD model, or data obtained through 3D scanning. This data is then uniformly converted into point cloud format, along with relevant text descriptions, such as those from product specifications, design documents, or bills of materials. This trimodal data is fused to generate a single, highly condensed, and information-rich fused embedding vector for each part. This fused embedding vector is then associated with a unique identifier for the part, such as a SKU code or internal ID, and stored in a high-dimensional vector database specifically designed for efficient similarity searches, such as FAISS or Milvus.

[0036] S6. Extract the query vector from the part to be retrieved, and search the vector database based on the query vector to obtain a list of parts similar to the part to be retrieved.

[0037] In practical use, when a user needs to find a component, they can provide query information in any modality. For example, a user can upload a photo of the component taken on-site (2D image query), enter a text description (text query, such as "M6 socket head cap screw, grade 8.8"), or submit a 3D scan file (3D geometric query). The query information is sent to the corresponding feature extraction subsystem for processing. If multiple modalities of information are provided, these global feature vectors, 2D visual feature vectors, and semantic embedding vectors are fused into a unified query vector. Finally, this query vector is used to perform a k-nearest neighbor (k-NN) search in a high-dimensional vector database. By calculating metrics such as cosine similarity, a list of components most similar to the query is returned, sorted from highest to lowest similarity.

[0038] This embodiment achieves complementary fusion of three modal features by extracting global feature vectors to represent 3D structures, 2D visual feature vectors to capture appearance details, and semantic embedding vectors to describe associated attributes. This covers the spatial structure and appearance of parts, and also incorporates semantic information such as materials, models, and functions, enabling the fused embedding vectors to comprehensively and accurately represent the essential characteristics of parts. Subsequently, the fused embedding vectors are associated with the unique identifiers of parts to construct a vector database, laying the core foundation for refined retrieval. In actual use, only a single modal information needs to be input to achieve accurate retrieval of parts.

[0039] Example 2 This embodiment provides a three-modal refined retrieval system for industrial parts, used to implement the three-modal refined retrieval method for industrial parts described in Embodiment 1. The system includes: a hierarchical 3D geometric feature extractor, a global context-aware 2D visual feature extractor, a domain-adaptive text feature extractor, a gated cross-attention fusion module, a vector database, and a query module.

[0040] A hierarchical 3D geometric feature extractor is used to obtain point clouds of components and extract global feature vectors from the point clouds.

[0041] The essential properties of industrial components are defined by their three-dimensional geometry. Unlike existing technologies that rely on 2D projection, this embodiment uses a hierarchical 3D geometric feature extractor that directly analyzes the true 3D structure of the part. The hierarchical 3D geometric feature extractor (PointNet++) used in this embodiment is chosen because PointNet++ can capture multi-scale local structures, which is crucial for distinguishing components with fine geometric details. For example, threads with different pitches, minute chamfers, or grooves all need to be identified by analyzing the neighborhood relationships of local point sets. This embodiment overcomes the deficiency of existing technologies that only use 2D information and fundamentally ignore the most critical identification information.

[0042] The hierarchical 3D geometric feature extractor receives a point cloud (e.g., containing 2048 points) sampled from the CAD model or 3D scan data of a part as input. Its core utilizes a series of Set Abstraction Levels. Each level consists of three key components: The sampling layer uses the Farthest Point Sampling (FPS) algorithm to select a subset of the acquired point cloud as the centroid of the local region. The use of the Farthest Point Sampling algorithm can ensure that the sampling points are evenly distributed on the surface of the component, thereby achieving effective coverage of the entire geometry.

[0043] The grouping layer uses each centroid selected by the sampling layer as the center and searches for all neighboring points in a spherical space with a predetermined radius around the center through a sphere query. Multiple local region point sets are constructed using all neighboring points. The sphere query method can capture local geometric context with a fixed size.

[0044] The PointNet layer normalizes the left side of the points in a local region relative to the centroid, and uses a mini PointNet network to extract global feature vectors. The PointNet network can learn the relationships between points within a local region, thereby extracting feature vectors of local geometric patterns such as edges, corners, and curvature.

[0045] Through the stacking of multiple layers of ensemble abstraction, a global feature vector is finally generated. This global feature vector encodes complete 3D geometric information, from microscopic details to macroscopic contours, in a hierarchical manner.

[0046] A global context-aware 2D visual feature extractor is used to acquire 2D images of components and extract 2D visual feature vectors from the 2D images.

[0047] While 3D geometry is the primary basis for recognition, 2D images still provide indispensable information, such as color (e.g., galvanized coating), texture (e.g., knurling), material reflection (e.g., the difference between stainless steel and aluminum alloy), and surface markings (e.g., model markings, rust, etc.). In this embodiment, the global context-aware 2D visual feature extractor (Swin Transformer) has significant advantages over traditional CNN models. Its unique architecture allows it to flexibly model at different scales, and its computational complexity is linearly related to image size. More importantly, its core self-attention mechanism can capture global contextual information, understanding components as a whole rather than merely as a collection of local features. The global context-aware 2D visual feature extractor receives 2D images of components as input. First, the 2D image is divided into several non-overlapping local windows. Self-attention calculation is restricted to within each window, which greatly reduces the computational load. Self-attention calculation is performed on the local windows. Next, the local windows are shifted multiple times, and self-attention calculation is performed on each shifted local window. This allows image patches that originally belonged to different windows in the previous layer to participate in attention calculation together in the new window. Feature transformation is performed on the self-attention calculation results before and after the shift to obtain the 2D visual feature vector.

[0048] The domain-adaptive text feature extractor is used to fuse global feature vectors, 2D visual feature vectors, and semantic embedding vectors to obtain a fused embedding vector.

[0049] General-purpose language models (such as the original BERT) lack the ability to understand engineering terminology. For example, they may fail to distinguish the semantics of "metric" versus "imperial" threads, or understand the meaning of "Rockwell hardness," or differentiate the subtle but crucial semantic differences between "countersunk screws" and "pan head screws." Therefore, models must be fine-tuned on domain-specific corpora to adapt to the target domain's linguistic environment. Models pre-trained on scientific or technical texts (such as SciBERT) significantly outperform general-purpose models when handling domain-specific tasks. This embodiment uses a pre-trained BERT model (e.g., bert-base-uncased) as a foundation and further trains and fine-tunes it on a customized specialized corpus. This specialized corpus is constructed by collecting a large number of product specifications for industrial parts, engineering design manuals, patent documents in related fields, and equipment maintenance guidelines. The fine-tuning process adjusts the model's weights to optimize its understanding of the vocabulary, syntactic structures, and semantic relationships specific to this technical field. This embodiment employs frequency truncation, adding new words that appear frequently in the corpus but are not in the original vocabulary, such as specific alloy grades and technical standard codes, as new lexical units.

[0050] The gated cross-attention fusion module is used to fuse global feature vectors, 2D visual feature vectors, and semantic embedding vectors to obtain a fused embedding vector. The gated cross-attention fusion module is a deep neural network block.

[0051] The internal workflow of the gated cross-attention fusion module is divided into two closely linked stages: 1. Cross-attention stage: In this stage, the gated cross-attention fusion module performs cross-attention calculations to achieve modality information gain. The module calculates the cross-attention of the global feature vector, 2D visual feature vector, and semantic embedding vector respectively. Based on the cross-attention scores, it enhances the global feature vector, 2D visual feature vector, and semantic embedding vector. Taking the enhancement of the semantic embedding vector as an example, the semantic embedding vector... As a query, the global feature vector and 2D visual feature vectors These serve as the key and value, respectively. Attention scores are calculated.

[0052] Based on this, it is possible to learn which part of 3D geometry (hexagonal outline) or which region of 2D vision (the hexagonal top surface with specular reflection) is most strongly correlated with a certain word in the text (such as "hexagonal head"). This process is performed on all modal pairs, thereby enhancing the global feature vector, 2D visual feature vector, and semantic embedding vector.

[0053] 2. Gating fusion stage: First, the enhanced global feature vector, 2D visual feature vector, and semantic embedding vector are fed into a gated multimodal unit for concatenation to obtain a context-aware vector. ).

[0054] Subsequently, the gate vector is computed through a fully connected layer and a sigmoid activation function. Gating vector It includes three weights , respectively, correspond to the weights of the global feature vector, the 2D visual feature vector, and the semantic embedding vector. The sum of these three factors is 1. These weights are dynamically generated based on the current input and reflect the importance judgment of each modality in this specific embodiment. The importance judgment of each modality in this specific instance is calculated based on the context-aware vector and the gating vector.

[0055] Fusion Embedded Vectors By analyzing context-aware feature vectors ( We obtain the weighted summation as follows:

[0056] The two-stage design of attention followed by gating in this embodiment is key to achieving performance improvements, and its logic has a profound causal relationship. The first stage, cross-attention, allows different modalities to exchange opinions and share knowledge, enabling each modality's representation to absorb contextual information from other modalities, becoming more comprehensive. For example, the word "stainless steel" in text can be associated with the metallic luster in a 2D image and the smooth surface in 3D geometry through the attention mechanism, thereby enhancing the representation capabilities of all modalities.

[0057] The second stage of gated fusion is a final decision made after sufficient information exchange. Based on the enriched information, it determines which modality's information is most reliable and discriminative for the specific component instance. For example, for a standard bolt with a simple shape and no special visual features, the gating mechanism might learn to assign higher weights to 3D geometry and textual descriptions; while for a part with unique visual markers, such as color coding or surface defects, the gating mechanism might dynamically increase the weights of 2D visual features. This is a completely data-driven, adaptive fusion strategy dependent on specific inputs.

[0058] The vector database is used to store fusion embedding vectors, which are associated with unique identifiers of components.

[0059] The hierarchical 3D geometric feature extractor, global context-aware 2D visual feature extractor, and domain-adaptive text feature extractor are also used to extract the global feature vector, 2D visual feature vector, and semantic embedding vector of the parts to be retrieved. The gated cross-attention fusion module is also used to fuse the global feature vector, and / or 2D visual feature vector, and / or 2D semantic embedding vector of the parts to be retrieved to obtain the query vector.

[0060] It should be noted that in actual use, when a user needs to find a component, query information for any modality can be provided.

[0061] The query module is used to search the vector database based on the query vector to obtain a list of parts similar to the part to be retrieved.

[0062] Furthermore, this embodiment also performs joint training on the hierarchical 3D geometric feature extractor, the global context-aware 2D visual feature extractor, and the domain-adaptive text feature extractor. The training method is trimodal contrastive training. Design Principle: To enable the system to perform retrieval tasks, its output embedding space must possess a specific topological structure: similar items are close to each other, while dissimilar items are far apart. Contrastive learning is currently the state-of-the-art training paradigm for achieving this goal. Research by TriCoLo et al. provides a direct and effective implementation reference for contrastive learning in trimodal scenarios. Working mechanism: The system is trained on batches containing a large number of data triples, and each triple has three modality vectors ( They all correspond to the same industrial component. .

[0063] Total loss function It is the sum of three bimodal contrast losses: The purpose of calculating bimodal contrast loss is to align vectors from two modalities, such as whether a text modality vector and an image vector correspond to the same part. By calculating the differences between each pair of vectors, the loss of the three modalities is measured, thereby aligning the vectors of the three modalities to the same vector space.

[0064] Each bimodal loss Both employ the symmetric NT-Xent loss function (NormalizedTemperature-scaled Cross Entropy Loss). Specifically, for any modality in the batch... The anchor embedding vector, which corresponds to the modality Embeddings of the same component are considered positive samples. All other components in the batch originating from the same modality are considered positive samples. The embedded parts are considered negative samples.

[0065] The goal of the loss function is to maximize the cosine similarity between positive sample pairs while minimizing the cosine similarity between the anchor point and all negative samples. Mathematically, for a given modality... arrive Loss items The calculation formula is as follows:

[0066] in, It is an embedded vector. and It is an index within the batch. It refers to the batch size. It is an adjustable temperature hyperparameter.

[0067] By minimizing this total loss, the system is driven to learn a unified embedding space in which representations of the same object in three different modalities are brought together, while representations of different objects are pushed apart.

[0068] Training strategy: To ensure the effectiveness of contrastive learning, a large batch size is used, as this provides a sufficient number of negative samples to help the model learn more discriminative feature boundaries. Furthermore, data augmentation techniques are extensively applied during training, such as random rotation, jittering, and translation of point clouds, and random cropping, color transformation, and flipping of 2D images, to improve the robustness and generalization ability of the representations learned by the model.

[0069] This embodiment includes three dedicated feature extraction subsystems: (1) a hierarchical 3D geometric feature extractor based on point cloud analysis for capturing the three-dimensional structure of objects; (2) a global context-aware 2D visual feature extractor using the Transformer model with global context awareness for analyzing the appearance, texture, and material of objects; and (3) a domain-adaptive text feature extractor finely tuned on a professional technical corpus for understanding precise technical descriptions. The feature vectors extracted by these three extractors are integrated through a gated cross-attention fusion module. The gated cross-attention fusion module can dynamically weigh and combine information from different modalities. The entire industrial parts trimodal fine-grained retrieval system is trained end-to-end using a trimodal contrastive loss function, thereby explicitly optimizing the learned embedding space, making it particularly suitable for similarity-based retrieval tasks.

[0070] Example 3 This embodiment provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction, which is loaded and executed by the processor to implement the three-modal refined retrieval method for industrial parts of Embodiment 1.

[0071] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0072] The memory can be used to store the computer program or module, and the processor runs or executes the stored program or module. Computer programs or modules stored in the memory, and the data stored in the memory, are implemented. The device reading method described herein has various functions. The memory may mainly include a program storage area and a data storage area. The stored program area can store the operating system, at least one application program required for a function, etc.; it can also store data. The storage area can store data generated based on the phone's usage. Furthermore, the storage can include high-speed random access memory. Access memory can also include non-volatile memory, such as hard disks, RAM, and plug-in hard disks. It can be a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0073] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0074] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0075] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An industrial parts trimodal refined search method characterized in that, The method comprises the following steps: obtaining a point cloud of a part, extracting a global feature vector from the point cloud, and obtaining the point cloud from a CAD model or 3D scanning data of the part; obtaining a 2D image of the part, and extracting a 2D visual feature vector from the 2D image; obtaining text information of the part, and extracting a semantic embedding vector from the text information; fusing the global feature vector, the 2D visual feature vector, and the semantic embedding vector to obtain a fused embedding vector; associating the fused embedding vector with a unique identifier of the part to construct a vector database; extracting a query vector from a part to be searched, and searching the vector database based on the query vector to obtain a list of similar parts to the part to be searched.

2. The industrial parts trimodal refined search method of claim 1, wherein, The step of obtaining a point cloud of a part and extracting a global feature vector from the point cloud comprises the following steps: using a farthest point sampling algorithm to select a subset from the obtained point cloud as a local region centroid; centering on each centroid, finding all nearby points in a spherical space with a predetermined radius around the center by means of a spherical query, and constructing a plurality of local region point sets from all the nearby points; normalizing the points of the local region point set to the left of the centroid, and extracting a global feature vector by using a PointNet network.

3. The industrial parts trimodal refined search method of claim 1, wherein, The step of obtaining a 2D image of a part and extracting a 2D visual feature vector from the 2D image comprises the following steps: dividing the 2D image into a plurality of non-overlapping local windows; performing self-attention calculation on the local windows; performing multiple shifts on the local windows; performing self-attention calculation on the local windows after each shift; performing feature transformation on the self-attention calculation results before and after the shift to obtain a 2D visual feature vector.

4. The industrial parts trimodal refined search method of claim 1, wherein, The step of fusing the global feature vector, the 2D visual feature vector, and the semantic embedding vector to obtain a fused embedding vector comprises the following steps: calculating cross-attention of the global feature vector, the 2D visual feature vector, and the semantic embedding vector respectively; performing enhancement processing on the global feature vector, the 2D visual feature vector, and the semantic embedding vector based on the cross-attention score; splicing the global feature vector, the 2D visual feature vector, and the semantic embedding vector after the enhancement processing to obtain a context-aware vector; calculating a gating vector; calculating the fused embedding vector according to the context-aware vector and the gating vector.

5. An industrial parts trimodal refined search system characterized by, The method for implementing the industrial part three-modal fine retrieval method according to any one of claims 1 to 4 comprises: a hierarchical 3D geometric feature extractor for obtaining a point cloud of a part and extracting a global feature vector from the point cloud; a global context-aware 2D visual feature extractor for obtaining a 2D image of the part and extracting a 2D visual feature vector from the 2D image; a domain adaptive text feature extractor for fusing the global feature vector, the 2D visual feature vector, and the semantic embedding vector to obtain a fused embedding vector; a gated cross-attention fusion module for fusing the global feature vector, the 2D visual feature vector, and the semantic embedding vector to obtain a fused embedding vector; a vector database for storing the fused embedding vector, wherein the fused embedding vector is associated with a unique identifier of the part; The hierarchical 3D geometric feature extractor, the global context-aware 2D visual feature extractor, and the domain adaptive text feature extractor are further configured to extract a global feature vector, a 2D visual feature vector, and a semantic embedding vector of the part to be retrieved, and the gated cross-attention fusion module is further configured to fuse the global feature vector, the 2D visual feature vector, and / or the 2D semantic embedding vector of the part to be retrieved to obtain a query vector. The query module is configured to search in the vector database based on the query vector to obtain a list of similar parts to the part to be retrieved.

6. The industrial parts trimodal refined retrieval system of claim 5, wherein, The hierarchical 3D geometric feature extractor includes a plurality of set abstraction layers, and each set abstraction layer includes: a sampling layer configured to select a subset from the acquired point cloud as a centroid of a local region by using a furthest point sampling algorithm; a grouping layer configured to find all neighboring points in a spherical space with a predetermined radius around the centroid by using a ball query method, and construct a plurality of local region point sets from the neighboring points; a PointNet layer configured to normalize points of the local region point sets relative to the centroid, and extract a global feature vector by using a PointNet network.

7. The industrial parts trimodal refined retrieval system of claim 5, wherein, The domain adaptive text feature extractor is obtained by training a pre-trained BERT model.

8. The industrial parts trimodal refined retrieval system of claim 7, wherein, In the process of training the pre-trained BERT model, a frequency truncation method is used to add new word pieces.

9. The industrial parts trimodal refined retrieval system of claim 5, wherein, Further comprising: jointly training the hierarchical 3D geometric feature extractor, the global context-aware 2D visual feature extractor, and the domain adaptive text feature extractor.

10. An electronic device, comprising: The industrial part three-modal fine retrieval method includes a processor and a memory, and the memory stores at least one instruction, which is loaded and executed by the processor to implement the industrial part three-modal fine retrieval method according to any one of claims 1 to 4.

Images