A method for training a large-scale multimodal retrieval model based on pseudo-label triplet data augmentation.

By constructing a pseudo-label hierarchical tree and conducting closed-loop iterative training of a large multimodal retrieval model, the problems of scarce high-dimensional triplet data and pseudo-associations in industrial scenarios are solved, realizing the automation and continuous optimization of high-precision industrial parts retrieval.

CN123069280APending Publication Date: 2026-11-10ROOTCLOUD TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611374060.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-04
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing technologies face challenges in industrial-grade scenarios, including a lack of high-dimensional triplet data, pseudo-associations and quality noise during triplet construction, and a lack of closed-loop governance in static training mechanisms. These issues result in poor model performance in high-precision industrial parts retrieval.

Method used

By constructing a pseudo-label hierarchical tree, utilizing the features of industrial text descriptions and 3D models, and expanding it into pseudo-label triples, a closed-loop iterative training mechanism is formed through confidence scoring and supervised fine-tuning using a multimodal retrieval large model, thus solving the problems of data scarcity and noise.

Benefits of technology

It enables the automated expansion of high-quality triplet data without requiring extensive manual annotation, improving the model's cross-modal retrieval capabilities and accuracy, reducing annotation costs, enhancing adaptability to industrial scenarios, and establishing a continuously evolving data flywheel mechanism.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123069280A_ABST
    Figure CN123069280A_ABST
Patent Text Reader

Abstract

This invention relates to the field of model training technology and discloses a method for training a large-scale multimodal retrieval model based on pseudo-labeled triplet data expansion. The method includes: collecting binary data composed of an industrial text description library and a 3D model; extracting and fusing features from the binary data to construct a pseudo-labeled hierarchical tree; collecting image data from an industrial environment and associating it with the pseudo-labeled hierarchical tree to expand the binary data into pseudo-labeled triplets composed of industrial text descriptions, industrial environment images, and 3D models; performing confidence scoring on the pseudo-labeled triplets and filtering samples; and inputting the supervised fine-tuned large-scale multimodal retrieval model into the process of expanding binary data into pseudo-labeled triplets, forming a closed-loop iteration. This invention expands binary data into triplet data, overcoming the bottleneck of scarce triplet data, and performs supervised fine-tuning of the large-scale multimodal retrieval model, forming a positive data flywheel of model optimization data and data-driven model, achieving closed-loop self-upgrading.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of model training technology, specifically to a method for training large multimodal retrieval models based on pseudo-label triplet data augmentation. Background Technology

[0002] With the rapid development of generative artificial intelligence and multimodal learning, cross-modal retrieval technology has evolved from the traditional "text-image" retrieval to the "text-image-3D" trimodal alignment stage. In the industrial field, achieving seamless retrieval between text, real-world images, and 3D CAD models has significant application value for intelligent manufacturing, digital warehousing, and parts management.

[0003] In existing technologies, there are mainly three technical paths: (1) An extension scheme based on pre-trained spatial alignment (such as OpenShape) is adopted: This scheme utilizes large-scale 3D datasets (such as Objaverse) to align 3D modalities with existing image and text representations by projecting 3D shape features into the pre-trained CLIP (Contrastive Language-Image Pre-training, a pre-trained model based on contrastive text-image pairs) image / text feature space. Its technical implementation usually focuses on solving the encoding problem of 3D geometric features, enabling the model to have zero-shot recognition capability; (2) Three-modal feature alignment schemes: (e.g., TAMM): e.g. Figure 1 As shown, this scheme aims to build a unified text-image-3D alignment framework, which typically includes three independent encoders. By introducing a triplet contrastive loss, the distance between text descriptions, 2D industrial images, and 3D models of the same object in the common vector space is explicitly reduced during training. (3) Cross-modal enhancement scheme based on generative large model: This scheme is based on diffusion model architecture. Some schemes attempt to utilize the generalization ability of large model to simulate real-shot images through known 3D model renderings to make up for the lack of real samples.

[0004] Although solutions such as OpenShape and TAMM have made significant progress in general applications, they still face serious performance bottlenecks and implementation challenges in industrial-level scenarios. These challenges mainly manifest in the extreme scarcity of high-dimensional triplet data (text-image-3D), pseudo-associations and quality noise during triplet construction, and the lack of closed-loop governance in static training mechanisms.

[0005] Against this backdrop, how to construct a positive iterative training mechanism to meet the practical needs of high-precision industrial parts retrieval has become an urgent problem to be solved. Summary of the Invention

[0006] This invention provides a method for training a large multimodal retrieval model based on pseudo-label triplet data augmentation, in order to solve one of the problems in the prior art.

[0007] In a first aspect, the present invention provides a method for training a large multimodal retrieval model based on pseudo-label triplet data augmentation, the method comprising: Collect binary data consisting of an industrial text description library and the corresponding 3D model, extract and fuse the features of the binary data, and construct a pseudo-label hierarchical tree. Collect image data in industrial environments, associate the image data in industrial environments with a pseudo-label hierarchical tree, and expand the binary data into pseudo-label triples composed of industrial text descriptions, images in industrial environments, and 3D models. A multimodal retrieval model is used to score the confidence of pseudo-label triples and filter samples. The multimodal retrieval model is fine-tuned under supervision using the filtered samples. The fine-tuned multimodal retrieval model is then used to expand binary data into pseudo-label triples, forming a closed-loop iteration.

[0008] This invention constructs a pseudo-label hierarchical tree by integrating features from an industrial text description library and a 3D model. This provides interpretable and measurable prior anchors for subsequent cross-modal association. By associating image data from an industrial environment with the pseudo-label hierarchical tree, sparse binary data is expanded into high-quality triple data, overcoming the bottleneck of scarce triple data. Confidence scores are used for sample filtering to avoid interference from noisy samples, ensuring the quality of model training data. Supervised fine-tuning of the large multimodal retrieval model is performed, feeding back into data expansion and forming a positive data flywheel of model optimization data and data-driven model. This achieves closed-loop self-upgrading without human intervention, improving model performance.

[0009] In one optional implementation, features of the binary data are extracted and fused to construct a pseudo-label hierarchical tree, including: Semantic features of an industrial text description library were extracted using a text editor. A 3D topology encoder is used to extract the geometric features of the 3D model corresponding to the industrial text description library; Hierarchical clustering is performed by integrating semantic and geometric features; The agglomerative tree output by the HDBSCAN algorithm is used to construct a pseudo-label hierarchical tree from the hierarchical clustering results. The nodes of the pseudo-label hierarchical tree represent industrial components with similar geometric or semantic features.

[0010] This invention extracts semantic features from an industrial text description library using a text editor to capture information such as the functional attributes of industrial parts. It extracts geometric features from a 3D topology encoder to deeply perceive the physical morphological differences of industrial parts. It integrates semantic and geometric features and performs hierarchical clustering to avoid clustering bias caused by relying solely on geometric similarity or text semantics. It constructs a pseudo-label hierarchical tree using the HDBSCAN algorithm to provide interpretable structured anchor points for subsequent image association.

[0011] In one alternative implementation, the image data is associated with a pseudo-label hierarchy tree, and the binary data is expanded into pseudo-label triples consisting of industrial text descriptions, images of industrial environments, and 3D models, including: An image encoder is used to extract image features from image data in an industrial environment. The image features of the image data in the industrial environment are then mapped to a pseudo-label hierarchical tree. The cosine similarity between the image features and the node features of the pseudo-label hierarchical tree is calculated. Based on the cosine similarity calculation results, image data in the industrial environment is associated with the corresponding nodes of the pseudo-label hierarchical tree, and the binary data is expanded into pseudo-label triples. The pseudo-label triples consist of industrial text description, images in the industrial environment, and 3D models.

[0012] This invention maps image features of image data in an industrial environment to a pseudo-label hierarchical tree, and uses a cosine similarity matching mechanism to associate the image data in the industrial environment with the corresponding nodes in the pseudo-label hierarchical tree. This embeds the originally isolated image data in the industrial environment into cross-modal semantic relationships, forming pseudo-label triples, thus completing the expansion from binary to triple and solving the problem of data silos in industrial scenarios.

[0013] In one optional implementation, a multimodal retrieval model is used to score the confidence of pseudo-label triples and filter samples, including: The ternary contrastive loss function is used to calculate the sum of the cosine distances in the common vector space of the pairwise modal features of industrial text description and image in industrial environment, industrial text description and 3D model, and image and 3D model in industrial environment in pseudo-label triplet; Samples whose sum of cosine distances exceeds a preset distance range are identified as noise and removed.

[0014] This invention calculates the sum of the cosine distances between pairwise modal features in the common vector space of pseudo-label triplets using a ternary contrast loss function, identifies noise, and removes noise. This effectively eliminates low-quality pseudo-labels caused by factors such as geometric similarity interference and domain gaps, ensuring the reliability of the dataset.

[0015] In one alternative implementation, supervised fine-tuning of the large multimodal retrieval model is performed using the filtered samples, including: The filtered samples are used as input data for training the multimodal retrieval model. The triples with the same pseudo-label are considered positive samples, and the triples with different pseudo-labels are considered negative samples. A supervised contrastive learning fine-tuning mechanism is adopted to adjust the network weight parameters of the text encoder, image encoder, and 3D encoder in the multimodal retrieval model. The optimization objective is to minimize the spatial distance between positive sample triples and maximize the distance between negative samples.

[0016] This invention employs a supervised contrastive learning fine-tuning mechanism to perform supervised fine-tuning on a large multimodal retrieval model, achieving adaptive alignment across modal feature spaces. This enables the model to possess stronger fine-grained cross-modal recognition capabilities, providing a core driving force for the closed-loop iteration of the data flywheel.

[0017] In one optional implementation, the supervised fine-tuned multimodal retrieval model is fed into the process of expanding binary data into pseudo-labeled triples, including: The multimodal retrieval model with supervised fine-tuning is used to filter candidate pseudo-label triples. The filtered candidate pseudo-label triples are then used as input data for the next round of training of the multimodal retrieval model, and a new round of supervised fine-tuning is performed on the multimodal retrieval model.

[0018] This invention improves the capabilities of the multimodal retrieval model by feeding the supervised fine-tuned multimodal retrieval model into the sample filtering of pseudo-label triples to form a positive enhancement loop.

[0019] Secondly, the present invention provides a multimodal retrieval large model training device based on pseudo-label triplet data augmentation, the device comprising: The module is used to collect binary data consisting of an industrial text description library and the corresponding 3D model, extract and fuse the features of the binary data, and construct a pseudo-label hierarchical tree. The expansion module is used to collect image data in industrial environments, associate the image data with a pseudo-label hierarchy tree, and expand the binary data into pseudo-label triples composed of industrial text descriptions, images in industrial environments, and 3D models. The sample filtering module is used to perform confidence scoring on pseudo-label triples using a multimodal retrieval model, thereby filtering the samples. The fine-tuning module is used to perform supervised fine-tuning of the multimodal retrieval model using the filtered samples. The supervised fine-tuned multimodal retrieval model is then fed into the process of expanding binary data into pseudo-label triples, forming a closed-loop iteration.

[0020] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the multimodal retrieval large model training method based on pseudo-label triplet data augmentation described in the first aspect or any corresponding embodiment.

[0021] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the multimodal retrieval large model training method based on pseudo-label triplet data augmentation described in the first aspect or any corresponding embodiment thereof.

[0022] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the multimodal retrieval large model training method based on pseudo-label triplet data augmentation as described in the first aspect or any corresponding embodiment. Attached Figure Description

[0023] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of a three-modal feature alignment scheme in the prior art; Figure 2 This is a flowchart illustrating the multimodal retrieval large model training method based on pseudo-label triplet data augmentation according to an embodiment of the present invention. Figure 3 This is a schematic diagram of the pseudo-label triplet construction process according to an embodiment of the present invention; Figure 4 This is a schematic diagram of a multimodal retrieval large model data flywheel architecture according to an embodiment of the present invention; Figure 5 This is a structural block diagram of a multimodal retrieval large model training device based on pseudo-label triplet data augmentation according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention in an appropriate manner in accordance with relevant laws and regulations, and their permission should be obtained.

[0027] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0028] Although solutions such as OpenShape and TAMM have made significant progress in general-purpose applications, they still face serious performance bottlenecks and implementation challenges in industrial-grade scenarios, mainly in the following aspects: (1) The "text-image-3D" high-dimensional triplet data is extremely scarce: Model training relies heavily on the complete triplet data of "high-quality text description-high-definition real-shot image-accurate 3D model". However, in the real industrial environment, such strongly aligned triplet data is extremely scarce. Although there are a large number of text-image pairs on the Internet, industrial-grade 3D CAD models are mostly core assets of enterprises and are extremely difficult to obtain through web crawling. Existing open-source 3D datasets (such as ShapeNet or Objaverse) are mostly civilian objects and lack professional industrial parts models. In industrial reality, these three types of data exist in completely isolated systems: 3D models (CAD / point cloud) exist in the Product Lifecycle Management (PLM) system, real-shot images exist in production line quality inspection records or digital warehouse logs, and text descriptions exist in the Bill of Materials (BOM) or technical specifications. Due to the lack of a unified association identifier, there is an extreme scarcity of strongly aligned triplet samples that can simultaneously satisfy "CAD of a specific industrial part - real-life photos of the corresponding environment - professional engineering semantics" in a natural state. (2) "Pseudo-associations" and quality noise in the construction of triples: "Pseudo-associations" refer to the incorrect matching relationships generated when automatically constructing "text-image-3D" unit groups. When attempting to automatically construct triples, existing technologies are prone to generating a large number of low-quality samples due to the lack of an effective cross-modal verification mechanism. This includes two aspects: First, geometric similarity interference: Industrial standard parts (such as bearings and bolts of different models) are highly similar in geometric structure. Traditional automatic association algorithms often mismatch similar 3D models with real-world images of different specifications. This "pseudo-labeling" causes feature collapse during model training, making it impossible to distinguish fine-grained differences in industrial features. Secondly, there is the domain gap: industrial real-world images often have complex lighting, oil stains, occlusions, and background interference. For example, real-world images of production lines may have oil stains and occlusions, while 3D models and their renderings are too idealized, leading to the incorrect association of "real-world image of model A" with "3D model B", resulting in "pseudo-labeling / pseudo-associations". This huge visual difference makes it difficult for simple binary expansion to form effective triple mapping, resulting in extremely poor model performance in real retrieval scenarios. (3) Static training mechanisms lack closed-loop governance: Existing solutions (such as OpenShape) mostly adopt a one-time training mode, which lacks the ability to self-evolve on low-quality data. Once the initial triple candidate set contains erroneous associations, the model will develop "illusions" and continue to learn incorrect feature alignments. There is a lack of an iterative architecture in the industry that can feed the "model retrieval results" back to the "data cleaning process", making it difficult for existing technologies to improve the association accuracy of triples while ensuring the data scale.

[0029] In summary, existing technologies exhibit significant imbalances in the large-scale generation of high-quality industrial triplet data, accurate cross-modal semantic alignment, and the ability to identify fine-grained industrial features. Currently, the industry lacks a training mechanism capable of automatically utilizing massive amounts of weakly labeled data to construct a positively iterative "data flywheel," making it difficult for existing technologies to meet the practical needs of high-precision industrial parts retrieval with low labeling costs.

[0030] This invention provides a method for training a large multimodal retrieval model based on pseudo-label triplet data expansion. The key features are: (1) introducing an automatic pseudo-label triplet generation mechanism to break the limitation on the number of modalities, constructing a hierarchical prior through "text-model" binary pairs, and forcibly associating massive images through feature space mapping, thus solving the problem of expanding industrial triplet data from scratch; (2) semantic-geometric collaborative representation based on hierarchical tree: using text to cluster the model and construct a hierarchical tree, which not only contains semantic information but also implies the hierarchical structure of industrial logic (e.g., fastener-bolt-internal hex bolt), improving the logical accuracy of pseudo-label generation; (3) industrial multimodal data flywheel architecture: proposing an iterative process of mutual enhancement between model and data, using the initially trained model as a filter, and through continuous screening and fine-tuning, enabling the model to automatically adapt to industrial scenarios without a large amount of manual annotation.

[0031] According to an embodiment of the present invention, a method for training a large multimodal retrieval model based on pseudo-label triplet data augmentation is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0032] This embodiment provides a method for training a large multimodal retrieval model based on pseudo-label triplet data augmentation. Figure 2 This is a flowchart of a multimodal retrieval large model training method based on pseudo-label triplet data augmentation according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps: Step S201: Collect binary data consisting of an industrial text description library and the corresponding 3D model, extract and fuse the features of the binary data, and construct a pseudo-label hierarchical tree.

[0033] In embodiments of the present invention, such as Figure 3 As shown, text descriptions from the industrial field (such as material texts in a BOM system) are collected, namely an industrial text description library, and the corresponding 3D model of the industrial text description library. The industrial text description library and the corresponding 3D model of the industrial text description library form a binary data set. The encoder is used to extract the features of the binary data set, and the extracted features are fused to construct a pseudo-label hierarchical tree.

[0034] Step S202: Collect image data in the industrial environment, associate the image data in the industrial environment with the pseudo-label hierarchical tree, and expand the binary data into pseudo-label triples composed of industrial text description, images in the industrial environment, and 3D models.

[0035] In embodiments of the present invention, such as Figure 3 As shown, a large amount of unlabeled or weakly labeled image data in industrial environments is collected, and this image data is associated with a pseudo-label hierarchical tree to expand the binary data and form a pseudo-label triple consisting of industrial text description, images in industrial environments, and 3D models.

[0036] Step S203: Use a multimodal retrieval model to score the confidence of pseudo-label triples and filter the samples.

[0037] In this embodiment of the invention, a multimodal retrieval model is used to score the confidence of the generated pseudo-label triples, and to filter samples to remove noisy samples that are too far away or have incorrect associations.

[0038] Step S204: The multimodal retrieval model is supervised and fine-tuned using the filtered samples. The supervised and fine-tuned multimodal retrieval model is then used to expand the binary data into pseudo-label triples, forming a closed-loop iteration.

[0039] In this embodiment of the invention, the multimodal retrieval model is supervised fine-tuning using filtered samples, enabling the multimodal retrieval model to deeply understand the cross-modal correlation features of industrial parts. The optimized multimodal retrieval model has stronger feature recognition capabilities and is then fed back into the process of expanding binary data into pseudo-label triples. Complex pseudo-label triples are filtered more accurately. This process of "model optimizing data - data driving model" forms a closed-loop iteration, achieving continuous improvement in model performance.

[0040] This embodiment provides a multimodal retrieval large-scale model training method based on pseudo-label triplet data augmentation. By integrating features from an industrial text description library and a 3D model, a pseudo-label hierarchical tree is constructed, providing interpretable and measurable prior anchors for subsequent cross-modal association. By associating image data from an industrial environment with the pseudo-label hierarchical tree, sparse binary data is augmented into high-quality triplet data, overcoming the bottleneck of triplet data scarcity. Confidence scoring is used for sample filtering to avoid interference from noisy samples, ensuring the quality of model training data. Supervised fine-tuning of the multimodal retrieval large-scale model is performed, feeding back into data augmentation and forming a positive data flywheel of model optimization data and data-driven model, achieving closed-loop self-upgrading without manual intervention and improving model performance.

[0041] This embodiment provides a method for training a large multimodal retrieval model based on pseudo-label triplet data augmentation. The process includes the following steps: Step S301: Collect binary data consisting of an industrial text description library and the corresponding 3D model, extract and fuse the features of the binary data, and construct a pseudo-label hierarchical tree.

[0042] Specifically, the step S301 above, which involves extracting and fusing features from the binary data to construct a pseudo-label hierarchical tree, includes: Step S3011: Use a text editor to extract semantic features from the industrial text description library; Step S3012: Use a 3D topology encoder to extract the geometric features of the 3D model corresponding to the industrial text description library; Step S3013: Integrate semantic features and geometric features to perform hierarchical clustering; Step S3014: The agglomerative tree output by the HDBSCAN algorithm is used to construct a pseudo-label hierarchical tree from the hierarchical clustering results. The nodes of the pseudo-label hierarchical tree represent industrial components with similar geometric or semantic features.

[0043] In embodiments of the present invention, such as Figure 3 As shown, a pre-trained CLIP is used as the basic text encoder, and a text editor is used to extract semantic features from the industrial text description library. A 3D topology encoder is used to extract the geometric features of the corresponding 3D model of the industrial text description library. The semantic and geometric features are fused, and HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) is used for hierarchical clustering according to distance metrics. For example, the first layer clusters into the large category "Fasteners", the second layer subdivides into "Bolts" and "Nuts" based on geometry and semantics, and the third layer further refines to the leaf nodes "Hex Bolts" or "M8 Internal Hex Bolts".

[0044] like Figure 3 As shown, the agglomerative tree output by the HDBSCAN algorithm transforms the hierarchical clustering results into a pseudo-label hierarchical tree structure, constructing a pseudo-label hierarchical tree where the nodes represent industrial components with similar geometric or semantic features.

[0045] Specifically, each node in the pseudo-label hierarchical tree contains a group, which aggregates a set of "standard text description-3D model" binary pairs with high-dimensional semantic and geometric similarity. These nodes serve as "anchors" to receive, align, and revert massive amounts of unlabeled industrial real-world images, enabling cross-modal bridging from "binary pairs" to "triple pairs".

[0046] Semantic features are extracted from an industrial text description library using a text editor to capture information such as the functional attributes of industrial parts. Geometric features of 3D models are extracted using a 3D topology encoder to deeply perceive the physical morphological differences of industrial parts. Semantic and geometric features are fused and hierarchical clustering is performed to avoid clustering bias caused by relying solely on geometric similarity or text semantics. A pseudo-label hierarchical tree is constructed using the HDBSCAN algorithm to provide interpretable structured anchor points for subsequent image association.

[0047] Step S302: Collect image data in the industrial environment, associate the image data in the industrial environment with the pseudo-label hierarchical tree, and expand the binary data into pseudo-label triples composed of industrial text description, images in the industrial environment, and 3D models.

[0048] Specifically, in step S302 above, the image data is associated with the pseudo-label hierarchy tree, and the binary data is expanded into pseudo-label triples composed of industrial text descriptions, images in the industrial environment, and 3D models, including: Step S3021: Use an image encoder to extract image features from image data in an industrial environment, map the image features of the image data in an industrial environment to a pseudo-label hierarchical tree, and calculate the cosine similarity between the image features and the node features of the pseudo-label hierarchical tree. Step S3022: Based on the cosine similarity calculation result, the image data in the industrial environment is associated with the corresponding node of the pseudo-label hierarchical tree, and the binary data is expanded into pseudo-label triples. The pseudo-label triples consist of industrial text description, images in the industrial environment, and 3D models.

[0049] In embodiments of the present invention, such as Figure 3 As shown, a pre-trained image encoder (such as CLIP-Vision-Transformer) is used to extract image features from massive unlabeled or weakly labeled image data in industrial environments, and the image features are mapped to the constructed pseudo-label hierarchical tree.

[0050] The cosine similarity between image features and node features of the pseudo-label hierarchical tree is calculated. The greater the similarity, the more similar the image features and node features of the pseudo-label hierarchical tree are. The method of calculating cosine similarity is the same as that used in the prior art, and will not be repeated here.

[0051] Based on the cosine similarity calculation results, a decision strategy is selected to determine the target node of the pseudo-label hierarchical tree for image data association, thereby expanding the binary data and ultimately forming a text-image-3D triplet pseudo-label sample library composed of "original industrial text description - associated real-world images - original 3D models". The decision strategy includes a maximum value allocation strategy and a threshold combined with a maximum value allocation strategy. The maximum value allocation strategy directly selects the node with the highest similarity as the target node. The threshold combined with a maximum value allocation strategy sets a similarity threshold; if the highest similarity is greater than or equal to the similarity threshold, the node is assigned; if the highest similarity is less than the similarity threshold, the node is rejected.

[0052] This approach enables automated, unsupervised data augmentation that breaks modal limitations. Strongly aligned text-image-3D triples are extremely scarce in industrial environments, while text-3D binary pairs (from PLM / BOM) and weakly labeled images (from quality inspection / warehousing) are relatively abundant. The goal is to use a hierarchical tree as a medium to automatically categorize massive amounts of scattered images into corresponding text-3D nodes, thereby achieving low-cost, large-scale batch synthesis of high-dimensional text-image-3D triple candidate sets.

[0053] By mapping the image features of image data in industrial environments to a pseudo-label hierarchical tree, and using a cosine similarity matching mechanism to associate the image data in industrial environments with the corresponding nodes in the pseudo-label hierarchical tree, the originally isolated image data in industrial environments is embedded into cross-modal semantic relationships, forming pseudo-label triples, completing the expansion from binary to triple, and solving the problem of data silos in industrial scenarios that are difficult to connect.

[0054] Step S303: Use a multimodal retrieval model to score the confidence of pseudo-label triples and filter the samples.

[0055] Specifically, step S303 includes: Step S3031: The ternary contrast loss function is used to calculate the sum of the cosine distances in the common vector space between the pairwise modal features of the industrial text description and the image in the industrial environment, the industrial text description and the 3D model, and the image in the industrial environment and the 3D model in the pseudo-label triplet. Step S3032: Samples whose sum of cosine distances exceeds a preset distance range are identified as noise and removed.

[0056] In this embodiment of the invention, a multimodal retrieval big model is used to score the confidence of pseudo-label triples. The confidence score is based on the joint feature distance of the triples measured by the current multimodal retrieval big model (version N).

[0057] The Triplet Contrastive Loss function is used to calculate the sum of cosine distances between pairwise modal features in the common vector space of pseudo-label triplets. Specifically, this includes the sum of cosine distances between pairwise modal features in the common vector space of images in an industrial environment, industrial text descriptions and 3D models, and images in an industrial environment and 3D models. First, the cosine similarity between pairwise modal features is calculated. Then, the difference between 1 and the cosine similarity is determined as the cosine distance. Finally, the cosine distances are added together to obtain the sum of the cosine distances between pairwise modal features.

[0058] If the sum of the cosine distances exceeds the preset distance range, it is determined that the pseudo-label sample is noise, the confidence score is low, and it is removed from the candidate set.

[0059] The ternary contrastive loss function is used to calculate the sum of the cosine distances between the pairwise modal features in the common vector space of the pseudo-label triplet. Noise is identified and removed, effectively eliminating low-quality pseudo-labels caused by factors such as geometric similarity interference and domain gap, thus ensuring the reliability of the dataset.

[0060] Step S304: The multimodal retrieval model is supervised and fine-tuned using the filtered samples. The supervised and fine-tuned multimodal retrieval model is then used to expand the binary data into pseudo-label triples, forming a closed-loop iteration.

[0061] Specifically, step S304 above, which involves supervised fine-tuning of the multimodal retrieval model using the filtered samples, includes: Step S3041: The filtered samples are used as input data for training the multimodal retrieval model. The positive samples are within the same pseudo-label triplet, and the negative samples are between different pseudo-label triplets as supervision information. A supervised contrastive learning fine-tuning mechanism is adopted to adjust the network weight parameters of the text encoder, image encoder, and 3D encoder in the multimodal retrieval model, with the optimization objective being to minimize the spatial distance between positive sample triplets and maximize the distance between negative samples.

[0062] In this embodiment of the invention, the "high-confidence high-quality sample set" remaining after high-quality sample filtering is used as the training input data of the multimodal retrieval model. Text descriptions, real-life images, and 3D models within the same pseudo-label triple are considered as positive samples. Although they are different in modal form, they all point to the same industrial part and should be aligned in the vector space. Any cross-modal combination between different pseudo-label triples is considered as negative samples, that is, they point to different industrial parts and should be far apart from each other in the vector space.

[0063] A supervised contrastive fine-tuning mechanism is employed to adjust the network weight parameters of the text encoder, image encoder, and 3D encoder in the multimodal retrieval model. The core objective of the entire optimization process is to minimize the spatial distance of positive sample triples, ensuring that the multimodal representations of the same part are tightly clustered in the common vector space, and to maximize the distance of negative samples, ensuring that the representations of different parts are separated in the common vector space. Through this fine-tuning process, the multimodal retrieval model gradually learns the accurate cross-modal alignment mapping relationship of industrial parts across the three modalities of semantics, vision, and geometry.

[0064] By employing a supervised contrastive learning fine-tuning mechanism to perform supervised fine-tuning on the large multimodal retrieval model, adaptive alignment across modal feature spaces is achieved, enabling the model to possess stronger fine-grained cross-modal recognition capabilities and providing core driving force for the closed-loop iteration of the data flywheel.

[0065] Specifically, step S304 above, which involves using the supervised fine-tuned multimodal retrieval model to expand binary data into pseudo-label triples, includes: Step S3042: The supervised fine-tuned multimodal retrieval model is used to filter candidate pseudo-label triples. The filtered candidate pseudo-label triples are used as input data for the next round of training of the multimodal retrieval model, and a new round of supervised fine-tuning is performed on the multimodal retrieval model.

[0066] In embodiments of the present invention, such as Figure 4 As shown, the supervised fine-tuned multimodal retrieval model (version N+1) possesses stronger fine-grained cross-modal recognition capabilities. It is then used as a "filter" in the next round of pseudo-label triple filtering, enabling more precise filtering of more complex pseudo-label triples. This allows it to identify previously undetectable deep noise and uncover more complex samples, providing a higher-quality training set as input data for the next round of training of the multimodal retrieval model. This iterative cycle achieves autonomous closed-loop governance and performance improvement without manual annotation. The process of "model optimizing data - data driving model" forms a closed-loop iteration, resulting in continuous improvement in model performance.

[0067] By feeding the supervised fine-tuned multimodal retrieval model into the sample filtering of pseudo-label triples, a positive reinforcement loop is formed, thereby improving the capabilities of the multimodal retrieval model.

[0068] Compared with existing technologies, the multimodal retrieval large model training method based on pseudo-label triple data augmentation provided in this embodiment has the following advantages: (1) Significantly reduce annotation costs: It breaks through the limitation that it must rely on manual annotation of strongly aligned “text-image-3D” data, and can automatically synthesize training sets using massive Internet images and existing CAD models in factories, thereby reducing annotation costs; (2) Enhance the adaptability to industrial scenarios: By deeply binding the professional semantics of the industrial field with 3D geometric features through the hierarchical tree structure, the retrieval problem of "similar in shape but different in meaning" or "same in meaning but different in shape" of industrial parts is effectively solved, and the retrieval accuracy is improved; (3) Solve the pain point of scarce 3D data: Creatively transform limited 3D model assets into massive multimodal training samples through data augmentation technology, making it possible to train large industrial models with large number of parameters and high generalization ability. (4) Continuous evolution capability: The established "data flywheel" mechanism ensures that the system can continuously absorb newly generated image data and upgrade itself in actual operation, and the model will become more and more accurate as the usage time increases.

[0069] This embodiment also provides a multimodal retrieval large model training device based on pseudo-label triplet data augmentation. This device is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0070] This embodiment provides a multimodal retrieval large model training device based on pseudo-label triple data augmentation, such as... Figure 5 As shown, it includes: Module 501 is used to collect binary data consisting of an industrial text description library and the corresponding 3D model, extract and fuse the features of the binary data, and construct a pseudo-label hierarchical tree. The expansion module 502 is used to collect image data in an industrial environment, associate the image data with a pseudo-label hierarchy tree, and expand the binary data into pseudo-label triples composed of industrial text descriptions, images in an industrial environment, and 3D models. The sample filtering module 503 is used to perform confidence scoring on pseudo-label triples using a multimodal retrieval model to filter samples. The fine-tuning module 504 is used to perform supervised fine-tuning of the multimodal retrieval model using the filtered samples. The supervised fine-tuned multimodal retrieval model is then used in the process of expanding binary data into pseudo-label triples to form a closed-loop iteration.

[0071] In some alternative implementations, the construction module 501 includes: The first extraction unit is used to extract semantic features from the industrial text description library using a text editor; The second extraction unit is used to extract the geometric features of the 3D model corresponding to the industrial text description library using a 3D topology encoder. Fusion units are used to fuse semantic and geometric features for hierarchical clustering; The building unit is used to construct a pseudo-label hierarchical tree from the agglomerative tree output by the HDBSCAN algorithm. The nodes of the pseudo-label hierarchical tree represent industrial components with similar geometric or semantic features.

[0072] In some alternative implementations, the expansion module 502 includes: The third extraction unit is used to extract image features from image data in an industrial environment using an image encoder, map the image features of the image data in the industrial environment to a pseudo-label hierarchical tree, and calculate the cosine similarity between the image features and the node features of the pseudo-label hierarchical tree. The expansion unit is used to associate image data in the industrial environment with the corresponding nodes in the pseudo-label hierarchy tree based on the cosine similarity calculation results, and expand the binary data into pseudo-label triples. The pseudo-label triples consist of industrial text description, images in the industrial environment, and 3D models.

[0073] In some alternative implementations, the sample filtering module 503 includes: The computational unit is used to calculate the sum of the cosine distances in the common vector space of the pairwise modal features of industrial text description and image in industrial environment, industrial text description and 3D model, and image and 3D model in industrial environment in pseudo-label triplet using the ternary contrast loss function; The rejection unit is used to identify samples whose sum of cosine distances exceeds a preset distance range as noise and reject the noise.

[0074] In some alternative implementations, the fine-tuning module 504 includes: The adjustment unit is used to take the filtered samples as input data for training the multimodal retrieval model. It uses positive samples within the same pseudo-label triplet and negative samples between different pseudo-label triplets as supervision information. It adopts a supervised contrastive learning fine-tuning mechanism to adjust the network weight parameters of the text encoder, image encoder and 3D encoder in the multimodal retrieval model, with the optimization objective of minimizing the spatial distance of positive sample triplets and maximizing the distance of negative samples.

[0075] In some alternative implementations, the fine-tuning module 504 further includes: The new round of supervised fine-tuning unit is used to filter candidate pseudo-label triples using the supervised fine-tuned multimodal retrieval model, and use the filtered candidate pseudo-label triples as input data for the next round of training of the multimodal retrieval model to perform a new round of supervised fine-tuning of the multimodal retrieval model.

[0076] The multimodal retrieval large model training device based on pseudo-label triplet data augmentation provided in this embodiment of the invention can execute the multimodal retrieval large model training method based on pseudo-label triplet data augmentation provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method. Further functional descriptions of the above modules and units are the same as in the corresponding embodiments described above, and will not be repeated here.

[0077] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0078] The following is a detailed reference. Figure 6 This diagram illustrates a suitable structural design for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 601, which can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) 602 or a program loaded from memory 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of the electronic device. The processor 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0079] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0080] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a memory 608, or installed from a ROM 602. When the computer program is executed by the processor 601, it performs the functions defined in the multimodal retrieval large model training method based on pseudo-label triplet data augmentation according to embodiments of the present invention.

[0081] Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0082] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the multimodal retrieval large model training method based on pseudo-label triplet data augmentation shown in the above embodiments is implemented.

[0083] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0084] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and all such modifications and variations fall within the scope defined by the appended invention.

Claims

1. A method for training a large-scale multimodal retrieval model based on pseudo-label triplet data augmentation, characterized in that, The method includes: Collect binary data consisting of an industrial text description library and the corresponding 3D model; extract and fuse the features of the binary data; and construct a pseudo-label hierarchical tree. Collect image data in an industrial environment, associate the image data in the industrial environment with the pseudo-label hierarchy tree, and expand the binary data into pseudo-label triples composed of industrial text description, images in the industrial environment, and 3D models. A multimodal retrieval model is used to score the confidence of the pseudo-label triples and filter the samples. The multimodal retrieval model is fine-tuned under supervision using the filtered samples. The fine-tuned multimodal retrieval model is then used to expand binary data into pseudo-label triples, forming a closed-loop iteration.

2. The method according to claim 1, characterized in that, The step of extracting and fusing features from the binary data to construct a pseudo-label hierarchical tree includes: Semantic features of an industrial text description library were extracted using a text editor. A 3D topology encoder is used to extract the geometric features of the 3D model corresponding to the industrial text description library; The semantic features and the geometric features are fused together to perform hierarchical clustering; The agglomerative tree output by the HDBSCAN algorithm is used to construct a pseudo-label hierarchical tree from the hierarchical clustering results. The nodes of the pseudo-label hierarchical tree represent industrial components with similar geometric or semantic features.

3. The method according to claim 1, characterized in that, The step of associating the image data in the industrial environment with the pseudo-label hierarchy tree, and expanding the binary data into pseudo-label triples composed of industrial text descriptions, images in the industrial environment, and 3D models, includes: An image encoder is used to extract image features from image data in an industrial environment. The image features of the image data in the industrial environment are then mapped to a pseudo-label hierarchical tree. The cosine similarity between the image features and the node features of the pseudo-label hierarchical tree is calculated. Based on the cosine similarity calculation results, the image data in the industrial environment is associated with the corresponding node of the pseudo-label hierarchical tree, and the binary data is expanded into pseudo-label triples. The pseudo-label triples consist of industrial text description, images in the industrial environment, and 3D models.

4. The method according to claim 1, characterized in that, The process of using a multimodal retrieval model to score the confidence of the pseudo-label triples and filter samples includes: The ternary contrastive loss function is used to calculate the sum of the cosine distances in the common vector space of the pairwise modal features of industrial text description and image in industrial environment, industrial text description and 3D model, and image and 3D model in industrial environment in pseudo-label triplet; Samples whose sum of cosine distances exceeds a preset distance range are identified as noise and removed.

5. The method according to claim 1, characterized in that, The process of using filtered samples to perform supervised fine-tuning of the multimodal retrieval model includes: The filtered samples are used as input data for training the multimodal retrieval model. The triples with the same pseudo-label are considered positive samples, and the triples with different pseudo-labels are considered negative samples. A supervised contrastive learning fine-tuning mechanism is adopted to adjust the network weight parameters of the text encoder, image encoder, and 3D encoder in the multimodal retrieval model. The optimization objective is to minimize the spatial distance between positive sample triples and maximize the distance between negative samples.

6. The method according to claim 1, characterized in that, The process of using the supervised fine-tuned multimodal retrieval model to expand binary data into pseudo-labeled triples includes: The multimodal retrieval model with supervised fine-tuning is used to filter candidate pseudo-label triples. The filtered candidate pseudo-label triples are then used as input data for the next round of training of the multimodal retrieval model, and a new round of supervised fine-tuning is performed on the multimodal retrieval model.

7. A training device for a multimodal retrieval large model based on pseudo-label triplet data augmentation, characterized in that, The device includes: The construction module is used to collect binary data consisting of an industrial text description library and the corresponding 3D model, extract and fuse the features of the binary data, and construct a pseudo-label hierarchical tree. An expansion module is used to collect image data in an industrial environment, associate the image data with the pseudo-label hierarchy tree, and expand the binary data into pseudo-label triples composed of industrial text descriptions, images in an industrial environment, and 3D models. The sample filtering module is used to perform confidence scoring on the pseudo-label triples using a multimodal retrieval model, thereby filtering the samples. The fine-tuning module is used to perform supervised fine-tuning of the multimodal retrieval model using the filtered samples. The supervised fine-tuned multimodal retrieval model is then fed into the process of expanding binary data into pseudo-label triples, forming a closed-loop iteration.

8. An electronic device, characterized in that, include: The system includes a memory and a processor, which are interconnected. The memory stores computer instructions, and the processor executes the computer instructions to perform the multimodal retrieval large model training method based on pseudo-label triplet data augmentation as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the multimodal retrieval large model training method based on pseudo-label triplet data augmentation as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The method includes computer instructions for causing a computer to execute the multimodal retrieval large model training method based on pseudo-label triplet data augmentation as described in any one of claims 1 to 6.