Multimodal CAD model retrieval method from monocular images based on pre-trained large model

Abstract

The present invention discloses a monocular image multimodal CAD model retrieval method based on a pre-trained large model. The method combines image segmentation, depth estimation, semantic generation and multimodal feature extraction technology, and utilizes an image segmentation model, a depth estimation model and a text generation model to fully extract image, depth and text information. Through a multimodal feature fusion module, weighted fusion of features is achieved to generate a final multimodal query feature vector. Thereafter, multimodal contrast learning is used to improve feature alignment capabilities to ensure that the multimodal query feature vector matches the CAD model features in a unified feature space. To improve retrieval efficiency, the present invention further proposes a triple index quantized graph search method, which combines local sensitive hashing, hierarchical navigable small-world graphs and product quantization technology to achieve efficient model retrieval in a large-scale CAD model database, showing significant advantages in multimodal information processing and large-scale CAD data retrieval.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal information processing, and in particular to a monocular image multimodal CAD model retrieval method based on a pre-trained large model. Background Art

[0002] With the widespread application of computer-aided design (CAD) technology, a large number of complex 3D CAD models are used in manufacturing, architecture, product design and other fields. However, how to efficiently and accurately retrieve CAD models that match a certain scene or object in the huge CAD model library has become a technical problem that needs to be solved urgently.

[0003] Traditional CAD model retrieval methods mostly rely on retrieval algorithms based on shape, geometric features or text descriptions. These methods often require users to input precise geometric features or complete object descriptions for effective retrieval. However, matching based solely on shape or text features is susceptible to ambiguity and incomplete information, resulting in unsatisfactory retrieval results. In addition, many existing methods fail to fully combine multimodal information such as images, text and depth, thus limiting their ability to perform accurate matching in complex scenes.

[0004] In recent years, deep learning and large model technology have made significant breakthroughs in the field of computer vision. Pre-trained large models have performed well in tasks such as image segmentation, semantic understanding, and depth estimation, especially in complex scenes, and can accurately extract multimodal information. This provides new possibilities for CAD model retrieval based on multimodal features.

[0005] However, existing multimodal CAD model retrieval methods still face the following challenges:

[0006] 1. Limitations of single-modal processing: Traditional methods cannot effectively combine multimodal information such as images, text, and depth, making it difficult to fully understand the features of objects in complex scenes.

[0007] 2. Bottleneck of efficient retrieval: Faced with massive CAD models, existing retrieval technologies are often inefficient and difficult to achieve fast retrieval while ensuring accuracy.

[0008] 3. Difficulty in feature alignment and fusion: How to align information from different modalities such as images, texts, and depth in a unified feature space is still a difficulty in multimodal retrieval.

[0009] Therefore, there is an urgent need for a new method that can fully utilize the multimodal feature fusion capabilities of large models to improve the retrieval efficiency and accuracy of CAD models under monocular images. Summary of the invention

[0010] In view of the shortcomings of the prior art, the present invention provides a monocular image multimodal CAD model retrieval method based on a pre-trained large model, which solves the problems of incomplete multimodal information processing, low CAD model retrieval efficiency, and difficulty in feature alignment in the prior art. The method combines image segmentation, depth estimation, semantic generation, feature extraction, feature fusion and efficient retrieval technology to accurately retrieve the matching CAD model from a single image.

[0011] The main purpose of the present invention is to fully extract image, depth and text information by using the pre-trained Segment Anything Model (SAM) image segmentation model, the Dense Prediction Transformer (DPT) depth estimation model and the GPT-4V text generation model, and improve the accuracy and efficiency of retrieval through a multimodal feature fusion module; and then realize efficient retrieval in a large-scale CAD model database through a triple index quantized graph search (TQGS) method.

[0012] To solve the above technical problems, the present invention provides the following technical solution: a monocular image multimodal CAD model retrieval method based on a pre-trained large model, comprising the following steps:

[0013] S1. Segment a single original scene image Image based on the image segmentation model to obtain segmentation sub-images s of multiple target objects i And each segmentation subgraph s i Bounding box information of

[0014] S2, generate the depth map I of the original scene image based on the depth estimation model depth , and extract the corresponding depth information d from the depth map according to the position information of the segmented sub-image i ;

[0015] S3, the original scene image Image, segmentation sub-image s i and its depth information d i In the input text generation model GPT-4V, GPT-4V generates each sub-graph s based on the segmentation results i Text description of i ;

[0016] S4, segment sub-graph s through image encoder, text encoder and depth encoder i , text description and depth information d i Perform feature extraction and generate corresponding image features g I , text features g T and the deep feature g D , and map it to a unified feature space for alignment;

[0017] S5, image feature g is fused through multimodal feature fusion module I , text features g T and the deep feature g D Perform weighted fusion to generate the multimodal query feature vector f of the final scene Q ;

[0018] S6, feature alignment method based on multimodal contrastive learning, by multimodal query feature vector f Q and the CAD model feature vector g P Comparative learning can achieve alignment of feature space and improve retrieval accuracy;

[0019] S7, a CAD model retrieval acceleration method based on triple index quantized graph search (TQGS) method, used to achieve fast and accurate model retrieval in large-scale CAD model databases;

[0020] S8. A training dataset construction method based on multimodal contrastive learning is used to enhance the model feature alignment capability in the CAD model retrieval system.

[0021] Furthermore, in step S1, the specific process includes the following steps:

[0022] S101, the original scene image Image is sent to the image segmentation model SAM. Through the segmentation processing of the model, n segmentation sub-graphs can be obtained, denoted as s 1 ,s 2 ,s 3 ,...,s n , where each segmentation subgraph s i Corresponding to a target object;

[0023] S102: In the process of generating the segmented sub-image, the SAM image segmentation model outputs the bounding box information of each segmented sub-image, denoted as l i ={x i1 ,y i1 ,x i2 ,y i2}, where (x i1 ,y i1 ) represents the coordinate of the upper left corner of the segmentation sub-image, (x i2 ,y i2 ) represents the coordinates of the lower right corner of the segmented sub-image.

[0024] Furthermore, in step S2, the specific process includes the following steps:

[0025] S201: The input original scene image Image is first processed by the pre-trained depth estimation model DPT to generate a depth map I of the image. depth ;

[0026] S202, for each segmentation sub-graph s i , whose bounding box coordinates are l i ={x i1 ,y i1 ,x i2 ,y i2}, use these bounding box coordinates from the depth map I depth Cut out the corresponding depth area d i , that is, d i =I depth [x i1 :x i2 ,y i1 :y i2 ].

[0027] Furthermore, in step S3, the specific process includes the following steps:

[0028] S301. Based on the input data, design a specific prompt word Prompt to query the text generation model GPT-4V;

[0029] S302, the text generation model GPT-4V determines whether each segmented sub-image contains an object that can be represented by a CAD model according to the prompt word Prompt. If the segmented sub-image contains an object associated with the CAD model, the model will output the corresponding object category; otherwise, "False" is returned to indicate an irrelevant object;

[0030] S303, when the text generation model GPT-4V recognizes that there is an object that can be represented by a CAD model in the segmented sub-image, it not only returns the category name of the object, but also generates a detailed appearance description, which includes the shape, material, and color appearance features of the object;

[0031] S304, after generating the object category and appearance description, filter out the segmented sub-images that are identified as being associated with the CAD model, and retain the object category, text description, and other information of the sub-image. i The filtered segmentation sub-image will be used as the input data for the subsequent multimodal feature alignment and retrieval steps.

[0032] Furthermore, in step S4, the specific process includes the following steps:

[0033] S401, using the pre-trained SLIP image encoder E I For each segmentation subgraph s i Processing, image encoder E IBased on the deep neural network structure, the visual features of the input image data are extracted by performing convolution operations, and the final image feature vector f is generated, which can represent the visual information such as shape, texture and color of the object. I (i) ;

[0034] S402: The text description of each segmented sub-graph generated by the text generation model GPT-4V is i Input to SLIP text encoder E T In the process, the semantic features in the text are extracted to generate the corresponding text feature vector containing the important semantic information of the object category and appearance description.

[0035] S403: The depth information d of each segmented sub-image is i Input to the visual Transformer ViT deep encoder E D Generate a high-dimensional deep feature vector that represents the geometric structure and three-dimensional spatial information of the object

[0036] S404, using feature mapping function F I ,F T ,F D Map image, text and depth feature vectors to a unified feature space respectively To achieve unified representation and alignment of multimodal features, the specific formula is as follows:

[0037]

[0038]

[0039]

[0040] Among them, W I ,W T ,W D are the weight matrices for mapping image, text, and depth features, respectively, and b I ,b T ,b D They are the corresponding bias items for mapping image, text and depth features respectively.

[0041] Furthermore, in step S5, the specific process includes the following steps:

[0042] S501, the image feature g I , text features g T and the deep feature g D The weighted combination is performed through weight parameters α, β, and γ to form a multimodal feature sequence F i :

[0043]

[0044] Among them, N=3 represents the number of modes, and D is the feature dimension;

[0045] S502: input multimodal feature sequence F i Compute queries Q, keys K, and values ​​V to capture dependencies between modalities;

[0046] S503, the interdependence between different modalities is calculated through the attention weight matrix A, and the features are weighted and summed to obtain the fused multimodal feature O that includes the interaction and dependency between the features of each modality:

[0047]

[0048] O = AV;

[0049] S504: Perform a residual connection on the fused multimodal feature O and the original input multimodal feature F, and perform a normalization process on the feature F through layer normalization to obtain a normalized feature F. (1) :

[0050] F (1) =LayerNorm(F+O);

[0051] S505, through a two-layer fully connected neural network, the standardized feature F (1) Perform nonlinear transformation and generate deeper feature representation through feedforward neural network. The formula is as follows:

[0052] FFN(F (1) )=ReLU(F (1) W 1 +b 1 )W 2 +b 2 ;

[0053] in, and is the weight matrix of the fully connected layer, mapping the input features to the hidden layer dimension D f And then mapped back to the original feature space dimension D, the bias term and They are used to adjust the output of each layer.

[0054] After the feedforward neural network, the residual connection and layer normalization operations are continued to add and normalize the output and input features of the feedforward network to generate a richer feature representation F (2) :

[0055] F(2) =LayerNorm(F (1) +FFN(F (1) ));

[0056] S506, use global average pooling to represent the richer features after processing F (2) Aggregate and finally generate the multimodal query feature vector f Q As a feature representation that integrates multimodal information, it is used for subsequent CAD model retrieval tasks:

[0057]

[0058] Furthermore, in step S6, the specific process includes the following steps:

[0059] S601, using the cosine similarity formula to calculate the multimodal query feature vector f Q and the CAD model feature vector g P The calculated similarity value reflects the degree of matching between the two. The higher the similarity, the stronger the similarity of the features in the space:

[0060]

[0061] S602, design a loss function based on contrastive learning, the core goal of which is to reduce the correctly matched multimodal query feature vector f Q and the CAD model feature vector g P The distance between them is increased, which increases the distance of incorrect matches and thus improves the model's ability to distinguish:

[0062]

[0063] Where τ is the temperature parameter, Z is the number of candidate CAD models;

[0064] S603, setting the temperature parameter to control the sensitivity of the loss function, and improving the model's ability to distinguish subtle differences by reducing or increasing τ;

[0065] S604, through continuous iterative training, the model continuously adjusts the multimodal query feature vector f using the optimized contrastive learning loss function Q , the multimodal query feature vector f Q and the CAD model feature vector g P Alignment, as the training process progresses, the model is gradually able to accurately identify and match the multimodal query feature vector f Q Most similar CAD model.

[0066] Furthermore, in step S7, the specific process includes the following steps:

[0067] S701, based on local sensitive hashing LSH, the CAD model feature vector g P Hash and map it to a specific bucket. By clustering similar feature vectors in the same bucket, the search scope is reduced and the retrieval efficiency is improved.

[0068] S702, after locating the relevant LSH bucket, use the hierarchical navigable small world graph HNSW to organize the CAD models in the bucket, and realize layer-by-layer navigation from global to local through the hierarchical graph structure to accelerate retrieval;

[0069] S703, perform neighbor search through multi-level HNSW graph structure, multi-modal query feature vector f Q Starting from the top layer of the image, it is passed down step by step until it reaches the bottom layer, thus achieving accurate CAD model matching;

[0070] S704, CAD model feature vector g P Apply the product quantity PQ to transform the CAD model feature vector g P Split into M sub-vectors: g P =[g P1 ,g P2 ,...,g PM ], each sub-vector is obtained by finding the codebook center C closest to it i To quantify: Reduce the CAD model feature vector g P dimensions to improve computational efficiency;

[0071] S705: Use the product quantization method to quantize the multimodal query feature vector f Q and the CAD model feature vector g P The similarity between them is calculated, and the multimodal query feature vector f is calculated through the feature vector quantized by PQ. Q and the CAD model feature vector g P Distance: Finally, the CAD model with the smallest distance is returned as the retrieval result.

[0072] Furthermore, in step S8, the specific process includes the following steps:

[0073] S801, based on the images and 3D scan data in the Scannet25K dataset, obtain the real semantic segmentation information of the indoor scene, and generate the segmentation sub-graph of each object in the scene and its corresponding real bounding box position;

[0074] S802, extracting real depth map information from the original 3D scan data of Scannet25K, and segmenting the depth map of the scene into corresponding depth sub-maps based on the bounding box positions of the segmentation sub-maps;

[0075] S803, pairing a preset real text description for each segmented sub-image, where the description includes the category and appearance details of the object, ensuring that it is consistent with the object in the segmented sub-image;

[0076] S804, constructing a multimodal training dataset including input images, segmented sub-images, sub-image bounding box positions, text descriptions, depth sub-images, and matching relationships between sub-images and CAD models, to support multimodal comparative learning of the model.

[0077] By means of the above technical solution, the present invention provides a monocular image multimodal CAD model retrieval method based on a pre-trained large model, which has at least the following beneficial effects:

[0078] The present invention realizes the organic fusion of image, text and depth information and the alignment of unified feature space by innovatively combining the multimodal information processing capabilities of large models, significantly improving the accuracy and efficiency of CAD model retrieval. Compared with the prior art, the traditional retrieval methods of single mode or simple modal combination are difficult to cope with accurate matching in complex scenes. This method, with the help of pre-trained large models such as Segment Anything Model (SAM) image segmentation model, Dense Prediction Transformer (DPT) depth estimation model and GPT-4V text generation model, can fully capture and understand the semantic and geometric information in the image. Combined with the efficient triple index quantized graph search (TQGS) mechanism, it not only improves the multimodal feature fusion in complex scenes, but also solves the problem of rapid retrieval of CAD models in large-scale databases, showing significant advantages in multimodal information processing and efficient retrieval of large-scale data. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0080] Figure 1 Schematic diagram of the network training process of the monocular image multimodal CAD model retrieval method based on the pre-trained large model in the present invention;

[0081] Figure 2 Schematic diagram of the network reasoning process of the monocular image multimodal CAD model retrieval method based on the pre-trained large model in the present invention. DETAILED DESCRIPTION

[0082] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods, so that the implementation process of how the present application uses technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly.

[0083] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, so the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0084] Please refer to Figure 1-Figure 2 , shows a specific implementation of this embodiment. This embodiment mainly includes segmenting the input image, depth estimation, multimodal feature extraction and fusion, and matching in a large-scale CAD model library through an efficient retrieval algorithm, and finally accurately retrieving the CAD model from a single scene image. This method has significant innovation and efficiency, and is particularly suitable for multimodal information processing in complex scenes.

[0085] This embodiment proposes a monocular image multimodal CAD model retrieval method based on a pre-trained large model, and the method includes the following steps:

[0086] S1. Segment a single original scene image Image based on the image segmentation model to obtain segmentation sub-images s of multiple target objects i And each segmentation subgraph s i Bounding box information of

[0087] As a preferred implementation of step S1, the specific process includes the following steps:

[0088] S101, the original scene image Image is sent to the image segmentation model SAM. The image segmentation model SAM learns the global and local features of the image through a deep neural network based on the self-attention mechanism, thereby performing fine-grained segmentation of the target object in the image. Through the segmentation processing of the model, n segmentation sub-graphs can be obtained, denoted as s 1 ,s 2 ,s 3 ,...,s n , where each segmentation subgraph s i Corresponding to a target object;

[0089] S102: In the process of generating the segmented sub-image, the SAM image segmentation model outputs the bounding box information of each segmented sub-image, denoted as l i ={x i1 ,y i1 ,x i2 ,y i2}, where (x i1 ,y i1 ) represents the coordinate of the upper left corner of the segmentation sub-image, (x i2 ,y i2 ) represents the coordinates of the lower right corner of the segmented sub-image.

[0090] In this embodiment, the input scene image is a three-channel RGB image, denoted as Where H and W represent the height and width of the image, respectively, and 3 represents the color channel. The image can contain multiple target objects, which are the objects to be obtained through segmentation technology later. In order to perform accurate object segmentation, the pre-trained large model Segment Anything Model (SAM) is used. SAM is an advanced image segmentation model that can perform high-precision segmentation of the input image and generate multiple sub-images containing different objects. Secondly, through this bounding box information, the position of the target object in the original scene image Image can be accurately defined. These segmentation sub-images and the corresponding position information will be used for multimodal feature extraction and alignment in subsequent steps. The segmentation sub-images generated by the model not only contain the pixel information of the target objects, but also retain their geometric positions to ensure that each target object can be accurately matched with the corresponding CAD model during the multimodal alignment process.

[0091] S2, generate the depth map I of the original scene image based on the depth estimation model depth , and extract the corresponding depth information d from the depth map according to the position information of the segmented sub-image i ;

[0092] As a preferred implementation of step S2, the specific process includes the following steps:

[0093] S201: The input original scene image Image is first processed by the pre-trained depth estimation model DPT to generate a depth map I of the image. depth ;

[0094] S202, for each segmentation sub-graph s i , whose bounding box coordinates are l i ={x i1 ,y i1 ,x i2 ,y i2}, use these bounding box coordinates from the depth map I depth Cut out the corresponding depth area di , that is, d i =I depth [x i1 :x i2 ,y i1 :y i2 ].

[0095] In this embodiment, DPT is a neural network model specifically used for depth estimation. It uses a self-attention mechanism to perform global contextual understanding of each pixel in the image and can generate depth values ​​corresponding to the input image pixels. Each pixel value represents the relative depth information of the pixel in three-dimensional space, and the generated depth map has the same spatial resolution as the input image. The extracted depth information d i With the segmentation subgraph s i Together they characterize the geometric shape and spatial information of the object in the image. i They are all depth representations of the corresponding target objects, which will be used for subsequent multimodal feature extraction and alignment to ensure that the image, text, and depth information can be integrated in a unified feature space.

[0096] S3, the original scene image Image, segmentation sub-image s i and its depth information d i In the input text generation model GPT-4V, GPT-4V is a language-vision model with multimodal understanding capabilities. It can not only generate text descriptions based on the input image, but also judge the relevance of the object to the CAD model by processing the object segmentation information in the image and combining the bounding box data. After these input data are processed by GPT-4V, they will provide a basis for subsequent semantic judgments, and GPT-4V generates each sub-image s based on the segmentation results. i Text description of i ;

[0097] As a preferred implementation of step S3, the specific process includes the following steps:

[0098] S301. Based on the input data, design a specific prompt word Prompt to query the text generation model GPT-4V;

[0099] S302, the text generation model GPT-4V generates a text based on the prompt word Prompt, for example: "The following image contains a segmented object located at coordinates [x i1 ,y i1 ,x i2 ,y i2].Pleasedetermine if this object corresponds to a CAD model,and if so,provide theobject's category.",The Chinese meaning of this prompt is: "The following figure contains an object at coordinates [x i1 ,y i1 ,x i2 ,y i2 ] object. Please determine whether this object corresponds to the CAD model. If so, please provide the category of the object. "Based on this prompt, GPT-4V determines whether each segmented sub-image contains an object that can be represented by a CAD model. If the segmented sub-image contains an object associated with the CAD model, the model will output the corresponding object category; otherwise, it returns "False" to indicate an irrelevant object;

[0100] S303. When the text generation model GPT-4V recognizes that there is an object that can be represented by a CAD model in the segmented sub-image, it not only returns the category name of the object (such as "chair" or "table"), but also generates a detailed appearance description, which includes the shape, material, and color appearance features of the object; for example: "This chair features a rectangular seat and slightly curved backrest, supported by slender metal legs, with a soft dark-gray fabric cover." The Chinese meaning of this description is: "This chair has a rectangular seat and a slightly curved backrest, supported by slender metal legs, and covered with soft dark-gray fabric." This generated detailed description not only helps to enhance semantic understanding, but also provides semantic information support for subsequent CAD model matching.

[0101] S304, after generating the object category and appearance description, filter out the segmented sub-images that are identified as being associated with the CAD model, and retain the object category, text description, and other information of the sub-image. i The selected segmentation sub-images will be used as input data for the subsequent multimodal feature alignment and retrieval steps to ensure that subsequent processing focuses only on valid data that matches the CAD model.

[0102] S4, segment sub-graph s through image encoder, text encoder and depth encoder i , text description and depth information d i Perform feature extraction and generate corresponding image features g I, text features g T and the deep feature g D , and map it to a unified feature space for alignment;

[0103] As a preferred implementation of step S4, the specific process includes the following steps:

[0104] S401, using the pre-trained SLIP image encoder E I For each segmentation subgraph s i Processing, image encoder E I Based on the deep neural network structure, the visual features of the input image data are extracted by performing convolution operations, and the final image feature vector f is generated, which can represent the visual information such as shape, texture and color of the object. I (i) :f I (i) =E I (s i );

[0105] S402: The text description of each segmented sub-graph generated by the text generation model GPT-4V is i Input to SLIP text encoder E T In the process, the semantic features in the text are extracted to generate the corresponding text feature vector containing the important semantic information of the object category and appearance description.

[0106] S403: The depth information d of each segmented sub-image is i Input to the visual Transformer ViT deep encoder E D Generate a high-dimensional deep feature vector that represents the geometric structure and three-dimensional spatial information of the object

[0107] S404, using feature mapping function F I ,F T ,F D Map image, text and depth feature vectors to a unified feature space respectively To achieve unified representation and alignment of multimodal features, the specific formula is as follows:

[0108]

[0109]

[0110]

[0111] Among them, W I ,W T ,WD They are the weight matrices for mapping image, text, and depth features, respectively. Their function is to transform feature vectors of different modalities into the same feature space; and b I ,b T ,b D is the corresponding bias term, which is used to adjust the converted features to make them more suitable for subsequent feature fusion and alignment. Through these mapping functions, image, text and depth features are converted into the same feature space, thereby achieving unified representation and alignment of multimodal features.

[0112] S5, image feature g is fused through multimodal feature fusion module I , text features g T and the deep feature g D Perform weighted fusion to generate the multimodal query feature vector f of the final scene Q ;

[0113] As a preferred implementation of step S5, the specific process includes the following steps:

[0114] S501, the image feature g I , text features g T and the deep feature g D The weighted combination is performed through weight parameters α, β, and γ to form a multimodal feature sequence F i :

[0115]

[0116] Among them, N = 3 represents the number of modes, and D is the feature dimension; different modal features are weighted according to the preset weight parameters α, β, and γ to ensure that each modal information is properly represented and combined during the multimodal information fusion process.

[0117] S502: input multimodal feature sequence F i Calculate the query Q, key K and value V to capture the dependencies between modalities; through the self-attention mechanism, the model can identify the interactions and importance between different modal features; specifically, for the input multimodal feature sequence F i Calculate the query Q, key K and value V. The specific formula is as follows:

[0118] Q=FW Q ,K=FW K ,V=FW V ;

[0119] in, W represents the input multimodal feature sequence, N is the number of features in the sequence, and D is the dimension of each feature. F contains multimodal information obtained by fusion of image, text and deep features. By linearly transforming F, the model can generate new queries, keys and values, which are used to capture the interdependencies between modal features in the self-attention mechanism. Q , W K and W V is the linear transformation matrix with dimension Used to map input features to a new feature space;

[0120] S503, the interdependence between different modalities is calculated through the attention weight matrix A, and the features are weighted and summed. This step can integrate the feature information of different modalities so that the feature vector finally generated can fully represent the features of each modality, and obtain the fused multimodal feature O that includes the interaction and dependency between the features of each modality:

[0121]

[0122] O = AV;

[0123] S504: In order to enhance the stability and expression ability of the model, residual connection and layer normalization operations are applied to standardize the fused features. Residual connection helps to retain the original information of the input features, while layer normalization ensures that the fused features have better stability; the specific implementation is: residual connection is performed on the fused multimodal feature O and the original input multimodal feature F, and normalization is performed through layer normalization to obtain the standardized feature F. (1) :

[0124] F (1) =LayerNorm(F+O);

[0125] The residual connection operation retains the effective information of the original features, while layer normalization helps prevent the gradient vanishing problem and improves the training stability of the model.

[0126] S505, through a two-layer fully connected neural network, the standardized feature F (1) Perform nonlinear transformation and generate deeper feature representation through feedforward neural network. The formula is as follows:

[0127] FFN(F (1) )=ReLU(F (1) W 1 +b 1 )W 2 +b 2 ;

[0128] in, is the weight matrix of the fully connected layer, mapping the input features to the hidden layer dimension D f And then mapped back to the original feature space dimension D. Bias and They are used to adjust the output of each layer, making the nonlinear transformation of the neural network more flexible and avoiding feature mapping only through linear combination. Through these weight matrices and bias items, the feedforward neural network can perform a deeper feature representation of the input features and enhance the model's ability to express complex features.

[0129] After the feedforward neural network, the residual connection and layer normalization operations are continued to add and normalize the output and input features of the feedforward network to generate a richer feature representation F (2) :

[0130] F (2) =LayerNorm(F (1) +FFN(F (1) ));

[0131] S506, use global average pooling to represent the richer features after processing F (2) Aggregate and finally generate the multimodal query feature vector f Q As a feature representation that integrates multimodal information, the multimodal query feature vector f Q It contains a comprehensive representation of multimodal information such as images, text, and depth, which is used for subsequent CAD model retrieval tasks:

[0132] S6, feature alignment method based on multimodal contrastive learning, by multimodal query feature vector f Q and the CAD model feature vector g P Comparative learning can achieve alignment of feature space and improve retrieval accuracy;

[0133] As a preferred implementation of step S6, the specific process includes the following steps:

[0134] S601, using the cosine similarity formula to calculate the multimodal query feature vector f Q and the CAD model feature vector g P The calculated similarity value reflects the degree of matching between the two. The higher the similarity, the stronger the similarity of the features in the space:

[0135]

[0136] S602, design a loss function based on contrastive learning, the core goal of which is to reduce the correctly matched multimodal query feature vector f Qand the CAD model feature vector g P The distance between the two matches is increased, thereby increasing the distance of the incorrect matches, thereby improving the model's ability to distinguish. Through this process, the model can learn to distinguish the features of correct and incorrect matches:

[0137]

[0138] Where τ is the temperature parameter, Z is the number of candidate CAD models;

[0139] S603, set the temperature parameter to control the sensitivity of the loss function and balance the impact of feature similarity on model optimization. The temperature parameter adjusts the smoothness of the loss function to ensure that the model can stably learn the similarity between multimodal features. Specifically, by lowering or increasing τ, the model can pay more attention to the subtle differences between similar features or respond quickly to larger differences at different training stages. A lower temperature parameter can amplify the difference between similarities and improve the model's ability to distinguish subtle differences, while a higher temperature parameter makes the model more stable in terms of the smoothness of feature distribution.

[0140] S604, through continuous iterative training, the model gradually aligns the scene features from the image, text and depth information with the CAD model features in a unified feature space, thereby improving the retrieval accuracy of the CAD model; specifically, the model continuously adjusts the multimodal query feature vector f using an optimized contrastive learning loss function Q , the multimodal query feature vector f Q and the CAD model feature vector g P Alignment, as the training process progresses, the model is gradually able to accurately identify and match the multimodal query feature vector f Q The most similar CAD model can be found to improve the accuracy and efficiency of retrieval. The specific network learning process is as follows: Figure 1 As shown, the reasoning process is as follows Figure 2 shown.

[0141] S7, a CAD model retrieval acceleration method based on triple index quantized graph search (TQGS) method, used to achieve fast and accurate model retrieval in large-scale CAD model databases;

[0142] As a preferred implementation of step S7, the specific process includes the following steps:

[0143] S701, based on local sensitive hashing LSH, the CAD model feature vector g P Hash it and map it to a specific bucket. By clustering similar feature vectors in the same bucket, the search range is reduced and the retrieval efficiency is improved. The specific implementation is to perform a hash operation on the CAD model feature vector g PApply the LSH hash function h i :

[0144]

[0145] Among them, a i is a randomly generated vector, b i is the offset, and w is the width of the hash bucket. Through LSH, feature vectors are hashed into different buckets, so that similar feature vectors are divided into the same bucket, thereby reducing the amount of calculation for subsequent retrieval;

[0146] S702. After locating the relevant LSH bucket, use the hierarchical navigable small-world graph HNSW to organize the CAD models in the bucket, and realize layer-by-layer navigation from global to local through the hierarchical graph structure to accelerate retrieval; specifically, in each LSH bucket, build an HNSW graph structure. HNSW is a multi-level nearest neighbor search algorithm. The nodes at the top level are sparse, which is convenient for fast global navigation, and the nodes at the bottom level are dense, which is used for fine local search. In the HNSW graph structure, nodes represent CAD models, and edges represent the nearest neighbor relationship between models;

[0147] S703, perform neighbor search through multi-level HNSW graph structure, multi-modal query feature vector f Q Starting from the top layer of the graph, the feature vector f is gradually transferred downward until it reaches the bottom layer, thereby achieving accurate CAD model matching; the specific implementation is to start from the top layer of the HNSW graph and query the feature vector f Q Select the closest neighbor node at each layer and navigate down to the bottom layer layer by layer. At this point, the query feature will perform a precise search among the closest neighbor nodes to find the best matching CAD model;

[0148] S704, CAD model feature vector g P Apply the product quantity PQ to transform the CAD model feature vector g P Split into M sub-vectors: g P =[g P1 ,g P2 ,...,g PM ], each sub-vector is obtained by finding the codebook center C closest to it i To quantify: Reduce the CAD model feature vector g P The PQ method can compress high-dimensional feature vectors into low-dimensional representations, significantly improving the efficiency of subsequent similarity calculations.

[0149] S705: Use the product quantization method to quantize the multimodal query feature vector f Q and the CAD model feature vector gP The similarity between them is calculated, and the multimodal query feature vector f is calculated through the feature vector quantized by PQ. Q and the CAD model feature vector g P Distance: Finally, the CAD model with the smallest distance is returned as the retrieval result.

[0150] In this embodiment, a retrieval acceleration method that combines three technologies, namely locality sensitive hashing (LSH), hierarchical navigable small-world graph (HNSW), and product quantization (PQ), effectively reduces the computational complexity during retrieval and significantly improves the speed while maintaining retrieval accuracy.

[0151] S8. A training dataset construction method based on multimodal contrastive learning is used to enhance the model feature alignment capability in the CAD model retrieval system.

[0152] As a preferred implementation of step S8, the specific process includes the following steps:

[0153] S801, based on the images and 3D scan data in the Scannet25K dataset, obtain the real semantic segmentation information of the indoor scene, and generate the segmentation sub-graph of each object in the scene and its corresponding real bounding box position; the specific implementation is as follows: first, extract the image and 3D scan data of each indoor scene from the Scannet25K dataset, and use the provided real semantic segmentation information to annotate and segment the objects in the scene. Each segmentation sub-graph s i corresponds to an object and contains its exact bounding box position l i ={x i1 ,y i1 ,x i2 ,y i2}, the bounding box information is used for subsequent depth map segmentation and feature alignment.

[0154] S802, extracting the real depth map information from the original 3D scan data of Scannet25K, and segmenting the depth map of the scene into corresponding depth sub-maps based on the bounding box position of the segmentation sub-map; specifically implemented as follows: using the original 3D scan data to generate a depth map of each scene Each pixel value represents the depth information of the location. According to the bounding box l of each segmentation sub-image i , from the depth map I depth Extract the depth region d corresponding to the sub-image i , forming the segmented depth sub-image d i =I depth [x i1 :x i2 ,y i1 :y i2], ensuring the correspondence with the three-dimensional geometric information of the object;

[0155] S803, pairing a preset real text description for each segmented sub-image, the description content includes the category and appearance details of the object, ensuring that it is consistent with the object in the segmented sub-image; specifically implemented as follows: for each sub-image s i Generate the corresponding real text description t i , the description includes the category and appearance details of the object, for example, "This is a gray sofa with metal legs and a rectangular shape." The Chinese meaning of this description is: "This is a gray sofa with metal legs and a rectangular shape." These text descriptions ensure that the category and appearance characteristics of the objects in the sub-image are accurately reflected, which facilitates the subsequent multimodal feature alignment;

[0156] S804, construct a multimodal training data set including input images, segmented sub-images, sub-image bounding box positions, text descriptions, depth sub-images, and the matching relationship between sub-images and CAD models to support the multimodal comparative learning of the model. The specific implementation is: summarize the above processed information to construct a multimodal training data set, which includes the input scene image, segmented sub-image s, the bounding box l of each sub-image, the text description t, the corresponding depth sub-image d, and the CAD model c corresponding to each sub-image. Through this training data set, the model can effectively learn the alignment relationship between image, text, and depth features in multimodal comparative learning, thereby improving the retrieval ability of the model.

[0157] In this embodiment, a multimodal training dataset is constructed by combining images, 3D scanning data, and semantic information in the Scannet25K dataset to support the model in multimodal comparative learning, thereby improving the feature alignment capability in the CAD model retrieval system.

[0158] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0159] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0160] The above implementation methods have been described in detail. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A monocular image multimodal CAD model retrieval method based on a pre-trained large model, characterized in that: The following steps are involved: S1. Segment a single original scene image Image based on the image segmentation model to obtain segmentation sub-images s of multiple target objects i And each segmentation subgraph s i Bounding box information of S2, generate the depth map I of the original scene image based on the depth estimation model depth , and extract the corresponding depth information d from the depth map according to the position information of the segmented sub-image i ; S3, the original scene image Image, segmentation sub-image s i and its depth information d i In the input text generation model GPT-4V, GPT-4V generates each sub-graph s based on the segmentation results i Text description of i ; S4, segment sub-graph s through image encoder, text encoder and depth encoder i , text description and depth information d i Perform feature extraction and generate corresponding image features g I , text features g T and the deep feature g D , and map it to a unified feature space for alignment; S5, image feature g is fused through multimodal feature fusion module I , text features g T and the deep feature g D Perform weighted fusion to generate the multimodal query feature vector f of the final scene Q ; S6, feature alignment method based on multimodal contrastive learning, by multimodal query feature vector f Q and the CAD model feature vector g P Comparative learning can achieve alignment of feature space and improve retrieval accuracy; S7. A CAD model retrieval acceleration method based on a triple index quantized graph search (TQGS) method is used to achieve fast and accurate model retrieval in a large-scale CAD model database; the specific process includes the following steps: S701, based on local sensitive hashing LSH, the CAD model feature vector g P Hash and map it to a specific bucket. By clustering similar feature vectors in the same bucket, the search scope is reduced and the retrieval efficiency is improved. S702, after locating the relevant LSH bucket, use the hierarchical navigable small world graph HNSW to organize the CAD models in the bucket, and realize layer-by-layer navigation from global to local through the hierarchical graph structure to accelerate retrieval; S703, perform neighbor search through multi-level HNSW graph structure, multi-modal query feature vector f Q Starting from the top layer of the image, it is passed down step by step until it reaches the bottom layer, thus achieving accurate CAD model matching; S704, CAD model feature vector g P Apply the product quantity PQ to transform the CAD model feature vector g P Split into M sub-vectors: g P =[g P1 ,g P2 ,…,g PM ], each sub-vector is obtained by finding the codebook center C closest to it i To quantify: Reduce the CAD model feature vector g P dimensions to improve computational efficiency; S705: Use the product quantization method to quantize the multimodal query feature vector f Q and the CAD model feature vector g P The similarity between them is calculated, and the multimodal query feature vector f is calculated through the feature vector quantized by PQ. Q and the CAD model feature vector g P Distance: Finally, the CAD model with the smallest distance is returned as the retrieval result; S8. A training dataset construction method based on multimodal contrastive learning is used to enhance the model feature alignment capability in the CAD model retrieval system.

2. The method for retrieval of multimodal CAD models from monocular images based on pre-trained large models according to claim 1, characterized in that: In step S1, the specific process includes the following steps: S101, send the original scene image Image to the image segmentation model SAM, and obtain n segmentation sub-graphs through the segmentation processing of the model, denoted as s1, s2, s3, ..., s n , where each segmentation subgraph s i Corresponding to a target object; S102: In the process of generating the segmented sub-image, the SAM image segmentation model outputs the bounding box information of each segmented sub-image, denoted as l i ={x i1 ,y i1 ,x i2 ,y i2 }, where (x i1 ,y i1 ) represents the coordinate of the upper left corner of the segmentation sub-image, (x i2 ,y i2 ) represents the coordinates of the lower right corner of the segmented sub-image.

3. The method for retrieval of multimodal CAD models from monocular images based on pre-trained large models according to claim 1, characterized in that: In step S2, the specific process includes the following steps: S201: The input original scene image Image is first processed by the pre-trained depth estimation model DPT to generate a depth map I of the image. depth ; S202, for each segmentation sub-graph s i , whose bounding box coordinates are l i ={x i1 ,y i1 ,x i2 ,y i2 }, use these bounding box coordinates from the depth map I depth Cut out the corresponding depth area d i , that is, d i =I depth [x i1 :x i2 ,y i1 :y i2 ].

4. The method for retrieval of multimodal CAD models from monocular images based on pre-trained large models according to claim 1, characterized in that: In step S3, the specific process includes the following steps: S301. Based on the input data, design a specific prompt word Prompt to query the text generation model GPT-4V; S302, the text generation model GPT-4V determines whether each segmented sub-image contains an object represented by the CAD model according to the prompt word Prompt. If the segmented sub-image contains an object associated with the CAD model, the model will output the corresponding object category; otherwise, "False" is returned to indicate an irrelevant object; S303, when the text generation model GPT-4V recognizes that there is an object that can be represented by a CAD model in the segmented sub-image, it not only returns the category name of the object, but also generates a detailed appearance description, which includes the shape, material, and color appearance features of the object; S304, after generating the object category and appearance description, filter out the segmented sub-images that are identified as being associated with the CAD model, and retain the object category, text description, and other information of the sub-image. i The filtered segmentation sub-image will be used as the input data for the subsequent multimodal feature alignment and retrieval steps.

5. The method for retrieval of multimodal CAD models from monocular images based on pre-trained large models according to claim 1, characterized in that: In step S4, the specific process includes the following steps: S401, using the pre-trained SLIP image encoder E I For each segmentation subgraph s i Processing, image encoder E I Based on the deep neural network structure, the visual features of the input image data are extracted by performing convolution operations, and the final image feature vector that can represent the shape, texture and color visual information of the object is generated. S402: The text description of each segmented sub-graph generated by the text generation model GPT-4V is i Input to SLIP text encoder E T In the process, the semantic features in the text are extracted to generate the corresponding text feature vector containing the important semantic information of the object category and appearance description. S403: The depth information d of each segmented sub-image is i Input to the visual Transformer ViT deep encoder E D Generate a high-dimensional deep feature vector that represents the geometric structure and three-dimensional spatial information of the object S404, using feature mapping function F I ,F T ,F D Map image, text and depth feature vectors to a unified feature space respectively To achieve unified representation and alignment of multimodal features, the specific formula is as follows: Among them, W I ,W T ,W D are the weight matrices for mapping image, text, and depth features, respectively, and b I ,b T ,b D They are the corresponding bias items for mapping image, text and depth features respectively.

6. The method for retrieval of multimodal CAD models from monocular images based on a pre-trained large model according to claim 5, characterized in that: In step S5, the specific process includes the following steps: S501, the image feature g I , text features g T and the deep feature g D The weighted combination is performed through weight parameters α, β, and γ to form a multimodal feature sequence F i : Among them, N=3 represents the number of modes, and D is the feature dimension; S502: input multimodal feature sequence F i Compute queries Q, keys K, and values ​​V to capture dependencies between modalities; S503, the interdependence between different modalities is calculated through the attention weight matrix A, and the features are weighted and summed to obtain the fused multimodal feature O that includes the interaction and dependency between the features of each modality: O = AV; S504: Perform a residual connection on the fused multimodal feature O and the original input multimodal feature F, and perform a normalization process on the feature F through layer normalization to obtain a normalized feature F. (1) : F (1) =LayerNorm(F+O); S505, through a two-layer fully connected neural network, the standardized feature F (1) Perform nonlinear transformation and generate deeper feature representation through feedforward neural network. The formula is as follows: <h2 style=";text-align:left;direction:ltr">FFN(F<h2 style=";text-align:left;direction:ltr"> (1) <h2 style=";text-align:left;direction:ltr"> ReLU(F)<h2 style=";text-align:left;direction:ltr"> (1) <h2 style=";text-align:left;direction:ltr"> W1+b1)W2+b2; in, and is the weight matrix of the fully connected layer, mapping the input features to the hidden layer dimension D f And then mapped back to the original feature space dimension D, the bias term and They are used to adjust the output of each layer. After the feedforward neural network, the residual connection and layer normalization operations are continued to add and normalize the output and input features of the feedforward network to generate a richer feature representation F (2) : F (2) =LayerNorm(F (1) +FFN(F (1) )); S506, use global average pooling to represent the richer features after processing F (2) Aggregate and finally generate the multimodal query feature vector f Q As a feature representation that integrates multimodal information, it is used for subsequent CAD model retrieval tasks:

7. The method for retrieval of multimodal CAD models from monocular images based on pre-trained large models according to claim 1, characterized in that: In step S6, the specific process includes the following steps: S601, using the cosine similarity formula to calculate the multimodal query feature vector f Q and the CAD model feature vector g P The calculated similarity value reflects the degree of matching between the two. The higher the similarity, the stronger the similarity of the features in the space: S602, design a loss function based on contrastive learning, the core goal of which is to reduce the correctly matched multimodal query feature vector f Q and the CAD model feature vector g P The distance between them is increased, which increases the distance of incorrect matches and thus improves the model's ability to distinguish: Where τ is the temperature parameter, Z is the number of candidate CAD models; S603, setting the temperature parameter to control the sensitivity of the loss function, and improving the model's ability to distinguish subtle differences by reducing or increasing τ; S604, through continuous iterative training, the model continuously adjusts the multimodal query feature vector f using the optimized contrastive learning loss function Q , the multimodal query feature vector f Q and the CAD model feature vector g P Alignment, as the training process progresses, the model is gradually able to accurately identify and match the multimodal query feature vector f Q Most similar CAD model.

8. The method for retrieval of multimodal CAD models from monocular images based on pre-trained large models according to claim 1, characterized in that: In step S8, the specific process includes the following steps: S801, based on the images and 3D scan data in the Scannet25K dataset, obtain the real semantic segmentation information of the indoor scene, and generate the segmentation sub-graph of each object in the scene and its corresponding real bounding box position; S802, extracting real depth map information from the original 3D scan data of Scannet25K, and segmenting the depth map of the scene into corresponding depth sub-maps based on the bounding box positions of the segmentation sub-maps; S803, pairing each segmented sub-image with a preset real text description, where the description includes the category and appearance details of the object, ensuring that it is consistent with the object in the segmented sub-image; S804, constructing a multimodal training dataset including input images, segmented sub-images, sub-image bounding box positions, text descriptions, depth sub-images, and matching relationships between sub-images and CAD models, to support multimodal comparative learning of the model.

Images