Three-dimensional shape generation method fusing image retrieval and sketch regulation

Abstract

The invention discloses a three-dimensional shape generation method fusing image retrieval and sketch regulation, which comprises the following steps of: firstly, mapping an input sketch to a shared intermediate feature space by constructing a sketch-image mapping network, and retrieving a plurality of images which are closest to sketch semantics from an image library based on cosine similarity; secondly, a pre-trained CLIP visual encoder is adopted to carry out feature extraction on the retrieved image, fusion is carried out through a maximum pooling mode, and final image features are obtained; introducing a shape adapter with a cross attention mechanism into a de-noising network of a diffusion generation model, and fusing the geometric features of the input sketch into the de-noising network to obtain a three-dimensional latent vector consistent with the input sketch in structure and semantics; and finally, a decoder of the three-dimensional vector quantization variational auto-encoder is reduced into a three-dimensional symbol distance function for representation. According to the method, the quality of the three-dimensional shape generated by sketch driving in the aspects of structural consistency and local detail reduction is improved, and the method has high generalization ability and practical application value.

Description

Technical Field

[0001] This invention relates to the technical fields of computer-aided design and computer graphics, and in particular to a method for generating three-dimensional shapes that integrates image retrieval and sketch control, applicable to application scenarios such as industrial design, intelligent manufacturing, and virtual reality that require the generation of three-dimensional models from sketches. Background Technology

[0002] In fields such as industrial design, intelligent manufacturing, and virtual reality, three-dimensional shapes are widely used as an important form of intuitive expression. While traditional computer-aided design (CAD) technology possesses powerful modeling capabilities, its high requirements for user expertise and operational skills limit its universality. Sketches, due to their concise and intuitive expression, have become a commonly used input method in the early stages of design. However, existing sketch-based 3D reconstruction methods have several shortcomings: firstly, sketches are inherently sparse, containing limited semantic information; their simple lines and sparse information result in insufficient geometric information expression; secondly, existing generation methods often lack modeling of the overall structural relationship between the sketch and the 3D shape, easily leading to inconsistencies between the generated 3D shape and the input sketch in terms of overall structure, resulting in problems such as proportional imbalance or unreasonable shape. Summary of the Invention

[0003] The purpose of this invention is to provide a three-dimensional shape generation method that integrates image retrieval and sketch control, so as to alleviate the problems of sparse sketch expression, insufficient semantic information and weak geometric structure control, thereby improving the structural accuracy and detail restoration ability of the generated three-dimensional shapes.

[0004] To achieve the above objectives, the technical solution provided by this invention is: a three-dimensional shape generation method integrating image retrieval and sketch control, comprising the following steps:

[0005] S1: Train the sketch-image mapping network, build an image library, input a sketch, and extract features from the sketch and all images in the image library to obtain intermediate features of the sketch and intermediate features of all images;

[0006] S2: Calculate the cosine similarity between the sketch middle features and all image middle features, and retrieve the top k image middle features with the highest cosine similarity to the sketch middle features;

[0007] S3: Match the intermediate features of the first k images to the image library to obtain the corresponding k images;

[0008] S4: Feed the corresponding k images into the pre-trained CLIP visual encoder to extract features, and perform max pooling fusion on the dimensions of the extracted features to form the final image features;

[0009] S5: The final image features are used as the main condition input to the denoising network of the downstream diffusion generation model. A shape adapter is constructed, and a ResNet encoder is used to extract the geometric features of the input sketch. The geometric features of the input sketch are then incorporated into the denoising network through a cross-attention mechanism.

[0010] S6: After the denoising network completes the denoising, the three-dimensional latent vector in the latent space corresponding to the denoising network is obtained. The decoder of the three-dimensional vector quantization variational autoencoder is used to decode the three-dimensional latent vector and restore it to the three-dimensional symbolic distance function representation, thereby completing the modeling task from sketch to three-dimensional shape.

[0011] Furthermore, the sketch-image mapping network is used to map sketches to all images in the image library as intermediate features with a uniform form to meet the retrieval requirements of cosine similarity. The sketch-image mapping network consists of a pre-trained CLIP visual encoder and a trainable feature mapper. During training, the pre-trained CLIP visual encoder extracts features from the sketches and corresponding images in the training data to obtain sketch features and corresponding image features in the training data. Then, the sketch features in the training data are further adjusted by embedding them through the feature mapper to make them closer to the corresponding image features and farther away from other image features in the training data.

[0012] During training, triplet training data is constructed, including anchor sketches s, positive sample images p, and negative sample images n, and triplet loss L is used. triplet Supervised training is performed to optimize the parameters of the feature mapper, with triplet loss L. triplet The definition is as follows:

[0013] L triplet =max(0,d(f) s ,f p )-d(f s ,f n )+m)

[0014] In the formula, f s f p f n d(·,·) represents the intermediate features of the anchor sketch, the intermediate features of the positive sample image, and the intermediate features of the negative sample image, respectively; d(·,·) is the cosine similarity between features; m is the minimum boundary of the distance difference between positive and negative samples, used to improve the feature discrimination ability.

[0015] Furthermore, the cosine similarity between the sketch's intermediate features and the intermediate features of all images in the image database is calculated, and the images are sorted according to their cosine similarity. The k most similar image intermediate features are then selected, as shown in the following formula:

[0016]

[0017] In the formula, f represents the intermediate feature of the sketch; f i This represents the intermediate features of an image in the image database; CosSim(·,·) represents the cosine similarity between the two images.

[0018] Furthermore, features are re-extracted from the corresponding k images using a pre-trained CLIP visual encoder, resulting in k image features. To preserve representative semantic details, including edge contours, and to compress redundant information, a max-pooling strategy is employed for fusion, defined as follows:

[0019] F fused [j] = max(F1[j],F2[j],…,F k [j]), j∈(1,N)

[0020] In the formula, F k F represents the features of the k-th image; fused N represents the final image features after fusion, where N is the dimension of the features.

[0021] Furthermore, the shape adapter encodes the geometric structure of the sketch using a structured neural network ResNet to obtain the geometric features of the input sketch, and integrates the obtained geometric features of the input sketch into the intermediate network layer of the denoising network through a cross-attention mechanism to achieve control guidance;

[0022] In this context, the geometric features of the obtained input sketch are used as the query matrix Q, the intermediate network layer features of the denoising network are used as the key matrix K and the value matrix V, and the cross-attention mechanism is specifically represented as follows:

[0023]

[0024] In the formula, d K represents the scaling factor; T represents the transpose; the cross-attention mechanism enables fine-grained interaction between the geometric features of the input sketch and the intermediate network layer features of the denoising network, automatically aligning the corresponding structural positions.

[0025] Furthermore, the generation of the three-dimensional shape relies on the latent space constructed by the three-dimensional vector quantization variational autoencoder. In this latent space, the three-dimensional shape X, represented by the three-dimensional signed distance function, is encoded by the encoder of the three-dimensional vector quantization variational autoencoder. Compressed encoding is performed to obtain the three-dimensional latent vector z. t ,Right now:

[0026]

[0027] A conditional diffusion model denoising network is constructed in this latent space. The final image features and the geometric features of the input sketch are used as conditional inputs to guide the gradual denoising process of the initial noise in the denoising network, generating a 3D latent vector consistent with the sketch structure and semantics. Finally, the generated 3D latent vector z′0 is vector quantized and input into the decoder D of the 3D vector quantization variational autoencoder. τ Decode the data to restore it to the three-dimensional symbolic distance function representation X′, i.e.:

[0028] X′=D τ (VQ(z′0))

[0029] The three-dimensional symbolic distance function represents the explicit three-dimensional geometric mesh as the final output after X′ is processed.

[0030] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0031] 1. This invention enhances sketch features by constructing a sketch-image mapping network and introducing an image retrieval mechanism. Compared to methods that directly extract features from sketches, this invention first aligns the sketch and image in an intermediate feature space, and then replaces sketch features with image features, effectively alleviating the problems of sparse sketch input information and limited semantic expressive power. Under the condition of image features, the denoising network can obtain a more accurate overall structural representation and more detailed geometric reconstruction, significantly improving the accuracy and completeness of 3D shape generation.

[0032] 2. This invention designs a shape adapter to incorporate the geometric features of the input sketch into a denoising network. This shape adapter employs a ResNet encoder and integrates the geometric features of the input sketch into the denoising network through a cross-attention mechanism, strengthening the structural constraint relationship between the sketch and the generated shape. This mechanism can dynamically capture the deviation between the geometric features of the input sketch and the features of the intermediate network layers of the denoising network, effectively improving the structural matching degree of 3D shape generation and solving the problem of inconsistency or geometric deformation between the generated 3D shape and the input sketch in existing methods.

[0033] 3. The method of this invention adopts a phased training and ensemble inference framework design, decoupling image retrieval and sketch control during training and using them collaboratively. This fully leverages the semantic understanding and feature extraction capabilities of the pre-trained CLIP visual encoder, as well as the high-fidelity generation advantages of the diffusion model. This framework has a clear structure and strong scalability, demonstrating good robustness and generalization ability when facing various input formats, including synthetic sketches and real hand-drawn sketches, and possesses significant practical application value. Attached Figure Description

[0034] Figure 1 This is an architectural diagram of the method of the present invention.

[0035] Figure 2 This is a schematic diagram illustrating the reasoning process of the method of the present invention.

[0036] Figure 3 This is an architecture diagram of the shape adapter.

[0037] Figure 4 This is a multi-view effect illustration of an embodiment of the present invention. Detailed Implementation

[0038] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0039] like Figure 1 As shown, this embodiment discloses a 3D shape generation method that integrates image retrieval and sketch control, mainly comprising two stages: First, a sketch-image mapping network is trained to enhance the geometric feature representation of the input sketch through image retrieval, making it closer to the image feature level; second, based on a diffusion generation model, a shape adapter combined with a cross-attention mechanism is trained to incorporate the geometric features of the input sketch into the denoising network to enhance control. The specific implementation of each stage will be described in detail below.

[0040] 1) The first stage involves training a sketch-image mapping network. Feature mapping further aligns sketch features with corresponding image features, mapping them to a new intermediate feature space for subsequent sketch-based image retrieval. Specifically, this embodiment uses a pre-trained CLIP visual encoder to extract sketch and image features, leveraging CLIP's powerful semantic understanding capabilities to map them to a shared feature space. To optimize alignment, this embodiment introduces a feature mapper after extraction and uses triplet loss to update the network parameters of the feature mapper. During training, triplet pairs are constructed consisting of an anchor sketch, a positive image, and a negative image. The feature mapper is constrained so that the cosine similarity between the intermediate features of the anchor sketch and the intermediate features of the positive image is significantly higher than the cosine similarity between the intermediate features of the anchor sketch and the intermediate features of the negative image. The triplet loss function is defined as follows:

[0041] L triplet =max(0,d(f) s ,f p )-d(f s ,f n )+m)

[0042] Among them, f s f p and f nd(·,·) represents the intermediate features of the anchor sketch, the intermediate features of the positive sample image, and the intermediate features of the negative sample image, respectively; d(·,·) is the cosine similarity between features; m is the minimum boundary of the distance difference between positive and negative samples, used to improve the feature discrimination ability.

[0043] In the process of reasoning, such as Figure 2 As shown, this embodiment retrieves the k most similar images to the sketch by calculating the cosine similarity between the intermediate features of the sketch and the intermediate features of the image. The formula for calculating the cosine similarity is as follows:

[0044]

[0045] Where f and f i These represent intermediate features from the sketch and intermediate features from a specific image in the image database, respectively. After retrieving k most similar images, this embodiment uses a pre-trained CLIP visual encoder to extract their features again and perform feature fusion. Specifically, the retrieved k image features are fused using a max-pooling strategy to retain representative semantic details, including edge contours, and to compress redundant information, as defined below:

[0046] F fused [j] = max(F1[j],F2[j],…,F k [j]), j∈(1,N)

[0047] In the formula, F k F represents the features of the k-th image; fused The final image features are fused, where N is the dimension of the features. Finally, the resulting image features are used as the master condition input to the denoising network of the downstream diffusion generation model.

[0048] 2) In the second stage of modeling, this embodiment introduces and trains a shape adapter based on SDFusion, drawing on the design concept of T2I-Adapter in text-to-image generation tasks. The aim is to dynamically capture the discrepancies between the geometric features of the input sketch and the intermediate network layer features of the denoising network. The core objective of this shape adapter is to introduce the geometric features of the input sketch into the denoising network, thereby improving the consistency between the generated 3D shape and the input sketch.

[0049] like Figure 3 As shown, the shape adapter uses a ResNet encoder to extract the geometric features of the input sketch. Then, a cross-attention (cross-attn) mechanism is introduced to integrate the geometric features of the input sketch into the intermediate layers of the denoising network, automatically aligning corresponding structural positions. The cross-attention mechanism can be expressed by the following formula:

[0050]

[0051] In the formula, Q represents the query matrix; K and V represent the key and value matrices, respectively; d K `T` is a scaling factor used to normalize the attention score; `T` represents transpose; `softmax(·)` represents the normalization operation. This automatic alignment allows the geometric features of the input sketch to precisely control the 3D shape generation denoising network, further improving the responsiveness of 3D shape generation to the input sketch.

[0052] In terms of training strategy, this embodiment also adopts the T2I-Adapter concept, keeping the weights of the original diffusion generation model frozen and training only the shape adapter. This strategy not only preserves the pre-trained diffusion generation model's ability in 3D shape generation tasks, but also ensures that the geometric features of the input sketch can effectively contribute to 3D shape generation.

[0053] 3) In the inference phase, this embodiment integrates the sketch-image mapping network and shape adapter trained in the two phases into a unified inference framework, realizing a complete process from input sketch to 3D shape generation, such as... Figure 2 As shown.

[0054] First, the input sketch is mapped to intermediate features of the input sketch through a sketch-image mapping network. Then, based on cosine similarity, the top k image intermediate features semantically closest to these intermediate features are retrieved, resulting in k approximate images. To construct the image library, this embodiment uses a dataset built on DISN, selecting corresponding images according to their IDs in the ShapeNet-Synthetic test set. The intermediate features of these images are extracted using the sketch-image mapping network trained in the first stage and stored as index intermediate features for subsequent retrieval. Subsequently, the retrieved k approximate images have their image features extracted by a pre-trained CLIP visual encoder, and feature fusion is performed using max pooling to generate the final image features. These final image features are used to replace the input sketch features directly extracted by the pre-trained CLIP visual encoder, thereby indirectly enhancing the input sketch features.

[0055] Next, the final image features are used as the master condition input to the denoising network of the downstream diffusion generation model to perform the 3D shape generation task. In this denoising network, the geometric features of the input sketch are integrated into the denoising network through a trained shape adapter and a cross-attention mechanism, further modulating the features of the intermediate network layers of the denoising network to better match the input sketch.

[0056] Ultimately, the 3D latent vectors generated by the denoising network are restored to a complete 3D symbolic distance function representation by the decoder of the 3D vector quantization variational autoencoder, realizing the generation of a 3D shape from a sketch to one with clear structure and consistent semantics.Figure 4 The example demonstrates the generation effect on chair-like objects. It can be seen that the final generated Mesh presents a complete and consistent geometric structure from different viewpoints.

[0057] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for generating three-dimensional shapes that integrates image retrieval and sketch control, characterized in that, Includes the following steps: S1: Train the sketch-image mapping network, build an image library, input a sketch, and extract features from the sketch and all images in the image library to obtain intermediate features of the sketch and intermediate features of all images; S2: Calculate the cosine similarity between the sketch middle features and all image middle features, and retrieve the top k image middle features with the highest cosine similarity to the sketch middle features; S3: Match the intermediate features of the first k images to the image library to obtain the corresponding k images; S4: Feed the corresponding k images into the pre-trained CLIP visual encoder to extract features, and perform max pooling fusion on the dimensions of the extracted features to form the final image features; S5: The final image features are used as the main condition input to the denoising network of the downstream diffusion generation model. A shape adapter is constructed, and a ResNet encoder is used to extract the geometric features of the input sketch. The geometric features of the input sketch are then incorporated into the denoising network through a cross-attention mechanism. S6: After the denoising network completes the denoising, the three-dimensional latent vector in the latent space corresponding to the denoising network is obtained. The decoder of the three-dimensional vector quantization variational autoencoder is used to decode the three-dimensional latent vector and restore it to the three-dimensional symbolic distance function representation, thereby completing the modeling task from sketch to three-dimensional shape.

2. The three-dimensional shape generation method integrating image retrieval and sketch control according to claim 1, characterized in that, The sketch-image mapping network is used to map sketches to all images in the image library as intermediate features with a uniform form to meet the retrieval requirements of cosine similarity. The sketch-image mapping network consists of a pre-trained CLIP visual encoder and a trainable feature mapper. During training, the pre-trained CLIP visual encoder extracts features from the sketches and corresponding images in the training data to obtain sketch features and corresponding image features in the training data. Then, the feature mapper embeds the sketch features in the training data for further adjustment, making its features closer to the corresponding image features and farther away from other image features in the training data. During training, triplet training data is constructed, including anchor sketches s, positive sample images p, and negative sample images n, and triplet loss L is used. triplet Supervised training is performed to optimize the parameters of the feature mapper, with triplet loss L. triplet The definition is as follows: L triplet =max(0,d(f s ,f p )-d(f s ,f n )+m) In the formula, f s f p f n These represent the intermediate features of the anchor point sketch, the intermediate features of the positive sample image, and the intermediate features of the negative sample image, respectively. d(·,·) represents the cosine similarity between features; m is the minimum boundary of the distance difference between positive and negative samples, used to improve the feature discrimination ability.

3. The three-dimensional shape generation method integrating image retrieval and sketch control according to claim 1, characterized in that, Calculate the cosine similarity between the sketch's intermediate features and the intermediate features of all images in the image database, sort them by cosine similarity, and select the k most similar image intermediate features, as shown in the following formula: In the formula, f represents the intermediate feature of the sketch; f i This represents the intermediate features of an image in the image database; CosSim(·,·) represents the cosine similarity between the two images.

4. The three-dimensional shape generation method integrating image retrieval and sketch control according to claim 1, characterized in that, The pre-trained CLIP visual encoder is used to re-extract features from the corresponding k images, resulting in k image features. To preserve representative semantic details, including edge contours, and to compress redundant information, a max pooling strategy is used for fusion, defined as follows: F fused [j]=max(F1[j],F2[j],…,F k [j]),j∈(1,N) In the formula, F k The features of the k-th image; F fused N represents the final image features after fusion, where N is the dimension of the features.

5. The three-dimensional shape generation method integrating image retrieval and sketch control according to claim 1, characterized in that, The shape adapter encodes the geometric structure of the sketch using a structured neural network ResNet to obtain the geometric features of the input sketch, and integrates the obtained geometric features of the input sketch into the intermediate network layer of the denoising network through a cross-attention mechanism to achieve control guidance. In this context, the geometric features of the obtained input sketch are used as the query matrix Q, the intermediate network layer features of the denoising network are used as the key matrix K and the value matrix V, and the cross-attention mechanism is specifically represented as follows: In the formula, d K represents the scaling factor; T represents the transpose; the cross-attention mechanism enables fine-grained interaction between the geometric features of the input sketch and the intermediate network layer features of the denoising network, automatically aligning the corresponding structural positions.

6. The three-dimensional shape generation method integrating image retrieval and sketch control according to claim 1, characterized in that, The generation of the three-dimensional shape relies on a latent space constructed by a three-dimensional vector quantization variational autoencoder. In this latent space, the three-dimensional shape X, represented by a three-dimensional signed distance function, is encoded by the encoder of the three-dimensional vector quantization variational autoencoder. Compressed encoding is performed to obtain the three-dimensional latent vector z. t ,Right now: A conditional diffusion model denoising network is constructed in this latent space. The final image features and the geometric features of the input sketch are used as conditional inputs to guide the gradual denoising process of the initial noise in the denoising network, generating a 3D latent vector consistent with the sketch structure and semantics. Finally, the generated 3D latent vector z′0 is vector quantized and input into the decoder D of the 3D vector quantization variational autoencoder. τ Decode the data to restore it to the three-dimensional symbolic distance function representation X′, i.e.: X′=D τ (VQ(z′0)) The three-dimensional symbolic distance function represents the explicit three-dimensional geometric mesh as the final output after X′ is processed.

Images