A multimodal automatic classification method for architectural sketches based on DINOv2 and CLIP

Abstract

The present invention relates to the field of image classification technology, and specifically to a multimodal automatic classification method for architectural sketches based on DINOv2 and CLIP, which includes collecting architectural sketch data samples; performing data augmentation on the architectural sketch data samples to form a data set of architectural sketches; performing text tagging on the data set of architectural sketches; performing quality inspection on the labeled text; using the obtained data set to jointly train visual features extracted by DINOv2 and text embedding features of CLIP, and jointly training to obtain a feature fusion layer; and using a classification head to classify and predict the fused features. The method realizes efficient and automated base map classification, and does not need to rely on manual classification of the base map when generating the map, thereby solving the problems of low accuracy, low efficiency, and high error rate of manual judgment, and at the same time solving the accuracy problem of automated sketch classification under the condition of insufficient samples.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to a multimodal automatic classification method for architectural sketches based on DINOv2 and CLIP. Background Art

[0002] In the process of using AIGC (Artificial Intelligence Generated Content) to generate architectural renderings, the use of architectural sketches as base maps has many advantages, and the accuracy of architectural sketch classification has an important impact on the quality of generated images. As an important input for Stable Diffusion image generation technology to assist in image generation, accurate classification of architectural sketches can more effectively assist in generating high-quality images.

[0003] However, the current traditional classification method mainly relies on human experience for classification. Due to the differences in experience and abilities of different personnel, manual classification of architectural sketches has the disadvantages of low accuracy, low efficiency, high error rate, and high initial learning threshold. It is difficult to meet the needs of the rapidly developing AI in the construction industry.

[0004] With the advancement of deep learning and computer vision technology, automated classification has become possible, but it still requires a lot of data annotation and resource-intensive deep learning models, facing problems such as insufficient sample data and low classification accuracy. Currently, the market lacks an accurate, efficient and automated classification method that can accurately and efficiently classify architectural sketches. Summary of the invention

[0005] The purpose of the present invention is to provide a multimodal automatic classification method for architectural sketches based on DINOv2 and CLIP, which solves the problems of insufficient sample data and low classification accuracy.

[0006] To achieve the above object, the present invention provides a multimodal automatic classification method for architectural sketches based on DINOv2 and CLIP, comprising the following steps:

[0007] Collecting architectural sketch data samples;

[0008] Use the Stable Diffusion model and IP-Adapter to augment the architectural sketch data samples to generate more pictures of the same style. The original collected architectural sketch data samples and the augmented pictures together form the architectural sketch data set.

[0009] Text labeling for a dataset of architectural sketches;

[0010] Perform quality check on the labeled text to obtain architectural sketches and corresponding architectural sketch datasets with text labels;

[0011] Jointly train the visual features extracted by DINOv2 and the text embedding features of CLIP to obtain the feature fusion layer;

[0012] Use the classification head to perform classification prediction on the fused features and output the classification label and confidence score of the building sketch.

[0013] The Stable Diffusion model and IP-Adapter are used to augment the architectural sketch data samples to generate more pictures of the same style. The originally collected architectural sketch data samples and the augmented pictures together form a dataset of architectural sketches. The steps also include:

[0014] The collected architectural sketches are used as the base map, and corresponding text prompts are defined for the architectural sketches. The text prompts include detailed descriptions of the architectural style and features that are expected to be generated.

[0015] Use Control Net for conditional control and select IP-Adapter as the model;

[0016] For each initial background image, the Stable Diffusion model is used in combination with the corresponding text prompts to generate multiple images with similar styles through IP-Adapter;

[0017] The above process is repeated several times to generate a sufficient number of augmented datasets.

[0018] The step of performing text labeling on the dataset of architectural sketches further includes:

[0019] Text labels are added to the dataset of architectural sketches manually or semi-automatically, and the text labels include architectural style, type and design details.

[0020] The obtained data set is used in combination with LoRA technology to perform lightweight model fine-tuning on the DINOv2 model, and the fusion layer is trained together with CLIP to train the fine-tuned DINOv2 model. The steps also include:

[0021] Load the pre-trained DINOv2 model;

[0022] Introducing the LoRA fine-tuning mechanism in the self-attention layer of the DINOv2 model;

[0023] Insert the low-rank matrix into the weight parameters of the query and key projection matrices of the self-attention layer;

[0024] Determine the rank of the low-rank matrix used in LoRA fine-tuning;

[0025] Fine-tune the DINOv2 model using the obtained dataset. During the training process, only the parameters of the low-rank matrix will be updated, while the original DINOv2 model parameters remain unchanged.

[0026] Through the gradient descent optimization algorithm, the parameters of the low-rank matrix are adjusted to optimize the performance of the DINOv2 model on specific tasks;

[0027] Evaluate the performance of the fine-tuned DINOv2 model on the validation set to ensure that the model maintains good performance in a small sample training environment;

[0028] According to the performance of the model on the validation set, adjust the parameters of LoRA fine-tuning, including the rank of the low-rank matrix or the learning rate of fine-tuning, and repeat the fine-tuning process until satisfactory performance is achieved.

[0029] The classification head is used to perform classification prediction on the fused features, and the classification label and confidence score of the building sketch are output. The step further includes:

[0030] The trained model is exported to ONNX format and optimized to improve the inference speed. The trained model includes DINOv2, CLIP, feature fusion layer, and classification head.

[0031] The present invention discloses a multimodal automatic classification method for architectural sketches based on DINOv2 and CLIP. DINOv2 provides finer-grained image feature extraction, supplements the deficiencies of CLIP in understanding image details, and enhances spatial perception capabilities, especially in base map classification, it can capture more image details and structural information. By combining the DINOv2 and CLIP models, the multimodal interaction of text prompts and sketches is utilized to achieve precise control of architectural sketch classification. Through DINOv2+LoRA fine-tuning, DINOv2 is lightly fine-tuned through the LoRA technology to reduce computing resource consumption and adapt to small sample scenarios. At the same time, data augmentation is performed through Stable Diffusion and IP-Adapter, which significantly increases the amount of training data and the diversity of samples, and improves the robustness of the model. Efficient and automated base map classification is achieved without relying on manual review of the base map, solving the problems of low accuracy, low efficiency, and high error rate of manual judgment, and solving the accuracy problem of automated sketch classification under insufficient sample conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0033] Figure 1It is a flow chart of the training and implementation method of the architectural sketch classification of the present invention.

[0034] Figure 2 It is a schematic diagram of the effect of using the Stable Diffusion model and IP-Adapter to perform data augmentation in the present invention.

[0035] Figure 3 It is a schematic diagram of the correspondence between the architectural sketch type and the ControlNet model of the present invention.

[0036] Figure 4 It is a flowchart of the steps of the multimodal automatic classification method of architectural sketches based on DINOv2 and CLIP of the present invention. DETAILED DESCRIPTION

[0037] Embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, but should not be construed as limiting the present invention.

[0038] The first embodiment of the present application is:

[0039] See also Figures 1 to 4 ,in, Figure 1 It is a flow chart of the training and implementation method of the architectural sketch classification of the present invention. Figure 2 It is a schematic diagram of the effect of using the Stable Diffusion model and IP-Adapter to perform data augmentation in the present invention. Figure 3 It is a schematic diagram of the correspondence between the architectural sketch type and the ControlNet model of the present invention. Figure 4 It is a flowchart of the steps of the multimodal automatic classification method of architectural sketches based on DINOv2 and CLIP of the present invention.

[0040] The present invention provides a multimodal automatic classification method for architectural sketches based on DINOv2 and CLIP, comprising the following steps:

[0041] S101: Collect architectural sketch data samples;

[0042] Specifically, we prepare architectural sketch data samples and collect architectural sketches for fine-tuning DINOv2 and joint training with CLIP, forming an initial dataset containing architectural sketches and text prompts for subsequent data augmentation and feature extraction.

[0043] S102: Use the Stable Diffusion model and IP-Adapter to augment the architectural sketch data samples to generate more pictures of the same style. The original collected architectural sketch data samples and the augmented pictures together form the architectural sketch data set;

[0044] Specifically, the initial image collected in step S101 is used as the base image, and the Stable Diffusion model is used in combination with text prompts. The IP-Adapter uses the initial collected image as the base image reference, and adjusts the details of the generated image according to the text prompts to generate more pictures of the same style. This step is to augment the data of the architectural sketch and enhance the diversity of the data set. Specifically, it includes: Step 1, define the corresponding text prompts for each building type, for example:

[0045] Residential Building: "A cozy residential building with a warm and inviting atmosphere."

[0046] Office Building: "A modern office building with large windows and sleek design."

[0047] Commercial complex: "A bustling commercial complex with shops, restaurants, and entertainment venues."

[0048] School: "A spacious school campus with classrooms, a library, and sports fields."

[0049] Commercial Street: "A vibrant commercial street lined with shops and cafes."

[0050] Cultural buildings: "An elegant cultural center with museums and theaters."

[0051] Sports stadium: "A state-of-the-art sports stadium with seating forthousands of spectators."

[0052] Step 2. Use ControlNet for conditional control. Select the control type IP-Adapter to generate an image similar to the picture style. Preprocessor: ip-adapter_clip_sd15, model: ip-adapter_sd15. IP-Adapter can adjust the image towards different styles and categories according to the settings, and make the initial image closer to the style of a specific building type, in preparation for further refinement.

[0053] The augmented architectural sketch samples are obtained, which cover different architectural styles, categories and details for subsequent model fine-tuning and training. The original collected data samples and the augmented data samples together form the dataset of architectural sketches.

[0054] S103: perform text labeling on the dataset of architectural sketches;

[0055] Specifically, text labeling is performed on the dataset of architectural sketches by manually or semi-automatically adding text labels describing their architectural style, type, design details, etc.

[0056] S104: Perform a quality check on the labeled text to obtain a building sketch and a corresponding building sketch dataset with text labels;

[0057] Specifically, the labeled text is quality checked to ensure that the text description of each architectural sketch is accurate, comprehensive, and consistent with the content of the actual image. A dataset of architectural sketches and corresponding architectural sketches with text labels is obtained. This dataset can be used to train the corresponding parameters of the CLIP model.

[0058] S105: jointly training the visual features extracted by DINOv2 and the text embedding features of CLIP, and jointly training to obtain a feature fusion layer, wherein the fusion layer is obtained by training to obtain parameters of the fusion layer;

[0059] Specifically, the obtained dataset is used in combination with LoRA (Low-Rank Adaptation) to perform lightweight model fine-tuning. The low-rank matrix is ​​inserted into the weight parameters of the query and key projection matrices of the self-attention layer of the DINOv2 model, and the model is optimized by adjusting a small number of parameters to adapt to the small sample training environment. The fine-tuned DINOv2 model can be trained, which has the ability to extract visual features of architectural sketches and perform multi-label classification.

[0060] Combine the visual features extracted by DINOv2 with the text embedding features of CLIP, and train parameters (such as feature fusion layer) that are used to fuse the features of DINOv2 and CLIP. Fusion can be performed through methods such as attention mechanism. The calculation process is as follows:

[0061] -Let the visual features extracted by DINOv2 be vector ,

[0062] -Let the text embedding features extracted by CLIP be vector ,

[0063] Concatenate the visual features of DINOv2 and the text features of CLIP:

[0064]

[0065] Generate query, key, and value vectors:

[0066]

[0067]

[0068]

[0069] Calculate the attention score:

[0070] Assume there are h heads, and the output of each head is recorded as headi:

[0071]

[0072] The results of all headers are combined together through the concatenation operation:

[0073]

[0074] After the multi-head self-attention output, residual connection and normalization are performed:

[0075]

[0076] Through the above process, a feature fusion layer is obtained to process the visual features extracted by DINOv2 and the text features extracted by CLIP. This layer realizes the alignment and fusion of multimodal features through the attention mechanism, and is finally used to generate the classification results of the architectural sketch.

[0077] After the model training is completed, the visual and text features extracted by DINOv2 and CLIP are aligned and fused through the feature fusion layer, and the fused features are passed to the classifier for classification prediction.

[0078] S106: Use the classification head to perform classification prediction on the fused features, and output the classification label and confidence score of the building sketch.

[0079] Specifically, the fused features are used for classification prediction, and the classification head (fully connected layer + softmax) is used to directly output the classification results. The calculation process is as follows:

[0080] -Let the feature vector output by the fusion layer be , where C is the number of categories for classification.

[0081] - Each element represents the unnormalized score of the model for category j.

[0082] Use the softmax activation function to calculate the probability of each category. , the formula is as follows:

[0083]

[0084] in, represents the probability of belonging to category j.

[0085] represents the score of the jth category in the feature vector.

[0086] -The scores are converted into probabilities between 0 and 1 through the softmax function, and the sum of the probabilities of all categories is 1.

[0087] S107: Model export and optimization.

[0088] In order to achieve an efficient inference process, the entire model (including DINOv2, CLIP, feature fusion layers, and classifiers) will be exported to the ONNX format so that it can be uniformly loaded and run through the ONNX Runtime during the inference phase. ONNX (Open NeuralNetwork Exchange) is a widely supported cross-platform model format. With ONNX, you can export the entire trained model, including DINOv2, CLIP, feature fusion layers, and classifiers, as a single ONNX file. ONNX supports efficient inference optimization and can be accelerated on platforms such as ONNX Runtime and TensorRT.

[0089] To improve inference performance, the model can be optimized when exported, using ONNX GraphOptimization or quantization methods to reduce the model size and increase inference speed.

[0090] Reasoning and implementation process during user use:

[0091] 1. When drawing, the user enters the architectural sketch and text prompts.

[0092] 2. Input preprocessing: The system preprocesses the architectural sketches and text prompts uploaded by users. For image input, it performs operations such as resizing and denoising; for text input, it does not require manual word segmentation or vectorization. Users directly input the original text prompt, and the CLIP model automatically processes the input text and converts it into an embedded vector.

[0093] 3. End-to-end reasoning: The preprocessed images and text are input into the end-to-end ONNX model, DINOv2 extracts the visual features of the image, and CLIP extracts the embedding features of the text. These two features are then fused through the feature fusion layer. The classification head will perform multi-label classification on the architectural sketches and output the category label of each sketch as well as the confidence score. These labels can describe the style of the building, the type of structure, or specific design elements.

[0094] 4. Based on the specific classification labels of the sketch, the system will automatically recommend the most suitable ControlNet type, such as line art extraction (LineArt Extraction), depth estimation (Depth Estimation), etc., to assist in subsequent architectural image generation or further processing.

[0095] The method advantages of the present invention are:

[0096] 1. Achieve automated and more accurate architectural sketch classification: DINOv2 provides more fine-grained image feature extraction, complementing the shortcomings of CLIP in understanding image details, and enhancing spatial perception, especially in base map classification, it can capture more image details and structural information. By combining DINOv2 and CLIP models, and using multimodal interaction between text prompts and sketches, precise control of architectural sketch classification is achieved.

[0097] 2. Reduce the amount of sample collection and labeling and other complex manual work: Through DINOv2+LoRA fine-tuning, DINOv2 is lightly fine-tuned through LoRA technology to reduce computing resource consumption and adapt to small sample scenarios. At the same time, data augmentation is performed through Stable Diffusion and IP-Adapter, which significantly increases the amount of training data and sample diversity, and improves the robustness of the model.

[0098] What is disclosed above is only one or more preferred embodiments of the present application, and cannot be used to limit the scope of rights of the present application. Ordinary technicians in this field can understand that all or part of the processes of implementing the above embodiments and equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A multimodal automatic classification method for architectural sketches based on DINOv2 and CLIP, characterized in that: The following steps are involved: Collecting architectural sketch data samples; Use the Stable Diffusion model and IP-Adapter to augment the architectural sketch data samples to generate more pictures of the same style. The original collected architectural sketch data samples and the augmented pictures together form the architectural sketch data set. Text labeling for a dataset of architectural sketches; Perform quality check on the labeled text to obtain architectural sketches and corresponding architectural sketch datasets with text labels; Jointly train the visual features extracted by DINOv2 and the text embedding features of CLIP to obtain the feature fusion layer; Use the classification head to perform classification prediction on the fused features and output the classification label and confidence score of the building sketch; Using the obtained data set and combining the LoRA technology, the DINOv2 model is lightly fine-tuned, and the fusion layer is trained together with CLIP to train the fine-tuned DINOv2 model. The steps also include: Load the pre-trained DINOv2 model; Introducing the LoRA fine-tuning mechanism in the self-attention layer of the DINOv2 model; Insert low-rank matrices into the weight parameters of the query and key projection matrices of the self-attention layer; Determine the rank of the low-rank matrix used in LoRA fine-tuning; Fine-tune the DINOv2 model using the obtained dataset. During the training process, only the parameters of the low-rank matrix will be updated, while the original DINOv2 model parameters remain unchanged. Through the gradient descent optimization algorithm, the parameters of the low-rank matrix are adjusted to optimize the performance of the DINOv2 model on the corresponding task; Evaluate the performance of the fine-tuned DINOv2 model on the validation set to ensure that the model maintains good performance in a small sample training environment; According to the performance of the model on the validation set, adjust the parameters of LoRA fine-tuning, including the rank of the low-rank matrix or the learning rate of fine-tuning, and repeat the fine-tuning process until satisfactory performance is achieved.

2. The multimodal automatic classification method of architectural sketches based on DINOv2 and CLIP as claimed in claim 1, characterized in that: The Stable Diffusion model and IP-Adapter are used to augment the architectural sketch data samples to generate more pictures of the same style. The originally collected architectural sketch data samples and the augmented pictures together form a dataset of the architectural sketch. The steps also include: The collected architectural sketches are used as the base map, and corresponding text prompts are defined for the architectural sketches. The text prompts include detailed descriptions of the architectural style and features that are expected to be generated. Use Control Net for conditional control and select IP-Adapter as the model; For each initial base image, the Stable Diffusion model is used in combination with the corresponding text prompts to generate multiple images through IP-Adapter; The above process is repeated multiple times to generate augmented datasets.

3. The multimodal automatic classification method of architectural sketches based on DINOv2 and CLIP as claimed in claim 1, characterized in that: Performing text labeling on the dataset of architectural sketches, the step further includes: Text labels are added to the dataset of architectural sketches manually or semi-automatically, and the text labels include architectural style, type and design details.

4. The multimodal automatic classification method of architectural sketches based on DINOv2 and CLIP as claimed in claim 1, characterized in that: The fused features are classified and predicted using a classification head, and the classification labels and confidence scores of the building sketches are output. The steps further include: The trained model is exported to ONNX format and optimized to improve the inference speed. The trained model includes DINOv2, CLIP, feature fusion layer, and classification head.

Images