A method, device and equipment for architectural image retrieval based on multimodality

Abstract

The present application discloses a multi-modal architectural image retrieval method, device and equipment, which obtains a set of architectural example images, a set of positive prompt words and a set of reverse prompt words input by a user; performs feature extraction on the set of architectural example images, the set of positive prompt words and the set of reverse prompt words, and calculates the image feature mean, the forward feature mean and the reverse feature mean; calculates the correlation between the set of architectural example images and each architectural image in the architectural image library according to the image feature mean, the forward feature mean and the reverse feature mean. The present application performs image retrieval through data of text modality and image modality, calculates the correlation between architectural example images and each architectural image in the architectural image library according to text features and image features, and uses the top N architectural images with the highest correlation as the target images for the LoRA method. The target image obtained has a high correlation with the architectural example image, and no manual screening is required, which further improves work efficiency.

Description

Technical Field

[0001] The present application relates to the technical field of cross-modal retrieval, and more specifically, to a multi-modal architectural image retrieval method, device and equipment. Background Art

[0002] In domain-specific image generation tasks, it is necessary to implement a general image generation model to achieve directional adaptation to domain-specific data distribution. However, large models with more than billions of parameters and strong capabilities usually present huge overhead in fine-tuning to adapt to their downstream tasks. To solve this problem, the LoRA (Low-Rank Adaptation) method is usually used. The LoRA method freezes the weights of the pre-trained image generation model and injects trainable layers into each embedding layer of each deep model for fine-tuning. Therefore, there is no need to calculate gradients for the model weights, which greatly reduces the number of parameters that need to be trained and reduces the memory requirements of the GPU.

[0003] At present, the LoRA method is also widely used in the task of generating architectural renderings in the field of architectural design. Users need to search the Internet based on the architectural design sketch of the current building through text-image retrieval or image-image retrieval to obtain similar architectural drawings of the current building. However, since the images searched on the Internet are not highly relevant to the current building, users are required to filter out similar architectural images with relatively high relevance from the searched similar architectural images as image data used by the LoRA method. Existing architectural retrieval image retrieval methods only rely on text or images for retrieval. The similar architectural images searched are less relevant to the current building and require further manual screening, which makes the work efficiency low. Summary of the invention

[0004] In view of this, the present application provides a multimodal architectural image retrieval method, device and equipment to solve the problem that the existing architectural image retrieval method only relies on text or images for retrieval, and the similar architectural images searched have low relevance to the current building and require further manual screening, resulting in low work efficiency.

[0005] To achieve the above objectives, the proposed solution is as follows:

[0006] A multimodal architectural image retrieval method, comprising:

[0007] Obtaining a set of example building images, a set of positive prompt words, and a set of reverse prompt words input by a user;

[0008] The ViT image encoder in the CLIP model is used to extract features from the building sample image set to obtain an image feature set, and the image feature mean of the image feature set is calculated;

[0009] The BERT text encoder in the CLIP model is used to extract features from the positive prompt word set and the reverse prompt word set respectively to obtain a positive feature set and a reverse feature set, and the positive feature mean of the positive feature set and the reverse feature mean of the reverse feature set are calculated;

[0010] According to the image feature mean, the forward feature mean and the reverse feature mean, the correlation between the building example image set and each building image in the building image library is calculated;

[0011] Sort the relevance from highest to lowest;

[0012] The top N building images ranked by relevance are taken as target building images, where N is a positive integer.

[0013] Preferably, before calculating the correlation between the building example image set and each building image in the building image library according to the image feature mean, the forward feature mean and the reverse feature mean, the method further includes:

[0014] The image feature of each building image in the building image library is extracted through the ViT image encoder in the CLIP model to obtain the image feature vector of each building image;

[0015] The BERT text encoder in the CLIP model is used to extract text features from each building image in the building image library to obtain the text feature vector of each building image.

[0016] Preferably, calculating the correlation between the set of example building images and each building image in the building image library according to the image feature mean, the forward feature mean and the reverse feature mean includes:

[0017] according to Calculate the correlation between the building example image set and each building image in the building image library, where Q is the query condition, v is the building image, and I v is the image feature vector of the building image, T v is the text feature vector of the building image, and I v and T v The transpose of I p is the mean value of image features, T q is the positive feature mean, S b is the reverse feature mean, and α, β, and γ are manually set hyperparameters.

[0018] Preferably, it also includes:

[0019] The image feature vector and text feature vector corresponding to each building image in the building image library are concatenated to obtain the overall representation vector of each building image.

[0020] Preferably, it also includes:

[0021] The image feature mean, the forward feature mean and the reverse feature mean are concatenated to obtain the query representation vector.

[0022] Preferably, after calculating the correlation between the set of example building images and each building image in the building image library according to the image feature mean, the forward feature mean and the reverse feature mean, the method further includes:

[0023] The ANN indexing algorithm is used to retrieve the overall representation vector of each building image to obtain an initial building image set.

[0024] Preferably, calculating the correlation between the set of example building images and each building image in the building image library according to the image feature mean, the forward feature mean and the reverse feature mean includes:

[0025] according to Calculate the correlation between the building example image set and each building image in the initial building image set, where Q is the query condition, v is the building image, and E Q is the query representation vector, E Q =[αI p βT q ; -γS b ], I p is the mean value of image features, T q is the positive feature mean, S b is the reverse feature mean, α, β, γ are manually set hyperparameters, For E v The transpose of E v is the overall representation vector of each building image.

[0026] A multi-modal architectural image retrieval device, comprising:

[0027] A condition acquisition unit, used to acquire a set of building example images, a set of positive prompt words, and a set of reverse prompt words input by a user;

[0028] An image feature extraction unit is used to extract features from a set of example building images through a ViT image encoder in a CLIP model to obtain an image feature set, and calculate an image feature mean of the image feature set;

[0029] A text feature extraction unit, used to extract features from the positive prompt word set and the reverse prompt word set respectively through the BERT text encoder in the CLIP model to obtain a positive feature set and a reverse feature set, and calculate a positive feature mean of the positive feature set and a reverse feature mean of the reverse feature set;

[0030] A correlation calculation unit, used to calculate the correlation between the building example image set and each building image in the building image library according to the image feature mean, the forward feature mean and the reverse feature mean;

[0031] A sorting unit, used to sort the relevance from high to low;

[0032] The image determination unit is used to take the top N building images ranked in relevance as target building images.

[0033] A multi-modal architectural image retrieval device comprises: a memory and a processor;

[0034] The memory is used to store programs;

[0035] The processor is used to execute the program to implement the various steps of the aforementioned multi-modal building image retrieval method.

[0036] It can be seen from the above technical scheme that the multimodal architectural image retrieval method provided by the present application obtains the architectural example image set, the positive prompt word set and the reverse prompt word set input by the user; performs feature extraction on the architectural example image set to obtain the image feature set, and calculates the image feature mean of the image feature set; performs feature extraction on the positive prompt word set and the reverse prompt word set to obtain the forward feature set and the reverse feature set, and calculates the forward feature mean and the reverse feature mean; calculates the correlation between the architectural example images in the architectural example image set and each architectural image in the architectural image library according to the image feature mean, the forward feature mean and the reverse feature mean. The architectural image retrieval method provided by the present application performs image retrieval based on data of text modality and image modality, calculates the correlation between the architectural example images and each architectural image in the architectural image library according to the text features and the image features, and uses the architectural images with the top N correlations as the target images for the LoRA method. The obtained target image has a high correlation with the architectural example image, and no manual screening is required, which further improves work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0038] Figure 1 A flowchart of a multi-modal architectural image retrieval method provided in an embodiment of the present application;

[0039] Figure 2 A schematic diagram of the structure of a multi-modal architectural image retrieval device provided in an embodiment of the present application;

[0040] Figure 3 A hardware structure block diagram of a multimodal architectural image retrieval device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0042] First, combine Figure 1 The present invention provides a multi-modal architectural image retrieval method. Figure 1 As shown, the method may include:

[0043] Step S01, obtaining a set of example building images, a set of positive prompt words, and a set of negative prompt words input by a user.

[0044] Specifically, the user can input a set of example building images, a set of positive prompt words, and a set of negative prompt words for retrieval. The set of example building images includes n example building images p1, p2, ..., p3, p4, p5, p6, p7, p8, p9, p10, p11, p12, ..., p13, p14, p15, p16, p17, p18, p19, p20, p21, p22, p23, p24, p25, p26, p27, p28, p29, p30, p31, p32, p33, p34, p35, p40, p41, p42, p43, p44, p45, p46, p47, p48, p49, p50, p51, p52, p53, p54, p55, p66, p70, p71, n ; The forward prompt word set includes m forward prompt words input by the user; the reverse prompt word set includes k reverse prompt words input by the user, and n, m and k are all integers greater than or equal to 0.

[0045] Step S02: extract features from the building example image set by using the ViT image encoder in the CLIP model to obtain an image feature set, and calculate the image feature mean of the image feature set.

[0046] Specifically, the image features of each building example image are extracted by using the ViT (Vision Transformer) image encoder in the pre-trained CLIP (Contrastive Language-Image Pre-training) model to obtain the image features of each building example image and obtain an image feature set. The image feature set includes Calculate the average value of each feature vector in the image feature set to obtain the image feature mean I p .in, The image feature mean I p A unified representation of image modalities for a collection of architectural example images as user input.

[0047] The CLIP model is pre-trained using multiple image-text data pairs as training sets and image feature vectors and text feature vectors as training labels.

[0048] Step S03, using the BERT text encoder in the CLIP model to perform feature extraction on the positive prompt word set and the reverse prompt word set respectively, to obtain a forward feature set and a reverse feature set, and to calculate the forward feature mean of the forward feature set and the reverse feature mean of the reverse feature set.

[0049] Specifically, the pre-trained CLIP model BERT (Bidirectional Encoder Representations from Transformers) text encoder extracts features from the positive prompt word set and the reverse prompt word set input by the user, extracts the text features of the positive prompt word set and the reverse prompt word set, and obtains the positive feature set of the positive prompt word set and the reverse feature set of the reverse prompt word set. The positive feature set includes The reverse feature set includes and through Calculate the positive feature mean T of the positive feature set q ,pass Calculate the reverse feature mean S of the reverse feature set b .

[0050] Step S04, calculating the correlation between the set of example building images and each building image in the building image library according to the image feature mean, the forward feature mean and the reverse feature mean.

[0051] Specifically, the correlation between the building example image set and each building image in the building image library can be calculated based on the image feature mean, forward feature mean and reverse feature mean of the building example image set, as well as the image feature vector and text feature vector of each building image pre-recorded in the building image library.

[0052] Step S05, sorting the correlations in descending order.

[0053] Specifically, the correlations are sorted from high to low. If the correlation corresponding to the building image is higher, it proves that the building image has a higher correlation with the building example image set input by the user, the positive prompt word set has a higher correlation, and the reverse prompt word has a lower correlation.

[0054] Step S06: taking the top N building images in correlation ranking as target building images.

[0055] Specifically, the top N building images ranked in terms of relevance may be taken as target building images, where N is a positive integer.

[0056] The multimodal architectural image retrieval method provided in the embodiment of the present application can obtain the architectural example image set, the positive prompt word set and the reverse prompt word set input by the user; extract the features of the architectural example image set to obtain the image feature set, and calculate the image feature mean of the image feature set; extract the features of the positive prompt word set and the reverse prompt word set to obtain the forward feature set and the reverse feature set, and calculate the forward feature mean and the reverse feature mean; calculate the correlation between the architectural example images in the architectural example image set and each architectural image in the architectural image library according to the image feature mean, the forward feature mean and the reverse feature mean, and obtain the correlation set; sort the correlation in order from high to low; and use the top N architectural images in the correlation as the target architectural image. The architectural image retrieval method provided in the embodiment of the present application performs image retrieval through data of text modality and image modality, calculates the correlation between the architectural example images and each architectural image in the architectural image library according to the text features and the image features, and uses the top N architectural images in the correlation as the target image for the LoRA method. The target image obtained has a high correlation with the architectural example image, and no manual screening is required, which further improves work efficiency.

[0057] In step S04, before calculating the correlation between the set of example building images and each building image in the building image library according to the image feature mean, the forward feature mean, and the reverse feature mean, after the building image is entered into the building image library, the embodiment of the present application may perform the following steps:

[0058] The image feature vector of each building image is obtained by extracting the image features of each building image in the building image library through the ViT image encoder in the CLIP model.

[0059] Specifically, the image feature vector I of each building image recorded in the building image library is extracted by using the ViT image encoder in the pre-trained CLIP model. v .

[0060] The BERT text encoder in the CLIP model is used to extract text features from each building image in the building image library to obtain the text feature vector of each building image.

[0061] Specifically, the Deepbooru model can be used to generate the text label set Π of each building image in the building image library. v , and then the text label set Π is trained by the BERT text encoder in the pre-trained CLIP model vPerform text feature extraction to obtain the text feature vector T of each building image v .

[0062] Based on this, step S04, the process of calculating the correlation between the set of example building images and each building image in the building image library according to the image feature mean, the forward feature mean and the reverse feature mean, may include:

[0063] according to Calculate the correlation between the building example image set and each building image in the building image library, where Q is the query condition, v is the building image, and I v is the image feature vector of the building image, T v is the text feature vector of the building image, and I v and T v The transpose of I p is the mean value of image features, T q is the positive feature mean, S b is the reverse feature mean, α, β, γ are manually set hyperparameters. The query condition Q includes the image feature mean, the forward feature mean and the reverse feature mean, Q = (I p ,T q ,S b ).

[0064] The embodiment of the present application provides a multimodal architectural image retrieval method, which calculates the correlation of each architectural image through the image and text features of each architectural image in the architectural example image set and the architectural image library. The calculated correlation is more accurate.

[0065] Considering that the size of the architectural image library is large, the number of architectural images in the architectural image library is large, and the complexity of traversing the entire architectural image library is high, the embodiment of the present application provides a multi-modal architectural image retrieval method. After obtaining the image feature vector and the text feature vector of the architectural image, the following steps can also be performed:

[0066] The image feature vector and text feature vector corresponding to each building image in the building image library are concatenated to obtain the overall representation vector of each building image.

[0067] Specifically, after the image feature vector and text feature vector corresponding to each building image in the building image library are extracted by the CLIP model, the image feature vector and text feature vector corresponding to each building image can be spliced ​​to obtain the overall representation vector E corresponding to each building image. v Among them, E v =[I v ; T v]. The overall representation vector can be used as an index to retrieve the building image corresponding to the overall representation vector.

[0068] The image feature mean, the forward feature mean and the reverse feature mean are concatenated to obtain the query representation vector.

[0069] Specifically, after extracting the features of the building example image set, the positive indicator word set, and the reverse indicator word set through the CLIP model, the image feature mean, the positive feature mean, and the reverse feature mean can be concatenated to obtain the query representation vector E Q Among them, E Q =[αI p βT q ; -γS b ], α, β, γ are manually set hyper parameters.

[0070] The ANN indexing algorithm is used to retrieve the overall representation vector of each building image to obtain an initial building image set.

[0071] Specifically, the vector search can be accelerated by using the ANN (Approximate Nearest Neighbor) indexing algorithm.

[0072] Based on this, step S04, the process of calculating the correlation between the set of example building images and each building image in the building image library according to the image feature mean, the forward feature mean and the reverse feature mean, may include:

[0073] according to Calculate the correlation between the building example image set and each building image in the initial building image set, where Q is the query condition, v is the building image, and E Q is the query representation vector, For E v The transpose of E v is the overall representation vector of each building image.

[0074] The embodiment of the present application provides a multi-modal architectural image retrieval method that combines a calculated image v for calculating correlation and an overall query condition Q=(I p ,T q ,S b ) correlation is converted to calculate E v and E Q The inner product of the vector is calculated and the ANN index algorithm is used to accelerate the inner product-based vector retrieval process. At this time, if the number of architectural images in the architectural image library is A, the computational complexity can be reduced from O(A) to O(log A).

[0075] The following is a description of a multimodal architectural image retrieval device provided in an embodiment of the present application. The multimodal architectural image retrieval device described below and the multimodal architectural image retrieval method described above can be referred to in correspondence with each other.

[0076] First combine Figure 2 , introduces the multimodal architectural image retrieval device, such as Figure 2 As shown, the multimodal building image retrieval device may include:

[0077] A condition acquisition unit 10, used to acquire a set of example building images, a set of positive prompt words, and a set of reverse prompt words input by a user;

[0078] An image feature extraction unit 20 is used to extract features from a set of example building images by using a ViT image encoder in a CLIP model to obtain an image feature set, and calculate an image feature mean of the image feature set;

[0079] A text feature extraction unit 30 is used to extract features from the positive prompt word set and the reverse prompt word set respectively through the BERT text encoder in the CLIP model to obtain a positive feature set and a reverse feature set, and calculate a positive feature mean of the positive feature set and a reverse feature mean of the reverse feature set;

[0080] A correlation calculation unit 40, used to calculate the correlation between the building example image set and each building image in the building image library according to the image feature mean, the forward feature mean and the reverse feature mean;

[0081] A sorting unit 50, used to sort the relevance in descending order;

[0082] The image determination unit 60 is configured to use the top N building images ranked in terms of relevance as target building images.

[0083] The multimodal building image retrieval apparatus provided in the embodiment of the present application can be applied to a multimodal building image retrieval device. Figure 3 The hardware structure diagram of the multi-modal architectural image retrieval device is shown in FIG. Figure 3 , the hardware structure of the device may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4;

[0084] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;

[0085] The processor 1 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;

[0086] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0087] The memory stores a program, and the processor can call the program stored in the memory, and the program is used to implement each processing flow in the aforementioned multi-modal architectural image retrieval solution.

[0088] An embodiment of the present application further provides a storage medium, which can store a program suitable for execution by a processor, and the program is used to implement each processing flow in the aforementioned multi-modal architectural image retrieval solution.

[0089] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0090] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0091] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal architectural image retrieval method, characterized in that: include: Obtaining a set of example building images, a set of positive prompt words, and a set of reverse prompt words input by a user; The ViT image encoder in the CLIP model is used to extract features from the building sample image set to obtain an image feature set, and the image feature mean of the image feature set is calculated; The BERT text encoder in the CLIP model is used to extract features from the positive prompt word set and the reverse prompt word set respectively to obtain a positive feature set and a reverse feature set, and the positive feature mean of the positive feature set and the reverse feature mean of the reverse feature set are calculated; According to the image feature mean, the forward feature mean and the reverse feature mean, the correlation between the building example image set and each building image in the building image library is calculated; Sort the relevance from highest to lowest; The top N building images ranked by relevance are taken as target building images, where N is a positive integer; Before calculating the correlation between the building example image set and each building image in the building image library according to the image feature mean, the forward feature mean and the reverse feature mean, the following is also included: The image feature of each building image in the building image library is extracted through the ViT image encoder in the CLIP model to obtain the image feature vector of each building image; The BERT text encoder in the CLIP model is used to extract text features from each building image in the building image library to obtain the text feature vector of each building image. The process of calculating the correlation between the building example image set and each building image in the building image library according to the image feature mean, the forward feature mean and the reverse feature mean includes: according to Calculate the correlation between the set of building example images and each building image in the building image library, where Q is the query condition, v is the building image in the building image library, and I v is the image feature vector of the building image, T v is the text feature vector of the building image, and I v and T v The transpose of I p is the mean value of image features, T q is the positive feature mean, S b is the reverse feature mean, α, β, γ are manually set hyperparameters, and the query condition Q includes the image feature mean, positive feature mean, and reverse feature mean of the building example image set. Q=(I p ,T q ,S b ).

2. The multimodal architectural image retrieval method according to claim 1, characterized in that: Also includes: The image feature vector and text feature vector corresponding to each building image in the building image library are concatenated to obtain the overall representation vector of each building image.

3. The multimodal architectural image retrieval method according to claim 2, characterized in that: Also includes: The image feature mean, the forward feature mean and the reverse feature mean are concatenated to obtain the query representation vector.

4. The multimodal architectural image retrieval method according to claim 3, characterized in that: After calculating the correlation between the building example image set and each building image in the building image library according to the image feature mean, the forward feature mean and the reverse feature mean, it also includes: The ANN indexing algorithm is used to retrieve the overall representation vector of each building image to obtain an initial building image set.

5. The multimodal architectural image retrieval method according to claim 4, characterized in that: According to the image feature mean, the forward feature mean and the reverse feature mean, the correlation between the building example image set and each building image in the building image library is calculated, including: according to Calculate the correlation between the building example image set and each building image in the initial building image set, where Q is the query condition, v is the building image, and E Q is the query representation vector, E Q =[αI p βT q ; -γS b ], I p is the mean value of image features, T q is the positive feature mean, S b is the reverse feature mean, α, β, γ are manually set hyperparameters, For E v The transpose of E v is the overall representation vector of each building image.

6. A multimodal architectural image retrieval device, characterized in that: Using the multimodal architectural image retrieval method described in claim 1, the retrieval device includes: A condition acquisition unit, used to acquire a set of building example images, a set of positive prompt words, and a set of reverse prompt words input by a user; An image feature extraction unit is used to extract features from a set of example building images through a ViT image encoder in a CLIP model to obtain an image feature set, and calculate an image feature mean of the image feature set; A text feature extraction unit, used to extract features from the positive prompt word set and the reverse prompt word set respectively through the BERT text encoder in the CLIP model to obtain a positive feature set and a reverse feature set, and calculate a positive feature mean of the positive feature set and a reverse feature mean of the reverse feature set; A correlation calculation unit, used to calculate the correlation between the building example image set and each building image in the building image library according to the image feature mean, the forward feature mean and the reverse feature mean; A sorting unit, used to sort the relevance from high to low; The image determination unit is used to take the top N building images ranked in terms of relevance as target building images.

7. A multimodal architectural image retrieval device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the multimodal building image retrieval method as described in any one of claims 1 to 5.

Images