Building information retrieval method and device, electronic equipment and storage medium

Abstract

The invention provides a building information retrieval method and device, electronic equipment and a storage medium. The method comprises the following steps: establishing a training data set; wherein the training data set is a training image-training text pair set; the training image is an image in the building field, and the training text is description of multiple different languages; inputting the training image-training text pair into the initial retrieval model, and obtaining a target retrieval model when the loss of a target loss function of the similarity between the first embedded vector and the second embedded vector is minimum; wherein the first embedded vector is obtained by converting a training text, the second embedded vector is obtained by a fused image, and the fused image is obtained by fusing feature information of a training image under different scales; obtaining to-be-retrieved data; and inputting the to-be-retrieved data into the target retrieval model to obtain a retrieval result. According to the method, the detail features of the building image are more concerned, the overall features are not lost, and the retrieval performance and adaptability of the target retrieval model in the building field are improved.

Description

Technical Field

[0001] The present application relates to the field of retrieval technology, and in particular to a building information retrieval method, device, electronic device and storage medium. Background Art

[0002] Building information retrieval involves the acquisition, storage, retrieval and application of various information about buildings. With the continuous development of the construction field and the widespread application of digital technology, building information retrieval has become the key to improving building design, management and cultural heritage protection.

[0003] However, traditional architectural information retrieval methods mainly rely on text keyword matching or image feature-based retrieval, which has certain limitations when dealing with complex architectural information. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a building information retrieval method, device, electronic device and storage medium to overcome the problems in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a method for building information retrieval, the method comprising: a training phase and a use phase; The training phase specifically includes: Establishing a training data set; wherein the training data set is a set of training image-training text pairs; the training images are images in the field of architecture, and the training texts are descriptions of the images in multiple different languages; Input the training image-training text pair into the initial retrieval model, and obtain the target retrieval model when the loss of the target loss function of the similarity between the first embedding vector and the second embedding vector is minimized; wherein the first embedding vector is obtained by transforming the training text, and the second embedding vector is obtained by a fused image, and the fused image is obtained by fusing feature information of the training image at different scales; The use phase specifically includes: Acquire the data to be retrieved; wherein the data to be retrieved includes images in the field of architecture and text descriptions of the images; The data to be retrieved is input into the target retrieval model to obtain the retrieval result output by the target retrieval model.

[0006] In some technical solutions of the present application, the training data set is established as described above; wherein the training data set is a set of training image-training text pairs; the training images are images in the field of architecture, and the training texts are text descriptions of the images in multiple different languages, including: Obtaining initial text in a first language in the field of architecture from a preset database, and determining an image corresponding to the initial text in the first language; translating the initial text in the first language into the initial text in the other language; The initial characters in the first language and the initial characters in other languages ​​are used as training texts; and the initial images corresponding to the initial characters are used as training images.

[0007] In some technical solutions of the present application, the initial retrieval model includes a text encoder, a diffusion scale network and an image encoder; The method obtains the target retrieval model by: Inputting the training text into the text encoder to obtain the first embedding vector output by the text encoder; Inputting the training image into the diffusion scale network, the diffusion scale network fuses feature information of the training image at different scales to obtain a fused image; Inputting the fused image into the image encoder, the image encoder outputs the second embedding vector; A target loss function of the similarity between the first embedding vector and the second embedding vector is constructed; and a target retrieval model is obtained when the loss of the target loss function is minimum.

[0008] In some technical solutions of the present application, the training image is input into the diffusion scale network, and the diffusion scale network fuses the feature information of the training image at different scales to obtain a fused image, including: According to a preset initial scale and Gaussian kernel, Gaussian convolution is performed on the training image to obtain a first convolution image; The initial scale is adjusted according to a preset adjustment method, and Gaussian convolution is performed on the training images based on the adjusted scale to obtain a second convolution image; Perform feature fusion on each pixel in the training image, the first convolution image, and the second convolution image to obtain a fused image.

[0009] In some technical solutions of the present application, the above-mentioned feature fusion of each pixel in the training image, the first convolution image and the second convolution image to obtain a fused image includes: Determine a first eigenvector, a second eigenvector, and a third eigenvector of any target pixel in the training image, the first convolution image, and the second convolution image, respectively; Taking the mean of the first eigenvector, the second eigenvector and the third eigenvector as a fused eigenvector; The fused image is obtained according to the fused feature vector.

[0010] In some technical solutions of the present application, the above-mentioned image encoder includes different stages and different stages use residual structures of different sizes; The step of inputting the fused image into the image encoder and outputting the second embedding vector from the image encoder comprises: At different stages, different levels of feature extraction are performed to obtain preliminary features at multiple levels; The second embedding vector is obtained by normalizing the preliminary features.

[0011] In some technical solutions of the present application, the above-mentioned constructing a target loss function of the similarity between the first embedding vector and the second embedding vector; when the loss of the target loss function is minimum, obtaining a target retrieval model includes: constructing a first loss function according to a first similarity between any of the first embedding vectors and each of the second embedding vectors; constructing a second loss function according to a second similarity between the second embedding vector and the corresponding first embedding vector; The target loss function is constructed according to the first loss function and the second loss function; when the loss of the target loss function is minimum, a target retrieval model is obtained.

[0012] In a second aspect, an embodiment of the present application provides a retrieval device, the device comprising: a training module and a use module; The training module is used to establish a training data set; wherein the training data set is a set of training image-training text pairs; the training images are images in the field of architecture, and the training texts are descriptions of the images in multiple different languages; Input the training image-training text pair into the initial retrieval model, and obtain the target retrieval model when the loss of the target loss function of the similarity between the first embedding vector and the second embedding vector is minimized; wherein the first embedding vector is obtained by transforming the training text, and the second embedding vector is obtained by a fused image, and the fused image is obtained by fusing feature information of the training image at different scales; The use module is used to obtain the data to be retrieved; wherein the data to be retrieved includes images in the field of architecture and text descriptions of the images; The data to be retrieved is input into the target retrieval model to obtain the retrieval result output by the target retrieval model.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned building information retrieval method when executing the computer program.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned building information retrieval method are executed.

[0015] The technical solution provided by the embodiments of the present application may have the following beneficial effects: The method of the present application is included in the training stage, specifically including: Establishing a training data set; wherein the training data set is a set of training image-training text pairs; the training images are images in the field of architecture, and the training texts are descriptions of the images in multiple different languages; Input the training image-training text pair into the initial retrieval model, and obtain the target retrieval model when the loss of the target loss function of the similarity between the first embedding vector and the second embedding vector is minimized; wherein the first embedding vector is obtained by transforming the training text, and the second embedding vector is obtained by a fused image, and the fused image is obtained by fusing feature information of the training image at different scales; The use phase specifically includes: Acquire the data to be retrieved; wherein the data to be retrieved includes images in the field of architecture and text descriptions of the images; The data to be retrieved is input into the target retrieval model to obtain the retrieval result output by the target retrieval model.

[0016] This application constructs a target retrieval model applied to the field of architecture. The model pays more attention to the detailed features of building images without losing the overall features, thereby improving the retrieval performance and adaptability of the target retrieval model in the field of architecture.

[0017] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A schematic diagram of a flow chart of a building information retrieval method provided in an embodiment of the present application is shown; Figure 2 An architectural word cloud diagram provided by an embodiment of the present application is shown; Figure 3a A text similarity heat map provided by an embodiment of the present application is shown; Figure 3b Another text similarity heat map provided by an embodiment of the present application is shown; Figure 4 A schematic diagram of a text encoder provided in an embodiment of the present application is shown; Figure 5 A schematic diagram of image fusion provided by an embodiment of the present application is shown; Figure 6 A schematic diagram of an image encoder provided in an embodiment of the present application is shown; Figure 7 A schematic diagram of a retrieval device provided in an embodiment of the present application is shown; Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0021] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0022] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0023] Building information retrieval involves the acquisition, storage, retrieval and application of various information about buildings. With the continuous development of the construction field and the widespread application of digital technology, building information retrieval has become the key to improving building design, management and cultural heritage protection. However, traditional building information retrieval methods mainly rely on text keyword matching or image feature-based retrieval, which has certain limitations when dealing with complex building information.

[0024] For example, text-based retrieval may not accurately express the visual features of buildings, while image-based retrieval cannot understand the semantic information of buildings. In addition, due to the diversity and complexity of architectural information, traditional architectural information retrieval methods not only fail to meet users' needs for diversified, cross-modal and cross-language architectural information, but also limit the digital development of the architectural field.

[0025] Although success has been achieved in the fields of computer vision and natural language processing, there are still many challenges in exploring cross-language and cross-modal retrieval models in the field of architecture: Dataset collection: Especially when it comes to datasets in professional fields, due to the complexity and diversity of architectural data, collecting representative multimodal datasets is a challenging task.

[0026] Extension of retrieval models: The extension of cross-language and cross-modal retrieval technology to the field of architecture requires consideration of how to effectively extend the existing model to adapt to the characteristics and needs of architectural data. This involves the adjustment and optimization of the model architecture, as well as model training and adjustment for architectural information retrieval tasks.

[0027] Extraction of architectural image features: Architectural images usually contain rich structures, including features such as the appearance, facades, and lines of the building. How to better extract and represent these features from architectural images is a key challenge. It is necessary to explore effective feature extraction methods and model design to ensure that the semantic and visual features of architectural images can be accurately captured, thereby supporting cross-language and cross-modal architectural information retrieval tasks.

[0028] Based on this, the embodiments of the present application provide a building information retrieval method, device, electronic device and storage medium, which are described below through embodiments.

[0029] Figure 1 A flow chart of a building information retrieval method provided by an embodiment of the present application is shown, wherein the method comprises steps S101 a training phase and S102 a use phase; specifically: The training phase specifically includes: Establishing a training data set; wherein the training data set is a set of training image-training text pairs; the training images are images in the field of architecture, and the training texts are descriptions of the images in multiple different languages; Input the training image-training text pair into the initial retrieval model, and obtain the target retrieval model when the loss of the target loss function of the similarity between the first embedding vector and the second embedding vector is minimized; wherein the first embedding vector is obtained by transforming the training text, and the second embedding vector is obtained by a fused image, and the fused image is obtained by fusing feature information of the training image at different scales; The use phase specifically includes: Acquire the data to be retrieved; wherein the data to be retrieved includes images in the field of architecture and text descriptions of the images; The data to be retrieved is input into the target retrieval model to obtain the retrieval result output by the target retrieval model.

[0030] This application constructs a target retrieval model applied to the field of architecture. The model pays more attention to the detailed features of building images without losing the overall features, thereby improving the retrieval performance and adaptability of the target retrieval model in the field of architecture.

[0031] Some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0032] In the training phase, a training data set is first established, where the training data set is a set of training image-training text pairs; the training images are images in the field of architecture, and the training texts are descriptions of the images in multiple different languages. Images in the field of architecture usually have rich details and complex structures, and traditional diffusion models may not be able to effectively understand these details and structures, and are more likely to destroy these details and structures during the generation process. In order to understand the details of images in the field of architecture, the embodiment of the present application performs comprehensive detail feature extraction during the training process.

[0033] When establishing a training image-training text pair, the embodiment of the present application first obtains the initial text of the first language and determines the image corresponding to the initial text of the first language; translates the initial text of the first language into the initial text of other languages; uses the initial text of the first language and the initial text of other languages ​​as training text; and uses the initial image corresponding to the initial text as the training image.

[0034] In the specific implementation, the training dataset is the CLBS dataset, where the first language can be English and the second language can be Chinese. For example, the dataset comes from Flickr30K and SUN database. The images in the CLBS dataset and their corresponding text descriptions mainly include street scenes, natural landscapes, and residential buildings. First, the texts in Flickr30K and SUNdatabase are processed respectively to extract and generate texts related to buildings. Then, the corresponding images are extracted according to the text identifiers. Finally, the text is translated into Chinese to construct a cross-language image-text retrieval dataset for architectural scenes.

[0035] In Flickr30K, each image is associated with five text descriptions. A data-driven approach based on statistics and analysis was used to identify key terms related to architecture. By collecting text data of architecture and applying word cloud technology, a word frequency analysis was performed on the collected text. By performing statistical analysis on the word frequency of each word, a set of high-frequency keywords was determined and used as the benchmark words for architecture. Figure 2 The word cloud of architectural terms is displayed, and we can see that multiple architectural-related terms such as 'building', 'construction', 'architecture', 'engineer', 'design', 'structure', and 'urban' are formed. The similarity between the open dataset and these specified terms is calculated using the spaCy-based text similarity calculation method. The formula is as follows:

[0036] a and b represent architecture-related terms and text descriptions from Flickr30k, respectively. Through the encoder layer, the text is mapped to a vector space and cosine similarity is calculated as the similarity score between the two vectors. When the similarity comparison is performed on words and sentences, a similarity score greater than 0.6 is considered a sign of architectural text. Then, based on the identified text number, the image corresponding to the extracted text is retrieved.

[0037] The SUN database mainly focuses on scene classification. Each image represents a specific building structure and is accompanied by a corresponding text description. For this dataset, all image-text pairs with long text descriptions are processed and extracted. However, for each text, four similar sentences are generated by Generative Pre-trained Transformer (GPT) and aligned with Flickr30K. Specifically, given a text sequence , GPT aims to maximize the conditional probability of each word in the sequence:

[0038] in, is the next word in the sequence, Represents model parameters. The GPT model learns parameters by optimizing the following objective function , that is, maximize the log-likelihood of the entire data set:

[0039] Use the GPT model to extract the given building description text Generate similar text and find words Make the probability To reach the maximum, gradually build a new text sequence by generating words one by one The generation process can be formalized as:

[0040] in, Is the model based on the original text The new text sequence generated.

[0041] In order to verify the accuracy of the generated text, several groups of generated texts are randomly selected to compare with the texts in Flicker30K. The similarity is calculated by Jensen-Shannon Divergence:

[0042]

[0043] For each text, we use TF-IDF to count the number of occurrences of each word and convert it into a vector. Each dimension of the vector corresponds to a unique word and is represented as a probability distribution P and Q. represents the probability of the average distribution M on event i. The text ID 0 represents the original text, and 1 to 4 represent the generated text. Figure 3a and Figure 3b As shown, it can be seen that the similarity of the generated text (SUN database) is generally higher than that of the real text (Flicker30K).

[0044] After completing the above tasks, a Chinese dataset was established. The images remained unchanged, and the text was translated using Helsinki-NLP. Helsinki-NLP is a Transformer-based pre-trained language model specifically designed to handle sequence-to-sequence tasks. The model uses a self-attention mechanism and multi-layer encoders and decoders to learn the mapping relationship between the input English text sequence and the corresponding Chinese text sequence. The sentences translated into Chinese need to be manually checked, modified or deleted for incorrect words.

[0045]

[0046] x represents the input text, h is the result of x being encoded by the transformer, is the result after decoding, that is, the result after translation. The final CLBS dataset sample is as follows Figure 5 shown.

[0047] After obtaining the training data set in the above manner, the training data set is used for model training: the training image-training text pair is input into the initial retrieval model, and the target retrieval model is obtained when the loss of the target loss function of the similarity between the first embedding vector and the second embedding vector is minimized; wherein the first embedding vector is obtained by transforming the training text, and the second embedding vector is obtained by fusion image, and the fusion image is obtained by fusing the feature information of the training image at different scales. In specific implementation, the initial retrieval model here can use CLIP.

[0048] The initial retrieval model in the embodiment of the present application includes a text encoder, a diffusion scale network and an image encoder. The training process of the initial retrieval model is: input the training text into the text encoder to obtain the first embedding vector output by the text encoder; input the training image into the diffusion scale network, the diffusion scale network fuses the feature information of the training image at different scales to obtain a fused image; input the fused image into the image encoder, the image encoder outputs the second embedding vector; construct a target loss function of the similarity between the first embedding vector and the second embedding vector; when the loss of the target loss function is the smallest, obtain the target retrieval model.

[0049] In the specific implementation, the text encoder in the embodiment of the present application adopts a text encoder based on BERT, that is, a bidirectional Transformer encoder. Figure 4 As shown, specifically, considering the difficulty of extracting cross-language text features, the embodiment of the present application adopts Bert as the text encoding layer of CLIP.

[0050] Formally, given a pair of tokens and .in, Corresponding to each word or Chinese character in the first half of the input text, Corresponding to each word or Chinese character in the second half of the input text. Construct the input sequence by connecting two sequences , and As two special tokens, they respectively split the input sequence and aggregate the representation information of the entire sequence. The representation corresponding to each token is actually composed of three parts: Token embeddings: Convert each word into a vector of fixed dimension. In CLIP, each word is converted into a 512-dimensional vector representation.

[0051] Segment embeddings: used to distinguish which sentence a token belongs to in a sentence pair. The first vector assigns 0 to each token in the first sentence, and the second vector assigns 1 to each token in the second sentence. If there is only one sentence in the input, the value of segment embedding is all 0.

[0052] Position embeddings: Since Transformers cannot encode the order of the input sequence, a vector representation must be learned at each position to encode the information of the sequence order. Specifically, if the same token appears in a sentence, the two token vector representations are different due to the different input order.

[0053] Then, X is transformed into a contextualized representation T(L)∈RN×d through a continuous Transformer, where N is the maximum sequence length and d is the dimension of the hidden layer.

[0054]

[0055] The training text is input into the text encoder, which encodes it into a first embedding vector of fixed length for comparison with image features.

[0056] The diffusion scale network in the embodiment of the present application is proposed based on the diffusion network, which includes a forward diffusion stage and a reverse diffusion stage. In the forward diffusion stage, the input data is gradually disturbed in several steps by adding Gaussian noise until the image becomes full of noise. In the reverse diffusion stage, the task of the model is to restore the original input data by gradually learning the reverse diffusion process.

[0057] Based on the concept of forward diffusion, a new diffusion scale network (DSN) is proposed and applied to architectural image processing. DSN can not only enhance the model's ability to process the relationship between details and structures in architectural images, but also enhance the model's anti-interference ability, thereby improving the robustness of the model. Since Gaussian noise needs to be added to the input image during the forward processing, DSN needs to specify an initial scale , and define a Gaussian kernel, for example, determine the size of the Gaussian convolution kernel to be 3×3:

[0058] x and y are the coordinates in the Gaussian kernel, is the initial scale; For the input image I, according to the initial scale Perform Gaussian convolution and calculate the mean of the convolved image and variance :

[0059] Where N is the total number of image pixels, Represents the pixel value of the convolution image i; Calculate scale based on variance Standard Deviation , and convolve the image Perform mean normalization:

[0060] Adjust the mean of the image to 0 and the standard deviation to 1; 0 represents zero mean and 1 represents unit standard deviation. Zero mean: Setting the mean of the image to 0 ensures that changes in the value of the center pixel during the convolution process are primarily determined by the weights of the neighboring pixels, rather than by the brightness of the entire image. This helps reduce bias and over-enhancement during image processing. Unit standard deviation: By setting the image standard deviation to 1, you ensure that the distribution of pixel values ​​in the image is more stable. This allows the convolution kernel to be applied more consistently across different image regions, helping to improve the stability and repeatability of the convolution operation.

[0061] Finally, by changing , get standardized images of different scales, and fuse the standardized images with the original images. Figure 5 As shown. The feature fusion layer uses the mean fusion method to integrate the feature information of the same image processed at different scales. Specifically, for each pixel of the image, in the feature vector obtained after processing at all scales, this scheme calculates the mean of each feature dimension to obtain the final fused feature representation and obtain the final output image H:

[0062] Where N represents the total number of all images including the original image.

[0063] The purpose of this is to extract local features of different sizes from the building image. Finally, the processed image is subjected to feature extraction and fusion to achieve balanced processing of the details and structure of the building image. It can achieve better results in preserving image details; using larger It can achieve better results in overall image fusion. You can start from 0.5 and take 0.5 as the step length until 10. Since in the experimental verification, when When the value is adjusted to greater than 10, the image does not change significantly. The maximum value is set to 10. By cleverly fusing Gaussian convolution kernels of different scales, DSN comprehensively captures the multi-scale information of the image, achieves balanced processing of local details and overall structure, and has good spatial adaptability, flexibility and noise resistance.

[0064] In the embodiment of the present application, ResNet50 is used as the image coding layer of CLIP. CLIP is a deep neural network widely used for image feature extraction. ResNet50 consists of multiple residual blocks, such as Figure 6 Stage 1-4 shown. The introduction of residual connections solves the problem of gradient disappearance and explosion in deep neural networks.

[0065] Specifically, in stage 0, global maximum pooling is used to perform preliminary feature extraction and size reduction on the input image:

[0066]

[0067] Among them, H is the output of DSN and serves as the input of Image encoder, and K is the convolution kernel.

[0068] In Stage 1-3, preliminary features at low, medium and high levels are extracted in turn. Stage 4 is used to further extract the highest level features of the image, followed by a normalized exponential function to obtain the final image feature vector.

[0069] The functions of Stage 1-4 are different because they use residual structures of different sizes. In a normal neural network, a shortcut branch structure is added, as shown by the dotted line in the figure, so that the entire network structure forms a residual structure. In form:

[0070] Where L represents the Stage number. represents the nonlinear transformation inside the residual module, Indicates the weight parameter inside the Stage.

[0071] Finally, the SoftMax function is used to convert Convert to a vector of the same size as the text encoding:

[0072] Here W and b are the weight matrix and bias of the softmax function.

[0073] After obtaining the first embedding vector output from the text encoder and the second embedding vector output from the image encoder, training is performed by contrastive learning: for a given batch of training image-text pairs, predict the full combination The correlation of image-text pairs is maximized using contrastive learning. The cosine similarity of the embedding vectors of the image-text pairs is minimized to minimize the other incorrect embedding vectors in this batch. The cosine similarity of the embedding vectors of the image-text pairs.

[0074] Specifically, for a text , calculate its correlation with each image Probability of pairing:

[0075] Among them, sim represents cosine similarity calculation.

[0076] Then, for each text , take the image with the same sequence number as The probability of pairing , we get the following loss function (constructing the first loss function):

[0077] Similarly, for an image , calculate its difference with each text Probability of pairing:

[0078] Then, for each image , take the text with the same sequence number The probability of pairing , we get the following loss function (constructing the second loss function):

[0079] Finally, the above two loss functions are averaged to obtain the final target loss function, minimizing the loss function ,Right now:

[0080] The CLIP model encodes images and texts in the embedding space using contrastive learning, making related images and texts closer in this space and unrelated images and texts farther apart. This training method enables CLIP to learn to understand the semantic relationships between images and texts.

[0081] In an alternative embodiment, during the usage phase, retrieve data to be retrieved; wherein, the data to be retrieved includes images in the construction field and text descriptions of the images; input the data to be retrieved into the target retrieval model to obtain the retrieval result output by the target retrieval model. That is, input the image to be retrieved or the text to be retrieved into the trained target retrieval model to obtain the query text corresponding to the image to be retrieved, or obtain the query image corresponding to the text to be retrieved. The processing process during usage corresponds to the training process and will not be elaborated here.

[0082] In an alternative embodiment, the retrieval result using the English query “Houses built on the beach”. The retrieval results of the target retrieval model consistently describe houses by the sea, while the existing model retrieved some houses by the sea and some scenic houses. In contrast, the retrieval results of the target retrieval model are more accurate.

[0083] The retrieval result using the English query “Boat sailing on the river”. The retrieval of the existing model focuses more on the river, while the retrieval results of the target retrieval model include the boat and the river, capturing these two basic elements.

[0084] The retrieval result using the Chinese query “基督教修道院或修道院” (Christian monastery or convent). The existing model tends to take similar buildings as the correct result, while the retrieval results of the target retrieval model are more accurate.

[0085] The retrieval result using the Chinese query “建筑施工现场” (construction site). The existing model tends to classify complex buildings as construction sites, while the retrieval results of the target retrieval model capture some important features unique to construction sites.

[0086] Compare and analyze the image retrieval text results, the search for retrieving English text a using image A. Both models can accurately retrieve the architectural scenes depicted in the image. However, the retrieval results of the target retrieval model show more detailed features, including the people in the corner of image A.

[0087] The search for retrieving Chinese text a’ using image A. Both models will retrieve the text describing the information in the image. However, the retrieval results of the target retrieval model maintain the overall feature description, focusing on the entire mountain range, while some of the retrieval results of the existing model tend to other features in the image.

[0088] Figure 7 A schematic diagram of the structure of a search device provided in an embodiment of the present application is shown, wherein the device comprises: Training modules and usage modules; The training module is used to establish a training data set; wherein the training data set is a set of training image-training text pairs; the training images are images in the field of architecture, and the training texts are descriptions of the images in multiple different languages; Input the training image-training text pair into the initial retrieval model, and obtain the target retrieval model when the loss of the target loss function of the similarity between the first embedding vector and the second embedding vector is minimized; wherein the first embedding vector is obtained by transforming the training text, and the second embedding vector is obtained by a fused image, and the fused image is obtained by fusing feature information of the training image at different scales; The use module is used to obtain the data to be retrieved; wherein the data to be retrieved includes images in the field of architecture and text descriptions of the images; The data to be retrieved is input into the target retrieval model to obtain the retrieval result output by the target retrieval model.

[0089] The training data set is established; wherein the training data set is a set of training image-training text pairs; the training images are images in the field of architecture, and the training texts are text descriptions of the images in multiple different languages, including: Obtaining initial text in a first language in the field of architecture from a preset database, and determining an image corresponding to the initial text in the first language; translating the initial text in the first language into the initial text in the other language; The initial characters in the first language and the initial characters in other languages ​​are used as training texts; and the initial images corresponding to the initial characters are used as training images.

[0090] The initial retrieval model includes a text encoder, a diffusion scale network and an image encoder; The target retrieval model is obtained by: Inputting the training text into the text encoder to obtain the first embedding vector output by the text encoder; Inputting the training image into the diffusion scale network, the diffusion scale network fuses feature information of the training image at different scales to obtain a fused image; Inputting the fused image into the image encoder, the image encoder outputs the second embedding vector; A target loss function of the similarity between the first embedding vector and the second embedding vector is constructed; and a target retrieval model is obtained when the loss of the target loss function is minimum.

[0091] The step of inputting the training image into the diffusion scale network, wherein the diffusion scale network fuses feature information of the training image at different scales to obtain a fused image, comprises: According to a preset initial scale and Gaussian kernel, Gaussian convolution is performed on the training image to obtain a first convolution image; The initial scale is adjusted according to a preset adjustment method, and Gaussian convolution is performed on the training images based on the adjusted scale to obtain a second convolution image; Perform feature fusion on each pixel in the training image, the first convolution image, and the second convolution image to obtain a fused image.

[0092] The step of fusing features of pixels in the training image, the first convolution image, and the second convolution image to obtain a fused image includes: Determine a first eigenvector, a second eigenvector, and a third eigenvector of any target pixel in the training image, the first convolution image, and the second convolution image, respectively; Taking the mean of the first eigenvector, the second eigenvector and the third eigenvector as a fused eigenvector; The fused image is obtained according to the fused feature vector.

[0093] The image encoder includes different stages and different stages use residual structures of different sizes; The step of inputting the fused image into the image encoder and outputting the second embedding vector from the image encoder comprises: At different stages, different levels of feature extraction are performed to obtain preliminary features at multiple levels; The second embedding vector is obtained by normalizing the preliminary features.

[0094] The step of constructing a target loss function for the similarity between the first embedding vector and the second embedding vector; and obtaining a target retrieval model when the loss of the target loss function is minimum, comprises: constructing a first loss function according to a first similarity between any of the first embedding vectors and each of the second embedding vectors; constructing a second loss function according to a second similarity between the second embedding vector and the corresponding first embedding vector; The target loss function is constructed according to the first loss function and the second loss function; when the loss of the target loss function is minimum, a target retrieval model is obtained.

[0095] like Figure 8 As shown, an embodiment of the present application provides an electronic device for executing the building information retrieval method in the present application, the device includes a memory, a processor, a bus, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the building information retrieval method when executing the computer program.

[0096] Specifically, the above-mentioned memory and processor may be general-purpose memory and processor, which are not specifically limited here. When the processor runs the computer program stored in the memory, the above-mentioned building information retrieval method can be executed.

[0097] Corresponding to the building information retrieval method in the present application, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned building information retrieval method are executed.

[0098] Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is run, the above-mentioned building information retrieval method can be executed.

[0099] In the embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. The system embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0100] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0101] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0102] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0103] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0104] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A building information retrieval method, characterized in that: The method comprises a training phase and a use phase; The training phase specifically includes: Establishing a training data set; wherein the training data set is a set of training image-training text pairs; the training images are images in the field of architecture, and the training texts are descriptions of the images in multiple different languages; Input the training image-training text pair into the initial retrieval model, and obtain the target retrieval model when the loss of the target loss function of the similarity between the first embedding vector and the second embedding vector is minimized; wherein the first embedding vector is obtained by transforming the training text, and the second embedding vector is obtained by a fused image, and the fused image is obtained by fusing feature information of the training image at different scales; The use phase specifically includes: Acquire the data to be retrieved; wherein the data to be retrieved includes images in the field of architecture and text descriptions of the images; The data to be retrieved is input into the target retrieval model to obtain the retrieval result output by the target retrieval model.

2. The method according to claim 1, characterized in that: The training data set is established; wherein the training data set is a set of training image-training text pairs; the training images are images in the field of architecture, and the training texts are text descriptions of the images in multiple different languages, including: Obtaining initial text in a first language in the field of architecture from a preset database, and determining an image corresponding to the initial text in the first language; translating the initial text in the first language into the initial text in the other language; The initial characters in the first language and the initial characters in other languages ​​are used as training texts; and the initial images corresponding to the initial characters are used as training images.

3. The method according to claim 1, characterized in that The initial retrieval model includes a text encoder, a diffusion scale network and an image encoder; The method obtains the target retrieval model in the following manner: Inputting the training text into the text encoder to obtain the first embedding vector output by the text encoder; Inputting the training image into the diffusion scale network, the diffusion scale network fuses feature information of the training image at different scales to obtain a fused image; Inputting the fused image into the image encoder, the image encoder outputs the second embedding vector; A target loss function of the similarity between the first embedding vector and the second embedding vector is constructed; and a target retrieval model is obtained when the loss of the target loss function is minimum.

4. The method according to claim 3, characterized in that The step of inputting the training image into the diffusion scale network, wherein the diffusion scale network fuses feature information of the training image at different scales to obtain a fused image, comprises: According to a preset initial scale and Gaussian kernel, Gaussian convolution is performed on the training image to obtain a first convolution image; The initial scale is adjusted according to a preset adjustment method, and Gaussian convolution is performed on the training images based on the adjusted scale to obtain a second convolution image; Perform feature fusion on each pixel in the training image, the first convolution image, and the second convolution image to obtain a fused image.

5. The method according to claim 4, characterized in that The step of fusing features of pixels in the training image, the first convolution image, and the second convolution image to obtain a fused image includes: Determine a first eigenvector, a second eigenvector, and a third eigenvector of any target pixel in the training image, the first convolution image, and the second convolution image, respectively; Taking the mean of the first eigenvector, the second eigenvector and the third eigenvector as a fused eigenvector; The fused image is obtained according to the fused feature vector.

6. The method according to claim 3, characterized in that The image encoder includes different stages and different stages use residual structures of different sizes; The step of inputting the fused image into the image encoder and outputting the second embedding vector from the image encoder comprises: At different stages, different levels of feature extraction are performed to obtain preliminary features at multiple levels; The second embedding vector is obtained by normalizing the preliminary features.

7. The method according to claim 1, characterized in that The step of constructing a target loss function for the similarity between the first embedding vector and the second embedding vector; and obtaining a target retrieval model when the loss of the target loss function is minimum, comprises: constructing a first loss function according to a first similarity between any of the first embedding vectors and each of the second embedding vectors; constructing a second loss function according to a second similarity between the second embedding vector and the corresponding first embedding vector; The target loss function is constructed according to the first loss function and the second loss function; when the loss of the target loss function is minimum, a target retrieval model is obtained.

8. A search device, characterized in that: The device comprises: a training module and a use module; The training module is used to establish a training data set; wherein the training data set is a set of training image-training text pairs; the training images are images in the field of architecture, and the training texts are descriptions of the images in multiple different languages; Input the training image-training text pair into the initial retrieval model, and obtain the target retrieval model when the loss of the target loss function of the similarity between the first embedding vector and the second embedding vector is minimized; wherein the first embedding vector is obtained by transforming the training text, and the second embedding vector is obtained by a fused image, and the fused image is obtained by fusing feature information of the training image at different scales; The use module is used to obtain the data to be retrieved; wherein the data to be retrieved includes images in the field of architecture and text descriptions of the images; The data to be retrieved is input into the target retrieval model to obtain the retrieval result output by the target retrieval model.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the building information retrieval method as described in any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the building information retrieval method as claimed in any one of claims 1 to 7 are executed.

Images