Dual retrieval enhanced generation method and system for single building level building material identification

Abstract

This invention relates to the field of remote sensing image building material identification technology, specifically providing a dual-retrieval enhanced generation method and system for single-building material identification. The method includes the following steps: acquiring multi-source data of the target building and preprocessing it; constructing a multimodal knowledge vector library: based on the multi-source data, performing feature extraction and alignment to construct a visual vector index library and a text vector index library; performing dual-channel retrieval: visual channel retrieval and text channel retrieval; fusing evidence and resolving conflicts between the visual channel retrieval results and the text channel retrieval results to obtain structured prompt information; inputting the remote sensing image of the target building and the structured prompt information into a multimodal large model to generate a single-building material description. The method provided by this invention can improve the accuracy, reliability, and practicality of single-building material identification and description.

Description

Technical Field

[0001] This invention relates to the field of building material identification technology in remote sensing images, and in particular to a dual-retrieval enhancement generation method and system for identifying the material of a single building. Background Technology

[0002] Traditional techniques for remote sensing image analysis and building recognition include: (1) Rule-based and traditional machine learning methods; early remote sensing image analysis relied heavily on spectral thresholds, texture features, shape and geometric rules (such as NDVI, texture statistics, shape index, etc.) for classification and segmentation. Its advantages are simple implementation and strong interpretability; its disadvantages are that feature design depends on experience, has poor generalization ability, and has limited ability to distinguish complex urban scenes and fine-grained materials. (2) Building classification / segmentation methods based on deep learning; with the application of convolutional neural networks (CNN), Transformer, etc., a large number of models for building detection, building segmentation, and land cover classification have emerged (such as those based on U-Net, FPN, DeepLab, etc.). Its advantages are significant effects in target detection, segmentation, and scene classification; its limitations are that it usually only outputs "building / non-building" or a few coarse categories, lacks fine-grained distinction of material types (such as metal roofs, tile roofs, concrete exterior walls, etc.), and does not have the ability to describe in natural language.

[0003] With the development of Visual Language Models (VLMs) such as CLIP, BLIP, and GPT-4V, the remote sensing field has begun to explore the use of VLMs for image captioning, question answering, and scene description. Existing remote sensing VLMs are characterized by: an emphasis on scene-level or region-level descriptions, such as "city blocks" or "intersections of rivers and roads"; a focus on macroscopic semantics with insufficient exploration of fine-grained attributes such as building materials, age, and function; and reliance on visual information and the "implicit knowledge" of internal model parameters, with almost no explicit access to external text such as building standards, material specifications, and urban knowledge. Therefore, building material identification and description based on existing remote sensing VLMs have significant limitations.

[0004] Retrieval Enhanced Generation (RAG) methods in recent years have introduced external knowledge into large models by retrieving relevant documents before generation, thereby improving factual consistency and reducing "illusions". However, in remote sensing and building material scenarios, existing solutions generally have the following problems: (1) Most are single-channel retrieval: either only the text library (such as regulations, encyclopedia) is searched, lacking the use of similar image instances; or only similar image retrieval is performed in the image feature space, lacking professional text constraints and standard terminology. (2) The retrieval objects are mostly general knowledge or Internet documents, rather than localized multimodal knowledge bases combined with specific cities and specific building instances. (3) The application of RAG in the field of remote sensing is mostly scene question answering or geographic semantic interpretation, and a systematic solution for single-building material identification has not yet been formed.

[0005] In complex urban remote sensing scenarios, existing methods are unable to accurately identify and describe building materials at the level of a single building. They cannot effectively utilize the fine-grained attributes of buildings (function, age, structural type, etc.) and industry standard text knowledge, resulting in: (1) insufficient material identification accuracy, especially in environments with shadows, reflections, occlusions, and mixed materials, which are prone to misjudgment; (2) the generated text descriptions lack professional terminology constraints and have an "illusion" phenomenon (fictitious materials or functions); (3) a lack of multimodal material data support covering the entire city and accurate to a single building.

[0006] Therefore, it is necessary to provide a dual-retrieval enhanced generation method and system for identifying the material of a single building to solve the above-mentioned technical problems. Summary of the Invention

[0007] The purpose of this invention is to provide a dual-retrieval enhanced generation method and system for identifying the material of a single building. The specific technical solution is as follows:

[0008] A dual-retrieval enhanced generation method for material identification of single-building structures includes the following steps:

[0009] S1. Acquire multi-source data of the target building and perform preprocessing; multi-source data includes high-resolution remote sensing images, street view images, fine-grained building attribute data, open street maps and point of interest data, standard text and building attribute association knowledge;

[0010] S2. Construct a multimodal knowledge vector library: Based on the multi-source data, perform feature extraction and alignment to construct a visual vector index library and a text vector index library;

[0011] S3. Perform dual-channel search:

[0012] S3.1 Visual Channel Retrieval: Input a remote sensing image containing the target building, retrieve image features from the visual vector index library, and output a visual query vector;

[0013] Text channel retrieval: Input text information, retrieve the semantic description of the building's material association from the text vector index library, and generate semantic knowledge prompts;

[0014] S3.2 Perform evidence fusion and conflict resolution on the visual channel retrieval results and text channel retrieval results to obtain structured prompt information;

[0015] S4. Input the remote sensing image and structured prompts of the target building into the multimodal large model to generate a material description of the single building.

[0016] Furthermore, in S1, the preprocessing of multi-source data includes:

[0017] Image preprocessing: Cropping and normalization of remote sensing images and street view images;

[0018] Text preprocessing: noise removal, text cleaning, and format normalization;

[0019] Spatial matching and geocoding: Geocoding is used to spatially align building vector data with remote sensing images and street view images, so that each record corresponds to a single building.

[0020] Furthermore, in S2, feature extraction and alignment include:

[0021] Visual feature extraction: Multi-scale visual features are extracted from remote sensing images and street view images using a pre-trained visual model;

[0022] Text feature extraction: The pre-trained language model is used to vectorize and embed the standard text and the knowledge of the association between building attributes to obtain text features;

[0023] Feature alignment: Align visual features with building attributes, using the building's individual location code as the alignment element; align text features with building attributes.

[0024] Furthermore, in S2, the aligned visual features and text features are mapped to a unified vector space to construct a visual vector index library and a text vector index library. In the index library, a multimodal entry is created for each building instance to achieve multimodal joint representation.

[0025] Furthermore, in S3.2, evidence fusion includes fusing the visual features, textual features, visual query vectors, and semantic knowledge cues of the target building into a unified context vector.

[0026] Furthermore, in S3.2, conflict resolution includes:

[0027] The visual channel retrieval results and text channel retrieval results are subjected to threshold filtering and similarity reordering. If there is a conflict between the two channels, a decision is made through preset rules or conflict resolution units. The final retained result is used as structured prompt information, and the reference information is retained to support result traceability.

[0028] Furthermore, the multimodal large model is any one of GLM-4-5V, GLM-4V-flash, GLM-4V-plus, Gemini-3-Pro-Vision, Doubao-seed-1-6-vision, Doubao-seed-1-6-lite, gpt-4o-mini, gpt-5-mini, and Qwen2.5-VL-7B-Instruct;

[0029] The building material description should include at least the roof material and the exterior wall material, as well as the material category label and the material-related attribute label.

[0030] A dual-retrieval enhanced generation system for material identification of single-building structures is used to implement the method described above, including: a multi-source data acquisition and preprocessing module, a multimodal knowledge vector base construction module, a dual-channel retrieval module, and a generation and response module;

[0031] The multi-source data acquisition and preprocessing module is used to acquire and process high-resolution remote sensing images, street view images, fine-grained building attribute data, open street maps and point of interest vector data, standard text and building attribute association knowledge, and align the preprocessed data to a single building.

[0032] The multimodal knowledge vector library construction module is used to extract and align features from the processed multi-source data, and to build a visual vector index library and a text vector index library.

[0033] The dual-channel retrieval module is used to implement dual-channel retrieval and output structured prompt information;

[0034] The generation and response module is used to generate building material descriptions.

[0035] Furthermore, the dual-channel retrieval module includes a visual channel retrieval unit, a text channel retrieval unit, and an evidence fusion and conflict resolution unit;

[0036] The visual channel retrieval unit is used to perform visual channel retrieval and output a visual query vector.

[0037] The text channel retrieval unit is used to perform text channel retrieval and generate semantic knowledge prompts;

[0038] The evidence fusion and conflict resolution unit is used to fuse the dual-channel retrieval results and resolve conflicts to generate structured prompt information.

[0039] Furthermore, the multi-source data acquisition and preprocessing module also integrates a spatial matching and geocoding unit to ensure spatial alignment between remote sensing images, street view images, building vector data, and building instances.

[0040] The application of the technical solution of the present invention has at least the following beneficial effects:

[0041] Compared with existing remote sensing visual language models, single-channel retrieval-enhanced generation, or traditional deep learning material classification methods, this invention achieves significant technical advancements in the accuracy, reliability, and practicality of material identification and description for single-building structures by introducing a dual-retrieval-enhanced generation framework. Specifically, the technical effects are as follows: First, this invention demonstrates high accuracy in identifying major building materials such as metal and cement. Compared to the unenhanced basic visual language model, the accuracy is significantly improved, effectively solving the problem of misjudgment caused by shadows, reflections, occlusion, and mixed materials in complex urban scenes. Second, through a dual-channel retrieval mechanism that integrates visual similarity cases and professional text specifications, the generated natural language descriptions show significant improvements in semantic consistency and information completeness. Third, this invention possesses excellent anti-hallucination capabilities and scene robustness. Even in challenging scenarios such as reflections, shadows, and complex backgrounds, the BLEU-4 index remains consistently above 63.00%, an improvement of over 38% compared to the unenhanced model, significantly suppressing material fabrication and erroneous inferences. Finally, this method enables automated and traceable material description output for the entire city and individual buildings, and can be directly applied to practical business scenarios such as urban planning, building surveys, and post-disaster assessments, demonstrating high engineering implementation value and industry application potential.

[0042] In addition to the objectives, features, and advantages described above, the present invention has other objectives, features, and advantages. The invention will now be described in further detail with reference to the figures. Attached Figure Description

[0043] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0044] Figure 1 This is a flowchart illustrating the dual-retrieval enhancement generation method for material identification of a single building in this invention.

[0045] Figure 2 This is a schematic diagram of a subset of difficult scenarios in an embodiment of the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0047] Example 1

[0048] See Figure 1 This invention proposes a dual-retrieval enhanced generation method for material identification of single-building structures, comprising the following steps:

[0049] S1. Acquire multi-source data of the target building and perform preprocessing;

[0050] Multi-source data includes high-resolution remote sensing imagery, street view images, fine-grained building attribute data, open street maps (OSM) and point of interest (POI) data, standard text, and knowledge related to building attributes; among which:

[0051] High-resolution remote sensing imagery provides information on building roof morphology, spectral density, and surrounding environment.

[0052] Street view images provide details of building facade textures, door and window structures, and materials;

[0053] Fine-grained attribute data for buildings includes the building's location coordinates, function type, construction date, structural type, material properties, height, etc.

[0054] Open street maps and point-of-interest data provide supplementary information such as building use, name, and surrounding features;

[0055] Standard texts and related knowledge of building attributes include building material specifications, construction standards, material classification standards, material property documents, building design guidelines, and urban regulations.

[0056] Preprocessing of multi-source data includes:

[0057] Image preprocessing: Cropping (aligning individual buildings), normalizing, unifying size (e.g., 512×512 pixels), and noise reduction of remote sensing images and street view images;

[0058] Text preprocessing: removing noisy text, standardizing punctuation, handling stop words, restoring part-of-speech tags, etc., and using large models (such as GPT-type models) for text cleaning and format normalization;

[0059] Spatial matching and geocoding: Geocoding is used to spatially align building vector data with remote sensing images and street view images, so that each record corresponds to a single building.

[0060] S2. Construct a multimodal knowledge vector library: Based on the multi-source data, perform feature extraction and alignment to construct a visual vector index library and a text vector index library;

[0061] Feature extraction and alignment include:

[0062] Visual feature extraction: Multi-scale visual features are extracted from remote sensing images and street view images using pre-trained visual models (such as ResNet-50);

[0063] Text feature extraction: Use a pre-trained language model (such as bge-m3) to vectorize and embed standard text and building attribute association knowledge to obtain text features;

[0064] Feature alignment: Align visual features with building attributes, using the building's individual location code as the alignment element, and encoding the building's spatial location information, functional category, age, structural type, etc., into vector or structured features; Align text features with building attributes, and encode the building's spatial location information, functional type, age, structural type, etc., into vector or structured features;

[0065] Aligned visual and textual features are mapped to a unified vector space to construct a visual vector index library and a text vector index library. In the index library, multimodal entries are created for each building instance, including remote sensing images, street view images, building attribute vectors, and relevant standard text fragments, to achieve multimodal joint representation.

[0066] In this embodiment, the specific process for aligning multi-source data with building instances is as follows:

[0067] Using the building's geometric center coordinates as spatial anchor points, an accurate mapping is established between the building's unique ID and the corresponding remote sensing image slice. Specifically, assuming the building's geometric center spatial coordinates ( ) and the geometric center coordinates of the remote sensing image segmentation ( Then the spatial distance between the two It can be calculated using the Euclidean distance formula:

[0068] ;

[0069] Based on the calculation results, the remote sensing image tile with the smallest distance is selected to achieve accurate spatial mapping. Furthermore, to further enhance the alignment accuracy between the remote sensing image and the building instance, the field of view (...) FOV Geometric constraint mechanisms, including field of view analysis, Euclidean distance calculation, and occlusion detection, are used to select the view that best presents the main facade of the building from neighboring street view data. Specifically, the field of view ( FOV The calculation formula for ) is as follows:

[0070] ;

[0071] in, This is the width of the building from the viewing angle. This formula allows us to filter out the view with the largest field of view, ensuring a complete presentation of the building facade.

[0072] Next, using spatial topology association algorithms, core attributes such as functional type, construction year, structural type, and height are extracted from POI and building fine-grained attribute data. Simultaneously, OSM data is integrated to obtain environmental context information such as road network, green space distribution, and land use. This multidimensional data is ultimately integrated into a spatially contextualized "building material attribute set" and stored using a unified ID index.

[0073] Through this series of refined processing steps, an instance-level aligned multimodal data system is constructed, integrating remote sensing imagery, street view, and semantic attributes.

[0074] In this embodiment, the visual vector index library is constructed as follows:

[0075] The construction of the visual vector index library aims to transform the visual representation of urban buildings into a structured vector space, thereby enabling large-scale cross-instance transfer and reuse of material experience from "visually similar buildings". This embodiment uses a pre-trained visual model ResNet-50 to extract high-dimensional semantic features from street view facade images and capture comprehensive visual attributes such as building shape contours, window patterns, surface textures, and material qualities. Assuming that from street view images... The extracted high-dimensional visual features are The process can then be represented by the following formula:

[0076] ;

[0077] in, To pre-train the visual encoder, the visual embeddings of all building instances in the city are integrated into a unified vector space, and an approximate nearest neighbor (ANN) index structure is constructed using the FAISS high-performance vector retrieval framework. This structure can be represented by the following formula:

[0078] ;

[0079] in, It queries the visual feature vectors of a building. This indicates the nearest neighbor index returned by the FAISS framework. The number of similar building examples in the city. KThis represents the number of candidate instances returned. This index can quickly locate a group of candidate instances that are highly similar to the visual features of the target building from a database of millions of buildings, providing powerful "visual case priors" and "cross-building experience knowledge" for the subsequent material recognition stage.

[0080] In this embodiment, the text vector index library is constructed as follows:

[0081] For preprocessed text information, a pre-trained text encoder is used to transform each processed semantic unit in the text information into a high-dimensional vector representation, forming a text vector index library; at the same time, the metadata applicable to each specification in the text information is encoded, such as "applicable function: industrial building", "time range: modern", "environmental conditions: coastal / corrosive".

[0082] S3. Perform dual-channel retrieval, which includes visual channel retrieval and text channel retrieval, specifically:

[0083] S3.1 Visual Channel Retrieval (RAG-G Classes):

[0084] The input is a remote sensing image containing the target building. The image features are retrieved from the visual vector index library, such as remote sensing image entries with similar shapes to the target building, similar or adjacent street view images, and corresponding fine-grained building attributes (material attributes, structural type, etc.). The output is a visual query vector (including visual examples, material labels of nearby buildings, and similar case evidence such as the attributes of neighboring buildings).

[0085] Text channel retrieval (RAG-D class): Input text information (text query constructed from building function, age, structural type, regional planning information, etc.), retrieve the material-related semantic description of the building from the text vector index library, and generate semantic knowledge prompts;

[0086] The semantic description of building materials includes material standard descriptions that match the type of building (such as "the exterior walls of reinforced concrete residential buildings in a certain era are mostly plastered or tiled"), industry standard clauses, construction specification fragments, material terminology glossaries, and prohibited material combinations (logical constraints), etc.

[0087] Semantic knowledge hints include normative rules or matching clauses such as standard descriptions, constraints, and terminology templates.

[0088] Specifically, query requests are automatically generated based on information such as the building's specific function. These requests are retrieved from a text vector index to find the most relevant regulatory content, and the results are further refined through keyword filtering. This process accurately obtains information highly relevant to the current architectural design context, including recommended materials, materials to be avoided, and the constraints behind them. Finally, this information is organized into structured prompts to ensure that the multimodal large model always has solid engineering specifications as a basis when making material decisions. This approach not only improves the model's accuracy but also strengthens its ability to comply with industry standards.

[0089] S3.2 Perform evidence fusion and conflict resolution on the visual channel retrieval results and text channel retrieval results to obtain structured prompt information;

[0090] Evidence fusion involves integrating the visual features, textual features, visual query vectors, and semantic knowledge cues of the target building into a unified context vector.

[0091] Conflict resolution includes:

[0092] The visual channel retrieval results and text channel retrieval results are subjected to threshold filtering and similarity reordering. If there is a conflict between the two channels, a decision is made through preset rules or conflict resolution units. The final retained result is used as structured prompt information, and the reference information is retained to support result traceability.

[0093] S4. Input the remote sensing image and structured cue information of the target building into the multimodal large model to generate a material description for the single building. The building material description includes at least the roof material type, color, and shape (e.g., flat roof / sloping roof), the exterior wall material and finish (e.g., concrete plaster, brick-concrete structure, glass curtain wall, etc.), a reasonable inference description or interpretable text description combining the building's function and age, and structured results such as material category labels and structural type labels.

[0094] In this embodiment, the multimodal large model is any one of GLM-4-5V, GLM-4V-flash, GLM-4V-plus, Gemini-3-Pro-Vision, Doubao-seed-1-6-vision, Doubao-seed-1-6-lite, gpt-4o-mini, gpt-5-mini, and Qwen2.5-VL-7B-Instruct;

[0095] In this embodiment, experimental verification was conducted based on sample data from 1.3 million buildings in a certain city.

[0096] A comparative experiment was conducted on the performance of different models in the task of generating building material descriptions. The experimental results are shown in Table 1.

[0097] `rouge-1-F1` represents the F1 score based on unit group matching, `rouge-1-precision` represents the precision based on unit group matching, `rouge-1-recall` represents the recall based on unit group matching, `rouge-2-F1` represents the F1 score based on bigram matching, `rouge-2-Precision` represents the precision based on bigram matching, `rouge-2-recall` represents the recall based on bigram matching, `rouge-L-F1` represents the F1 score based on longest common subsequence matching, `rouge-L-precision` represents the precision based on longest common subsequence matching, `rouge-L-recall` represents the recall based on longest common subsequence matching, `BLEU-1` represents the precision based on unit groups, `BLEU-2` represents the precision based on bigrams, `BLEU-3` represents the precision based on triples, `BLEU-4` represents the precision based on quadruples, and `BERTScore` represents translation accuracy.

[0098] As can be seen from the data in Table 1, the Qwen2.5-VL-7B-Instruct model has the highest values ​​for all indicators and performs the best. Therefore, in this invention, the multimodal large model used is Qwen2.5-VL-7B-Instruct. Single-building material identification is achieved by using Qwen2.5-VL-7B-Instruct with dual retrieval enhanced generation (DC-RAG).

[0099] Table 1. Performance of different models in the task of generating building material descriptions.

[0100]

[0101] A comparative experiment was conducted using the DC-RAG mechanism. The specific experimental results are shown in Table 2. The data in Table 2 demonstrates that the model equipped with the DC-RAG framework achieved significant improvements across all natural language evaluation metrics, validating the effectiveness of the dual-channel retrieval enhancement mechanism. In all three base models, the introduction of DC-RAG resulted in consistent performance improvements. Taking the best-performing model, Qwen2.5-VL-7B-Instruct + DC-RAG, as an example, its BLEU-4 score reached 63.65%, a significant improvement compared to the original model without enhancement. Simultaneously, its BERTScore score was 75.20%, indicating that the generated description semantically captured material features well. The ROUGE-L-F1 index, reflecting the semantic recall of the text, also reached 86.30%, indicating that the generated material description not only matched the reference text at the n-gram level but also highly reproduced the real material features in the long sentence logical structure.

[0102] Table 2 Comparison of results for different models before and after adding DC-RAG

[0103]

[0104] To further explore the model's performance on specific materials, the performance of Qwen2.5-VL-7B-Instruct + DC-RAG on seven material categories (brick, metal, asphalt, cement, bare soil, roof tiles, and wood) was analyzed. The experimental results are shown in Table 3. Table 3 shows that for common urban materials such as metal and cement, the model's BLEU-4 scores reached 63.22% and 63.14%, respectively. This is attributed to RAG-G's ability to retrieve abundant similar cases in industrial areas and densely populated modern building areas, providing the model with sufficient visual priors.

[0105] In terms of refined identification of complex textures, it is worth noting that even for the more visually complex tile and brick categories, the model maintains extremely high descriptive accuracy (62.79% and 62.85% respectively in BLEU-4). This excellent performance is mainly attributed to the strong correlation constraints introduced by DC-RAG regarding "pitched roof - tile" and "historical building - brick wall", which effectively corrects the texture confusion problem caused by simply relying on remote sensing overhead view.

[0106] Table 3 Performance results of Qwen2.5-VL-7B-Instruct + DC-RAG on different materials

[0107]

[0108] To verify the ability of DC-RAG to suppress model hallucinations, a "difficult scene subset" containing 2354 samples was constructed (see [link to sample set]). Figure 2 This subset covers three typical challenging scenarios: high reflectivity, shadow occlusion, and complex background interference (cluttered background). Sample selection followed these rules: First, scenarios commonly encountered in practical applications and prone to model errors were chosen, such as urban villages, old residential areas, and urban renewal zones; second, these samples possessed significant visual interference factors, such as strong reflections, severely occluded shadows, or complex background structures. Furthermore, all samples were manually labeled to ensure accurate description of the material and background information for each challenging scenario. Table 4 details the model's performance under these conditions.

[0109] Referring to Table 4, in the identification of highly reflective materials (such as metals and glass), traditional visual models are highly susceptible to illusions caused by lighting conditions. Experimental data shows that the original Qwen2.5-VL-7B-Instruct achieves only 24.60% BLEU-4 on reflective samples, indicating that the model is frequently misled by specular reflections, generating incorrect material labels. However, after introducing DC-RAG, this metric surges to 63.00%, a relative improvement of over 115%.

[0110] Referring to Table 4, for buildings obscured by tall buildings or trees, the original model often suffers from "guessing illusions" due to "indistinct visibility," with its BLEU-4 dropping to 24.96% on the shadow template. However, DC-RAG successfully supplemented the missing visual information by retrieving visually similar neighboring buildings through the RAG-G channel as references, restoring the BLEU-4 to 63.81%.

[0111] As shown in Table 4, in scenes with cluttered backgrounds (such as those mixed with billboards and temporary sheds), the performance of the unenhanced model is also severely compromised (BLEU-4 at 22.57%). DC-RAG, through dual constraints, forcibly focuses attention on the main building, ultimately achieving an accuracy of 63.33%.

[0112] This demonstrates that, regardless of whether it's under reflective, shadowy, or complex backgrounds, DC-RAG's performance consistently remains above 63.00%, with minimal difference from normal scenes. This stability strongly proves that DC-RAG can effectively mitigate the problem of material illusion generation.

[0113] Table 4 Comparison of Anti-Hallucination Experiment Results

[0114]

[0115] Example 2

[0116] This embodiment provides a dual-retrieval enhanced generation system for identifying the material of a single building, used to implement the method described in Embodiment 1, including: a multi-source data acquisition and preprocessing module, a multimodal knowledge vector base construction module, a dual-channel retrieval module, and a generation and response module;

[0117] The multi-source data acquisition and preprocessing module is used to acquire and process high-resolution remote sensing images, street view images, fine-grained building attribute data, open street maps and point of interest vector data, standard text and building attribute association knowledge, and align the preprocessed data to a single building.

[0118] The multimodal knowledge vector library construction module is used to extract and align features from the processed multi-source data, and to build a visual vector index library and a text vector index library.

[0119] The dual-channel retrieval module is used to implement dual-channel retrieval and output structured prompt information;

[0120] The generation and response module is used to generate building material descriptions.

[0121] In this embodiment, the dual-channel retrieval module includes a visual channel retrieval unit, a text channel retrieval unit, and an evidence fusion and conflict resolution unit;

[0122] The visual channel retrieval unit is used to perform visual channel retrieval and output a visual query vector.

[0123] The text channel retrieval unit is used to perform text channel retrieval and generate semantic knowledge prompts;

[0124] The evidence fusion and conflict resolution unit is used to fuse the dual-channel retrieval results and resolve conflicts to generate structured prompt information.

[0125] In this embodiment, the multi-source data acquisition and preprocessing module also integrates a spatial matching and geocoding unit to ensure spatial alignment between remote sensing images, street view images, building vector data and building instances.

[0126] Example 3

[0127] The present invention also provides an electronic device corresponding to the above embodiments. The electronic device may be a processing device for a client, such as a mobile phone, a laptop, a tablet computer, a desktop computer, etc., to execute the methods of the above embodiments.

[0128] The electronic device of this embodiment includes a memory, a processor, and a computer program stored in the memory; the processor executes the computer program in the memory to implement the steps of the method described in the above embodiment.

[0129] In some implementations, the memory may be high-speed random access memory (RAM), and may also include nonvolatile memory, such as at least one disk storage.

[0130] In other implementations, the processor can be any type of general-purpose processor, such as a central processing unit (CPU) or a digital signal processor (DSP), and there is no limitation here.

[0131] The present invention also provides a readable storage medium corresponding to the above embodiments, wherein a computer program / instructions are stored thereon. When the computer program / instructions are executed by a processor, they implement the steps of the methods described in the above embodiments.

[0132] A computer-readable storage medium can be a tangible device that holds and stores instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof.

[0133] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0134] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0135] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0136] The above description is only a preferred embodiment of the present invention and does not limit the scope of the present invention. All equivalent structural transformations made under the inventive concept of the present invention using the contents of the present invention specification and drawings, or direct / indirect applications in other related technical fields, are included within the protection scope of the present invention.

Claims

1. A dual retrieval enhancement generation method for single building material identification, characterized in that, Includes the following steps: S1. Acquire multi-source data of the target building and perform preprocessing; multi-source data includes high-resolution remote sensing images, street view images, fine-grained building attribute data, open street maps and point of interest data, standard text and building attribute association knowledge; Preprocessing of multi-source data includes: Image preprocessing: Cropping and normalization of remote sensing images and street view images; Text preprocessing: noise removal, text cleaning, and format normalization; Spatial matching and geocoding: Geocoding is used to spatially align building vector data with remote sensing images and street view images, so that each record corresponds to a single building; S2. Construct a multimodal knowledge vector library: Based on the multi-source data, perform feature extraction and alignment to construct a visual vector index library and a text vector index library; Feature extraction and alignment include: Visual feature extraction: Multi-scale visual features are extracted from remote sensing images and street view images using a pre-trained visual model; Text feature extraction: The pre-trained language model is used to vectorize and embed the standard text and the knowledge of the association between building attributes to obtain text features; Feature alignment: Align visual features with building attributes, using the building's individual location code as the alignment element; align text features with building attributes; S3. Perform dual-channel search: S3.1 Visual Channel Retrieval: Input a remote sensing image containing the target building, retrieve image features from the visual vector index library, and output a visual query vector; Text channel retrieval: Input text information, retrieve the semantic description of the building's material association from the text vector index library, and generate semantic knowledge prompts; S3.2 Perform evidence fusion and conflict resolution on the visual channel retrieval results and text channel retrieval results to obtain structured prompt information; Conflict resolution includes: Threshold filtering and similarity reordering are performed on the visual channel retrieval results and text channel retrieval results. If there is a conflict between the two channels, a decision is made through preset rules or conflict resolution units. The final retained result is used as structured prompt information, and the reference information is retained to support result traceability. S4. Input the remote sensing image and structured prompts of the target building into the multimodal large model to generate a material description of the single building.

2. The single building-oriented building material quality recognition dual search enhancement generation method of claim 1, wherein, In S2, the aligned visual features and text features are mapped to a unified vector space to construct a visual vector index library and a text vector index library. In the index library, a multimodal entry is created for each building instance to achieve multimodal joint representation.

3. The single building-oriented building material quality recognition dual search enhancement generation method of claim 2, wherein, In S3.2, evidence fusion includes fusing the visual features, textual features, visual query vectors, and semantic knowledge hints of the target building into a unified context vector.

4. The single building-oriented building material quality recognition dual search enhancement generation method of claim 1, wherein, The multimodal large model is any one of GLM-4-5V, GLM-4V-flash, GLM-4V-plus, Gemini-3-Pro-Vision, Doubao-seed-1-6-vision, Doubao-seed-1-6-lite, gpt-4o-mini, gpt-5-mini, and Qwen2.5-VL-7B-Instruct; The building material description should include at least the roof material and the exterior wall material, as well as the material category label and the material-related attribute label.

5. A dual-retrieval enhanced generation system for identifying the material of a single building, used to implement the method described in any one of claims 1-4, characterized in that, include: Multi-source data acquisition and preprocessing module, multi-modal knowledge vector base construction module, dual-channel retrieval module, generation and response module; The multi-source data acquisition and preprocessing module is used to acquire and process high-resolution remote sensing images, street view images, fine-grained building attribute data, open street maps and point of interest vector data, standard text and building attribute association knowledge, and align the preprocessed data to a single building. The multimodal knowledge vector library construction module is used to extract and align features from the processed multi-source data, and to build a visual vector index library and a text vector index library. The dual-channel retrieval module is used to implement dual-channel retrieval and output structured prompt information; The generation and response module is used to generate building material descriptions.

6. The single building level facade material identification oriented dual retrieval augmented generation system of claim 5, wherein, The dual-channel retrieval module includes a visual channel retrieval unit, a text channel retrieval unit, and an evidence fusion and conflict resolution unit. The visual channel retrieval unit is used to perform visual channel retrieval and output a visual query vector. The text channel retrieval unit is used to perform text channel retrieval and generate semantic knowledge prompts; The evidence fusion and conflict resolution unit is used to fuse the dual-channel retrieval results and resolve conflicts to generate structured prompt information.

7. The single building level facade material identification oriented dual retrieval augmented generation system of claim 6, wherein, The multi-source data acquisition and preprocessing module also integrates a spatial matching and geocoding unit to ensure spatial alignment between remote sensing images, street view images, building vector data, and building instances.

Images