Method and apparatus for matching digital assets based on large models

Abstract

The application provides a method and device for matching digital assets based on large models, and relates to the technical field of artificial intelligence. The method for matching digital assets based on large models comprises the following steps: constructing a three-dimensional digital asset database according to original structured documents of to-be-matched digital assets; generating a candidate building asset list according to the three-dimensional digital asset database; extracting at least one building instance sub-image from original image data of a to-be-identified building as visual image data; and matching the visual image data and the candidate building asset list based on a pre-constructed large language model and a visual language model to obtain a target building asset. The application constructs a semantic three-dimensional digital asset library, and then generates a candidate building asset list as a basis for matching. Visual modal data is extracted from original image data of a to-be-identified building as a matching target. The application realizes matching by using a two-way collaborative matching mechanism of a large language model-visual language model, and has high accuracy and does not require a large amount of manual intervention.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a method and apparatus for matching digital assets based on a large model. Background Technology

[0002] Currently, with the widespread application of 3D simulation technology in smart cities, digital twins, virtual reality, and other fields, how to quickly and accurately match real-world buildings with 3D digital assets has become a key technological challenge. Traditional asset matching methods mainly rely on text retrieval, image feature matching, or multimodal fusion technology.

[0003] Specifically, text-based retrieval methods achieve asset retrieval through keyword matching or semantic similarity calculation. The drawback of these methods is their heavy reliance on the completeness and accuracy of the text description; when there are discrepancies between the text description and visual features, a semantic gap can easily arise, leading to matching errors. Image feature-based matching methods use convolutional neural networks (CNNs) or visual transformers (ViTs) to extract visual features from the query image and asset database images, matching them based on feature similarity. However, this method is sensitive to image quality, viewpoint changes, and background interference, and performs poorly when handling assets with large intra-category differences. Multimodal fusion methods (such as the CLIP (Contrastive Language-Image Pre-training) model) map images and text to the same feature space through contrastive learning. While these methods bridge the modal gap to some extent, they still belong to a passive retrieval paradigm, unable to actively mine implicit attributes in visual information, and have limited performance on fine-grained matching tasks.

[0004] Specifically, in the field of 3D digital asset matching, existing technologies typically face the following technical shortcomings: First, most existing methods employ unidirectional feature extraction and matching strategies. When the asset database is large or the visual features are complex, this passive retrieval approach struggles to accurately locate the target asset. Second, traditional methods often rely on static feature alignment when dealing with cross-modal semantic gaps, which can easily lead to insufficient matching accuracy in complex scenarios. Third, existing technologies lack effective iterative optimization mechanisms, making it difficult to self-correct when initial matching is inaccurate. Therefore, traditional asset matching methods frequently result in matching errors or require significant manual intervention. Summary of the Invention

[0005] Based on this, this application provides a method and apparatus for matching digital assets based on a large model, which achieves high accuracy in matching digital assets without requiring a large amount of human intervention.

[0006] According to one aspect of this application, a matching method for digital assets based on a large model is proposed, comprising: constructing a three-dimensional digital asset database based on the original structured documents of the digital assets to be matched; generating a candidate building asset list based on the three-dimensional digital asset database; extracting at least one building instance sub-image from the original image data of the building to be identified as visual image data; and matching the visual image data and the candidate building asset list based on a pre-constructed large language model and visual language model to obtain the target building asset.

[0007] According to some embodiments, a three-dimensional digital asset database is constructed based on the original structured document of the digital asset to be matched, including: performing attribute-based semantic annotation on the original structured document of the digital asset to be matched; converting the original structured document after semantic annotation to obtain three-dimensional digital asset data in a preset format; and storing the three-dimensional digital asset data in the database to obtain the three-dimensional digital asset database.

[0008] According to some embodiments, extracting at least one building instance sub-image from the original image data of the building to be identified as visual image data includes: using a pre-built target detection model to locate building instance targets of a preset category in the original image data and obtain candidate boxes; based on the candidate boxes, performing instance mask-based segmentation on the original image data to obtain segmentation results; and based on the segmentation results, cropping the original image data to obtain at least one building instance sub-image as visual image data.

[0009] According to some embodiments, based on candidate boxes, the original image data is segmented using an instance mask to obtain a segmentation result, including: within the region defined by the candidate boxes, pixel-level segmentation is performed on the instance part and non-instance part of the original image data to obtain a segmentation result; the segmentation result is optimized, and the segmentation result is updated using the optimized result.

[0010] According to some embodiments, based on a pre-built large language model and visual language model, visual image data and a candidate building asset list are matched to obtain the target building asset, including: S1: using the candidate building asset list as the current candidate building asset list and initializing the question-and-answer history; S2: based on the large language model, generating the current matching question according to the current candidate building asset list, the question-and-answer history, and pre-set prompt words; S3: based on the visual language model, performing inference calculation on the matching degree between the visual image data and the current matching question to obtain a first inference result, which serves as the first answer to the current matching question; S4: selecting the target building asset from the current candidate building asset list. S5: Select building assets as candidate building assets to be matched; S6: Based on the large language model, infer the degree of matching between the candidate building assets to be matched and the current matching question, and obtain the second inference result as the second answer to the current matching question; S7: Update the current candidate building asset list according to the first answer and the second answer, and save the current matching question, the first answer and the second answer to the question and answer history; S8: Repeat steps S4-S6 until the current candidate building asset list is traversed; S9: Repeat steps S2-S7 until the preset termination condition is met, and output the current candidate building asset in the current candidate building asset list of the last iteration as the target building asset.

[0011] According to some embodiments, step S2 includes: analyzing the current candidate building assets in the current candidate building asset list based on a large language model, and extracting the common features and key difference features of the current candidate building assets as the analysis results; and generating the current matching question based on the analysis results, question and answer history, and prompt words.

[0012] According to some embodiments, step S3 includes: extracting visual features from visual image data based on a visual language model; performing semantic parsing on the current matching problem based on the visual language model to obtain a parsing result; and performing reasoning calculation on the degree of matching between the visual features and the parsing result based on the visual language model to obtain a first reasoning result, which serves as the first answer.

[0013] According to some embodiments, step S6 includes: if the first answer and the second answer are inconsistent, deleting the candidate building asset to be matched from the current candidate building asset list.

[0014] According to some embodiments, the preset termination conditions include: only a preset number of candidate building assets remain in the current candidate building asset list, the current candidate building asset list cannot be narrowed down, and / or the second answer includes relevant information about valid candidate building assets.

[0015] According to some embodiments, the prompts include: analysis and reasoning instructions, question generation constraints, historical questioning angle avoidance strategies, and / or examples of questioning strategies.

[0016] According to one aspect of this application, a matching device for digital assets based on a large model includes: a database module for constructing a three-dimensional digital asset database based on the original structured documents of the digital assets to be matched; a list module for generating a candidate building asset list based on the three-dimensional digital asset database; a sub-graph module for extracting at least one building instance sub-graph from the original image data of the building to be identified, as visual image data; and a matching module for matching the visual image data and the candidate building asset list based on a pre-constructed large language model and a visual language model to obtain the target building asset.

[0017] According to one aspect of this application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method as described above.

[0018] According to one aspect of this application, a computer-readable medium is provided that stores a computer program or instructions thereon, which, when executed by a processor, implement the method as described above.

[0019] Through the embodiments provided in this application, a semantic three-dimensional digital asset library is constructed based on the original structured documents of the digital assets to be matched, thereby generating a candidate building asset list as the basis for digital asset matching. Visual modal data is extracted from the original image data of the buildings to be identified as the search targets for digital asset matching. A dual-path collaborative matching mechanism of "large language model-driven questioning - visual language model-visual answering" is constructed using a large language model and a visual language model to achieve high accuracy in matching digital assets without requiring a large amount of manual intervention. Attached Figure Description

[0020] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application.

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings, without exceeding the scope of protection claimed by this application.

[0022] Figure 1 A flowchart illustrating the matching method for digital assets based on a large model provided in this application embodiment;

[0023] Figure 2 A flowchart illustrating the process of constructing a three-dimensional digital asset database based on the original structured documents of the digital assets to be matched, as provided in this embodiment of the application.

[0024] Figure 3 The flowchart provided in this application describes the extraction of at least one building instance sub-image from the original image data of the building to be identified, as a visual image data.

[0025] Figure 4 The flowchart provided in this application embodiment describes a process for matching visual image data and a list of candidate building assets based on a pre-built large language model and a visual language model to obtain the target building asset.

[0026] Figure 5 A flowchart of step S2 provided in an embodiment of this application;

[0027] Figure 6 A flowchart of step S3 provided in the embodiments of this application;

[0028] Figure 7 A block diagram of a matching device for digital assets based on a large model, provided in an embodiment of this application;

[0029] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0032] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0033] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0034] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of this application. As used herein, the term "and / or" includes all combinations of any one and more of the associated listed items.

[0035] For specific implementation details, please refer to the following examples.

[0036] Figure 1 A flowchart illustrating a method for matching digital assets based on a large model, provided in an embodiment of this application. Figure 1 As shown, the method includes steps S110-S140.

[0037] In step S110, a three-dimensional digital asset database is constructed based on the original structured documents of the digital assets to be matched.

[0038] To achieve accurate matching of 3D digital assets, a 3D digital asset database containing rich semantic information is constructed to include multiple types of core architectural assets.

[0039] The original data of the digital assets to be matched is stored in a structured document. The digital assets to be matched include various types such as residential, commercial, industrial, hangars, roads, and vegetation.

[0040] The original structured document of the digital asset to be matched can be in various data organization formats, including Excel spreadsheets, database management systems, etc.

[0041] According to example embodiments, database management systems include relational databases (MySQL, PostgreSQL) and NoSQL databases (MongoDB), etc.

[0042] The original structured documents of the digital assets to be matched can be stored and exchanged in different data formats, such as XML, YAML, and Protocol Buffers.

[0043] Furthermore, the 3D digital asset database can also integrate different digital asset platforms, such as Unity AssetStore, Unreal Marketplace, and other third-party resource libraries.

[0044] In step S120, a list of candidate building assets is generated based on the three-dimensional digital asset database.

[0045] The candidate building asset list includes 3D digital assets that may be successfully matched with the building to be identified. During the generation of the candidate building asset list, 3D digital assets of the area related to the building to be identified can be extracted from the 3D digital asset database, or all 3D digital assets in the 3D digital asset database can be extracted to generate the candidate building asset list.

[0046] In step S130, at least one building instance sub-image is extracted from the original image data of the building to be identified as visual image data.

[0047] The original image data of the building to be identified can be real-world images of the building, or remote sensing images, bird's-eye views, aerial photographs, or artificially synthesized training data of the building. This application does not impose any restrictions on this.

[0048] The original image data of the building to be identified is instantiated, parsed, and structured to generate at least one building instance sub-image for subsequent matching.

[0049] The building instance sub-graph contains the building itself and the necessary context.

[0050] The resolution of the building instance sub-images can be adjusted according to actual needs. For example, the original image data of the building to be identified includes a city street view image. Based on this, multiple building instance sub-images are generated. Each sub-image focuses on an independent building instance and retains the surrounding roads or vegetation as context. This type of structured visual data serves as direct input for subsequent visual language models to perform attribute judgment and question answering.

[0051] According to the example embodiment, the resolution of the building instance subgraph is not less than 224x224 pixels.

[0052] In step S140, based on the pre-built large language model and visual language model, the visual image data and the candidate building asset list are matched to obtain the target building asset.

[0053] Using the visual image of the building to be identified and the list of candidate building assets as input, the Large Language Model (LLM) is used as the "active inferrer" to dynamically generate discriminative questions, and the Visual Language Model (VLM) is used as the "visual verifier" to provide accurate visual judgments. The visual image data and the list of candidate building assets are matched to form an iterative closed-loop question answering process, which yields the target building asset that matches the original image data of the building to be identified.

[0054] This application fully leverages the complementary advantages of multimodal large models. The Language Model (LLM) performs logical reasoning and question generation based on rich prior knowledge, while the Visual Model (VLM) makes professional judgments based on visual content. The two work together to leverage the advantages of the Language Model in reasoning and planning, and utilize the expertise of the Visual Model in image understanding, resulting in a technical effect of "1+1>2".

[0055] Large language models and visual language models can be obtained by using publicly available models or by training on relevant datasets; this application does not impose any restrictions on this.

[0056] When using publicly available models as both large language models and visual language models, this application does not impose restrictions on the specific types of large language models and visual language models, and selection can be made according to the circumstances.

[0057] According to the example implementation, large language models include various large language models such as the GPT series, LLaMA series, and ChatGLM. Visual language models include multimodal visual language models such as BLIP-2, LLaVA, and InstructBLIP.

[0058] Furthermore, the modules responsible for executing steps S110-S140 can be deployed in a centralized or distributed manner.

[0059] In existing technologies, such as in the construction of digital twin cities, it is necessary to match buildings in remote sensing images or street view images with a 3D asset database. Traditional methods often lead to matching errors or require significant manual intervention because they cannot deeply understand the visual attributes and semantic information of buildings. Especially when dealing with buildings that have similar visual features but different semantics (such as industrial plants and large warehouses), the distinguishing ability of existing methods is clearly insufficient.

[0060] To address the problems in existing technologies, this application aims to provide a three-dimensional digital asset matching method based on the dual-path collaboration of Large Language Model (LLM) and Visual Large Model (VLM). Specifically, by constructing an iterative question-answering mechanism of "LLM-driven questioning - VLM visual answering", it achieves a leap from passive retrieval to active interrogation-style querying, effectively solving the limitations of traditional methods in processing complex visual information and accurately bridging the semantic gap between visual and textual modalities.

[0061] Through the embodiments provided in this application, a semantic three-dimensional digital asset library is constructed based on the original structured documents of the digital assets to be matched, thereby generating a candidate building asset list as the basis for digital asset matching. Visual modal data is extracted from the original image data of the buildings to be identified as the search targets for digital asset matching. A dual-path collaborative matching mechanism of "large language model-driven questioning - visual language model-visual answering" is constructed using a large language model and a visual language model to achieve high accuracy in matching digital assets without requiring a large amount of manual intervention.

[0062] According to some embodiments, refer to Figure 2 In step S110, a three-dimensional digital asset database is constructed based on the original structured document of the digital asset to be matched, which can be specifically implemented through steps S210-S230.

[0063] In step S210, the original structured document of the digital asset to be matched is semantically annotated based on attributes.

[0064] Each digital asset to be matched is constructed with a complete semantic profile through systematic attribute annotation.

[0065] According to the example implementation, the annotation content includes: basic identification information (such as unique asset identifier, resource name, Chinese name), physical and geometric attributes (such as building function type, number of floors, three-dimensional key dimensions (such as front view width, left view width, height)), material and visual features (such as main facade material, roof shape, roof color) and rich text semantic description (such as structured natural language description, note field that integrates physical attributes, visual features and spatial form).

[0066] Understandably, the structure of the asset attribute table can be adjusted, with the addition or deletion of certain attribute fields, as long as the core semantic descriptive information is retained.

[0067] In step S220, the original structured document after semantic annotation is converted to a new format to obtain three-dimensional digital asset data in a preset format.

[0068] Through a data preprocessing workflow, the semantically annotated raw structured documents are converted into a preset format that is easy for the program to process, serving as 3D digital asset data. Each 3D digital asset data object in this preset format contains all the aforementioned attribute fields, ensuring data standardization and program readability.

[0069] Furthermore, data preprocessing can be achieved using dedicated functions such as read_excel_sheet and dataframe_to_json_list.

[0070] According to the example embodiment, the default format is a JSON list format.

[0071] In step S230, the three-dimensional digital asset data is stored in the database to obtain the three-dimensional digital asset database.

[0072] Using the selected database management system, create a 3D digital asset database based on the 3D digital asset data.

[0073] Database management systems can be selected based on actual needs, such as relational databases (MySQL, PostgreSQL) and NoSQL databases (MongoDB).

[0074] According to the example embodiment, the three-dimensional digital asset database adopts a hierarchical attribute description framework, and each three-dimensional digital asset data contains the following core dimensions:

[0075] 1) Basic identification information:

[0076] building_id: A unique identifier for the asset (e.g., 1-26).

[0077] name: The resource name of the asset within the engine (e.g., BP_building7_02).

[0078] Chinese Name: A readable Chinese description of the asset (e.g., “Basic Building 1”).

[0079] 2) Physical and geometric properties (providing quantitative basis for matching):

[0080] type: The functional type of the building (e.g., residential, commercial, industrial, hangar, road).

[0081] floors: The number of floors in a building.

[0082] Front view width, left view width, and height: key dimensions of the building in three-dimensional space (unit: meters).

[0083] 3) Material and visual characteristics (providing key clues for VLM visual question answering):

[0084] material: The main material of the facade (such as brick, concrete, glass, asphalt, and trees).

[0085] Facade combination features: The note field describes in detail the combination relationship between materials and components (such as "brick + window" or "concrete + window").

[0086] roof_shape: Roof shape (e.g., flat roof, sloping roof, dome, stepped).

[0087] color (Roof): The main color of the roof.

[0088] 4) Rich text semantic description (providing context for LLM understanding and questioning):

[0089] Note: A structured natural language description that integrates physical attributes, visual features, and spatial morphology. For example:

[0090] For BP_building_27 (Basic Building 14): "This is a residential building with a white facade and roof. The facade features regularly arranged, arched windows. There is a gazebo on the roof."

[0091] For BP_Industrial_Building2 (Basic Building 20): "This is an industrial building with a yellow facade, a gray, sloping roof, and a few windows on the facade."

[0092] Based on the above embodiments, each JSON object in the 3D digital asset data represents a complete digital asset archive, and the structure is shown in Table 1:

[0093] Table 1. Structured 3D Digital Asset Data in JSON Format

[0094]

[0095] In this embodiment, the three-dimensional digital asset database serves as the cornerstone for subsequent LLM-VLM dual-path collaborative matching. Its rich structured attributes and unstructured semantic descriptions provide comprehensive and accurate information support for LLM to generate differentiated questions, perform logical reasoning, and for VLM to perform visual verification.

[0096] According to some embodiments, refer to Figure 3 In step S130, at least one building instance sub-image is extracted from the original image data of the building to be identified as visual image data, which can be specifically implemented through steps S310-S330.

[0097] In step S310, a pre-built target detection model is used to locate building instance targets of a preset category in the original image data and obtain candidate boxes.

[0098] Based on the object detection model, candidate boxes are used to locate building instance targets of a preset category in the original image data.

[0099] Among them, the building instance targets of the preset category are the category targets of target detection, such as "building", etc.

[0100] According to the example implementation, an open vocabulary detection model is selected as the target detection model.

[0101] Among them, the open vocabulary detection model includes a Transformer-based detection model. In actual operation, based on natural language prompts (such as "building") including building instance targets of preset categories, the model locates the target in the original image data of the building to be identified and generates candidate bounding boxes (i.e., candidate boxes).

[0102] According to example embodiments, open vocabulary detection models include Grounding DINO model, OV-DETR (open vocabulary object detection model), CLIP (contrastive language-image pre-trained) detector, etc.

[0103] In step S320, based on the candidate bounding boxes, the original image data is segmented using instance masking to obtain the segmentation result.

[0104] For the candidate bounding boxes in the original image data, a high-precision instance mask is generated to ensure the detail integrity of the segmentation boundaries and obtain the segmentation result.

[0105] In practice, a pipeline combining detection and segmentation can be used, or traditional instance segmentation methods such as Mask R-CNN and YOLACT can be used.

[0106] According to the example embodiment, step S320 is implemented using the Grounded-SAM pipeline (open vocabulary detection and general segmentation integrated process), where SAM (Segment Anything Model) generates a high-precision instance mask based on the candidate boxes.

[0107] In step S330, based on the segmentation results, the original image data is cropped to obtain at least one building instance sub-image as visual image data.

[0108] Based on the segmentation results, at least one building instance sub-image containing the main building and the necessary context is cropped from the original image data as visual image data.

[0109] According to some embodiments, in step S320, the original image data is segmented based on the candidate box using an instance mask to obtain the segmentation result, which can be specifically implemented through steps S321-S322.

[0110] In step S321, within the region defined by the candidate box, the instance and non-instance parts of the original image data are segmented at the pixel level to obtain the segmentation result.

[0111] During segmentation, image features and bounding box cue features from the original image data are combined to perform fine pixel-level segmentation within the regions defined by candidate bounding boxes, and an output mask is used as the segmentation result. In this mask, each pixel is marked as belonging to (e.g., marked as 1) or not belonging to (e.g., marked as 0) the instance.

[0112] In step S322, the segmentation results are optimized, and the segmentation results are updated using the optimized results.

[0113] During the optimization process, cross-frame optimization can be performed on the segmentation results of key frames of the original image data based on the video sequence to which the original image data belongs, to eliminate segmentation jitter and improve the robustness of instance trajectories.

[0114] According to the example implementation, the optimization process is carried out using optical flow estimation and multi-view geometric constraints, or it can be carried out using a Transformer-based video instance segmentation method (such as the Mask2Former model).

[0115] Furthermore, in some embodiments, the Grounding DINO model is used as the object detection model, and the Grounded-SAM pipeline is used for subgraph segmentation.

[0116] In practice, firstly, the Grounding DINO model is used to initially locate target categories such as "buildings" and obtain candidate bounding boxes. Then, the Grounded-SAM pipeline generates pixel-level accurate instance masks for the candidate regions to ensure the detail integrity of object boundaries and obtain segmentation results. Next, temporal consistency fusion is performed in the video sequence to which the original image data belongs. Optical flow and multi-view geometric constraints are used to optimize the segmentation results across frames and eliminate jitter, obtaining robust building instance trajectories. Finally, based on the optimized instance masks, high-quality instance sub-images containing the main building and necessary context are cropped from the original image. This structured visual data serves as the direct input for VLM to perform attribute judgment and question answering.

[0117] According to some embodiments, refer to Figure 4 In step S140, based on the pre-built large language model and visual language model, the visual image data and the candidate building asset list are matched to obtain the target building asset, which can be achieved through steps S1-S8.

[0118] In step S1, the candidate building asset list is used as the current candidate building asset list, and the question and answer history is initialized.

[0119] During the matching process, maintain the current list of candidate building assets (denoted as the current_possible_buildings list) and the question-and-answer history (denoted as the qa_history list).

[0120] In the initial state, the current candidate building list initially includes all candidate building assets in the candidate building asset list.

[0121] The question and answer history is used to record the history of questions and answers. In the initial state, the question and answer history is empty.

[0122] In step S2, based on the large language model, the current matching question is generated according to the current candidate building asset list, question and answer history, and pre-set prompt words.

[0123] If the current list of candidate building assets contains multiple candidate building assets, then the large language model is activated as the "active inferencer".

[0124] The large language model receives the current candidate building asset list and question-answering history as context, and guides the generation of high-quality, discriminative questions through pre-set prompts, which serve as the current matching question.

[0125] Inputting question-and-answer history ensures the diversity of questions generated and the accuracy of reasoning, avoiding repetitive cycles.

[0126] In step S3, based on the visual language model, the degree of matching between the visual image data and the current matching question is inferred and calculated to obtain the first inference result, which serves as the first answer to the current matching question.

[0127] After the large language model generates the current matching question, it sends the current matching question along with visual image data including instance segmentation subgraphs of the buildings to be identified to the visual language model.

[0128] The visual language model acts as a "visual verifier," extracting deep visual features from the input visual image data, parsing the semantics of the current matching question, determining whether the image content satisfies the question description through cross-modal reasoning, and generating an answer, which is denoted as the first answer.

[0129] The question-and-answer mechanism can be adjusted according to actual needs. For example, it can stipulate that the answer to the currently matched question is a binary answer ("yes" or "no"), or it can allow non-binary answers, introduce a confidence scoring mechanism, and add multi-model voting decision-making, etc.

[0130] Furthermore, after step S3, the method also includes: performing uncertainty calibration on the first answer to improve the reliability of the answer.

[0131] In step S4, select the current candidate building asset from the current candidate building asset list as the candidate building asset to be matched.

[0132] Select candidate building assets from the current list of candidate building assets and perform attribute judgment based on description.

[0133] In step S5, based on the large language model, the degree of matching between the candidate building assets to be matched and the current matching question is inferred and calculated to obtain a second inference result, which serves as the second answer to the current matching question.

[0134] For the selected candidate building assets to be matched, the large language model acts as a "building asset evaluator" to make attribute judgments based on description. Specifically, for the selected candidate building assets to be matched, the large language model receives its detailed attributes and the current matching question, judges whether the image content meets the question description based on its internal knowledge, and generates an answer, which is recorded as the second answer.

[0135] In step S6, the current candidate building asset list is updated based on the first and second answers, and the current matching question, the first answer, and the second answer are saved to the question and answer history.

[0136] After each round of question and answer, the current list of candidate building assets is updated based on the first and second answers, and the current matching question, the first answer, and the second answer are recorded as question and answer history in the question and answer history.

[0137] This application effectively utilizes the understanding ability of large language models for architectural text descriptions, combined with the visual judgment of visual language models, to accurately filter the current list of candidate architectural assets.

[0138] In step S7, steps S4-S6 are repeated until the current candidate building asset list is traversed.

[0139] For each current candidate building asset in the current candidate building asset list, repeat steps S4-S6 to perform the corresponding description-based attribute judgment.

[0140] In step S8, steps S2-S7 are repeated until the preset termination condition is met, and the current candidate building asset in the current candidate building asset list of the last iteration is output as the target building asset.

[0141] The candidate range is gradually narrowed through multiple rounds of iterative question answering. When the preset termination condition is met, the current candidate building asset in the current candidate building asset list is returned as the target building asset.

[0142] During the output process, the unique identifier of the current candidate building asset in the current candidate building asset list can be output as the final asset ID of the target building asset.

[0143] Furthermore, if the current candidate building asset list is empty when the preset termination condition is met, a "no match" result is returned. If the large language model directly returns a valid asset ID when the preset termination condition is met, then the asset is directly adopted.

[0144] According to the example implementation, the initial candidate building asset list contains 10 buildings. After the first round of question and answer, this is reduced to 5 buildings, then to 3 in the second round, and finally to 1 in the third round (e.g., building_id "1-26"). The identification is then complete. During the iteration process, the question and answer history is fed back to the LLM to ensure question diversity, avoid duplicate questions, and achieve efficient and accurate matching.

[0145] This application embodiment uses multiple rounds of LLM-VLM dialogue for iterative filtering, updates the current candidate building asset list and question-and-answer history, and repeats the steps of question generation, visual answering, and attribute judgment until the preset termination condition is met. This achieves a leap from "passive retrieval" to "active interrogation-style query", dynamically mines the deep attributes hidden in visual information, and ultimately accurately bridges the semantic gap.

[0146] According to some embodiments, refer to Figure 5 In step S2, based on the large language model, the current matching question is generated according to the current candidate building asset list, question and answer history and pre-set prompt words. This can be achieved through steps S510-S520.

[0147] In step S510, based on the large language model, the current candidate building assets in the current candidate building asset list are analyzed, and the common features and key difference features of the current candidate building assets are extracted as the analysis results.

[0148] The large language model performs in-depth analysis on the current candidate building assets in the current candidate building asset list. Based on the relevant attributes labeled, it extracts the common features and key differences among the current candidate building assets as the analysis results.

[0149] According to the example implementation, common features and key differences among the current candidate building assets are extracted from fields such as type, material, roof shape, and semantically rich note.

[0150] In step S520, based on the large language model, the current matching question is generated according to the analysis results, question and answer history, and prompt words.

[0151] To prevent getting stuck in an invalid loop, the large language model must ensure that the newly generated question is different from all the questions recorded in the question-and-answer history in terms of the questioning perspective.

[0152] At the same time, strict constraints are set in the prompt words.

[0153] According to the example embodiment, the constraints set in the prompt include: prohibiting questions about specific identifiers, names, or precise values ​​(such as height and width), and also prohibiting direct comparisons between multiple buildings, thereby ensuring that the question can be answered based solely on the visual content of the current image.

[0154] Based on this, the large language model generates at least one question that focuses on visual features and satisfies the question-answering mechanism, which serves as the current matching question.

[0155] In some embodiments, the current matching question is a binary question that focuses on visual features and expects an answer of "yes" or "no".

[0156] Furthermore, the generation strategy for the current matching problem can be optimized, such as by using methods based on information gain or uncertainty sampling.

[0157] Through steps S510-S520, the large language model can dynamically and intelligently generate precise questions such as "Is this building an industrial building?" or "Does this building have a large number of regularly arranged windows on its exterior walls?" thereby driving the dual-path collaborative system to perform effective iterative screening.

[0158] According to some embodiments, refer to Figure 6 In step S3, based on the visual language model, the degree of matching between the visual image data and the current matching question is inferred and calculated to obtain the first inference result, which serves as the first answer to the current matching question. This can be achieved through steps S610-S630.

[0159] In step S610, visual features of the visual image data are extracted based on the visual language model.

[0160] VLM visual response uses a multimodal large model, which can be implemented through function calls (such as ask_vlm_about_building).

[0161] Visual features are extracted from visual image data based on a visual language model.

[0162] In the feature extraction process, pre-trained visual encoders (such as ViT or ResNet variants) in visual language models are used to extract multi-level features from visual image data, focusing on attributes such as structure, appearance, texture, and color.

[0163] In step S620, semantic parsing is performed on the current matching problem based on the visual language model to obtain the parsing result.

[0164] By utilizing a text encoder in a visual language model (such as a variant of BERT or GPT), the current matching problem in natural language format is parsed and transformed into an internal query representation as the parsing result.

[0165] In step S630, based on the visual language model, the degree of matching between visual features and parsing results is inferred and calculated to obtain a first inference result, which serves as the first answer.

[0166] By using a visual language model, visual features and parsing results are fused based on visual and textual features. Alignment and reasoning are performed through an attention mechanism to determine whether the image content meets the question description. The first reasoning result that conforms to the question-and-answer mechanism is generated as the first answer, which can then be used as the selection criterion in subsequent iterations.

[0167] According to the example embodiment, the first answer is a concise response of "yes" or "no".

[0168] In this embodiment of the application, the VLM acts as a "visual verifier," receiving the current matching question and visual image data generated by the LLM, performing deep visual feature extraction and question semantic understanding on the image content, and generating an answer (e.g., "yes" or "no") through cross-modal reasoning decision-making.

[0169] According to some embodiments, in step S6, the current candidate building asset list is updated based on the first answer and the second answer, and the current matching question, the first answer and the second answer are saved to the question and answer history. This can be specifically implemented through step S61.

[0170] In step S61, if the first answer and the second answer are inconsistent, the candidate building asset to be matched is removed from the current candidate building asset list.

[0171] Compare the LLM's judgment on the candidate building asset to be matched (i.e., the second answer) with the VLM's answer (the first answer, which can be understood as the user's expected answer). If the two are inconsistent, the candidate building asset to be matched is removed from the current candidate building asset list.

[0172] If both match, the candidate building asset to be matched will be retained from the current candidate building asset list.

[0173] Furthermore, a new_possible_buildings list can be maintained. In each round of question and answer, the candidate building assets that match the first and second answers are saved to the new candidate building asset list. Step S7 is then implemented. The current candidate building asset list is traversed. After that, the new candidate building asset list is replaced with the new candidate building asset list. Step S8 is then implemented.

[0174] Furthermore, the current list of candidate building assets can also be updated in other ways, such as by introducing probability weights instead of direct deletion.

[0175] According to the example implementation, in actual operation, the interaction flow and output of the VLM clearly demonstrate its functionality: When the LLM poses the current matching question, "Is this building an industrial building?", it sends this question along with the visual image data of the building to be identified to the VLM. After processing, the VLM returns its judgment result. The actual record is as follows:

[0176] "System initialization: Successfully read sheet '0930' of file '. / UE_resource_v2.xls'. Starting building asset identification process..."

[0177] Nine-Grid Large Language Model LLM Question: Is this building an industrial building? [Image]

[0178] Nine-Grid Multimodal Basic Large Model (VLM) A: No

[0179] After filtering, **19** possible buildings remain.

[0180] List:**ID:1,2,3,9,10,11,12,13,14,15,16,17,18,19,21,23,24,25,26*”

[0181] In this embodiment, the VLM determines that the building is not an industrial building based on the image content and gives a "no" answer. This answer from the VLM will serve as a key basis for the subsequent LLM to screen candidate buildings, ensuring that the screening process is not only based on text descriptions, but also incorporates real-time and targeted visual perception information more efficiently and accurately, greatly improving the accuracy of matching.

[0182] LLM further functions as a "building asset appraiser," assessing the attributes of each candidate building based on the VLM's response. It compares the LLM's assessment of a single building with the VLM's response; if they match, the building is retained, and if they do not match, it is removed, thus achieving precise filtering of the candidate list.

[0183] According to some embodiments, the preset termination conditions include:

[0184] The current candidate building asset list only has a preset number of candidate building assets remaining, the current candidate building asset list cannot be narrowed down, and / or the second answer includes relevant information about valid candidate building assets.

[0185] According to the example embodiment, the preset quantity is 1.

[0186] According to the example implementation, the relevant information of the valid candidate building assets directly returned by LLM includes a valid identifier.

[0187] Furthermore, in some embodiments, the preset termination condition may also include a maximum number of iterations to prevent infinite loops. According to an example embodiment, the maximum number of iterations is set to 10 iterations.

[0188] Furthermore, an early termination mechanism can be added to the preset termination conditions. The specific implementation methods can be diversified, such as based on confidence thresholds or trend analysis.

[0189] According to some embodiments, the prompt words include:

[0190] Examples of analysis and reasoning instructions, question generation constraints, strategies for avoiding historical questioning angles, and / or questioning strategies.

[0191] Understandably, in the construction of the LLM-VLM dual-path collaborative question-answering mechanism, the LLM-driven question generation is guided by system prompts.

[0192] According to the example embodiment, the prompt word design includes the following key elements:

[0193] 1) Analysis and Reasoning Instructions: Requires LLM to perform in-depth analysis of the current candidate building list and extract common features and key differences from the labeled attribute fields (such as type, material, roof_shape, note, etc.).

[0194] 2) Question generation constraints: Limit LLM to generate questions based solely on visual features and whose answers conform to the question-and-answer mechanism (e.g., binarization). Prohibit asking for building_id, name, precise numerical values, or making direct comparisons. Ensure that questions can be answered solely based on image content.

[0195] 3) Historical avoidance and strategic guidance: Historical questioning angle avoidance strategies include: requiring LLMs to avoid repeating historical questioning angles; questioning strategy examples include: guiding the use of relative descriptions and overall form to formulate questions through examples (such as "Does this building have a large number of regularly arranged windows on its exterior walls?" "Is the overall shape of this building slender and tall?"), thereby efficiently narrowing down the candidate range.

[0196] Furthermore, reinforcement learning can be combined to optimize the prompt words, thereby providing better guidance to the large language model and improving the ability to discriminate generated questions.

[0197] Based on the above embodiments, this application realizes a paradigm shift from traditional passive retrieval to proactive interrogation-style querying. Existing technologies largely rely on unidirectional feature matching, making it difficult to deeply mine the implicit attributes in visual information. This application, however, uses LLM as an "active inferrer" to dynamically generate discriminative questions, and VLM as a "visual verifier" to provide accurate visual judgments, forming an iterative closed-loop question-and-answer system, effectively solving the semantic gap problem in complex visual scenarios. The multi-round iterative screening strategy significantly improves the accuracy and robustness of asset matching. Traditional methods often struggle to self-correct after a single matching failure, while this application, by maintaining a candidate list and question-and-answer history, dynamically adjusts the search scope in each iteration, gradually eliminating assets that do not meet the criteria, ultimately achieving accurate identification and greatly reducing the false matching rate. By constructing a standardized semantic asset library and a modular question-and-answer system, this method can adapt to asset matching needs of different scales, possessing good scalability and practicality. It demonstrates significant technical advantages in practical application scenarios such as digital twins and smart cities, transforming the traditional matching process, which requires a large amount of manual intervention, into an automated and intelligent pipeline operation.

[0198] The following describes an apparatus embodiment of this application, which can be used to perform the method embodiment of this application. For details not disclosed in the apparatus embodiment of this application, please refer to the method embodiment of this application.

[0199] Figure 7 A block diagram of a matching device for digital assets based on a large model, according to an exemplary embodiment, is shown.

[0200] Figure 7 The apparatus shown can perform the aforementioned matching method for digital assets based on a large model according to embodiments of this application.

[0201] like Figure 7 As shown, a matching device for digital assets based on a large model may include:

[0202] See Figure 7 Referring to the preceding description, the database module 710 is used to construct a three-dimensional digital asset database based on the original structured documents of the digital assets to be matched.

[0203] List module 720 is used to generate a list of candidate building assets based on a 3D digital asset database.

[0204] The subgraph module 730 is used to extract at least one building instance subgraph from the original image data of the building to be identified, as visual image data.

[0205] The matching module 740 is used to match visual image data and a list of candidate building assets based on a pre-built large language model and visual language model to obtain the target building asset.

[0206] The device performs functions similar to those described above; other functions are described in the preceding descriptions and will not be repeated here.

[0207] This application discloses an electronic device, including: a processor; and a memory storing a computer program, which, when executed by the processor, causes the processor to execute the above-described instruction generation method.

[0208] For example, refer to Figure 8 , Figure 8 The illustrated electronic device 800 includes a processor 801 and a memory 803. The processor 801 and the memory 803 are connected, for example, via a bus 802. Optionally, the electronic device 800 may also include a transceiver 804. It should be noted that in practical applications, the transceiver 804 is not limited to one type, and the structure of this electronic device 800 does not constitute a limitation on the embodiments of this application.

[0209] Processor 801 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in this application. Processor 801 may also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0210] Bus 802 may include a pathway for transmitting information between the aforementioned components. Bus 802 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 802 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0211] The memory 803 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other storage medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0212] The memory 803 stores application code that executes the scheme of this application, and its execution is controlled by the processor 801. The processor 801 executes the application code stored in the memory 803 to implement the content shown in the foregoing method embodiments.

[0213] Figure 8 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0214] This application discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, causes the processor to execute an instruction generation method.

[0215] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0216] The above are only some embodiments of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for matching digital assets based on large models, characterized in that, The method comprises the following steps: constructing a three-dimensional digital asset database according to original structured documents of digital assets to be matched; generating a candidate building asset list according to the three-dimensional digital asset database; extracting at least one building instance sub-image from original image data of a building to be identified as visual image data; matching the visual image data and the candidate building asset list based on a pre-constructed large language model and a visual language model to obtain a target building asset; wherein matching the visual image data and the candidate building asset list based on the pre-constructed large language model and the visual language model to obtain the target building asset comprises: S1: taking the candidate building asset list as a current candidate building asset list, and initializing a question and answer history record; S2: generating a current matching question based on the large language model according to the current candidate building asset list, the question and answer history record, and a pre-set prompt word; S3: performing inference calculation on the matching degree of the visual image data and the current matching question based on the visual language model to obtain a first inference result as a first answer to the current matching question; S4: selecting a current candidate building asset from the current candidate building asset list as a candidate building asset to be matched; S5: performing inference calculation on the matching degree of the candidate building asset to be matched and the current matching question based on the large language model to obtain a second inference result as a second answer to the current matching question; S6: updating the current candidate building asset list according to the first answer and the second answer, and saving the current matching question, the first answer, and the second answer to the question and answer history record; S7: repeating steps S4-S6 until the current candidate building asset list is traversed; S8: repeating steps S2-S7 until a pre-set ending condition is met, and outputting a current candidate building asset in the current candidate building asset list in the last iteration as the target building asset.

2. The method of claim 1, wherein, constructing a three-dimensional digital asset database according to original structured documents of digital assets to be matched, comprising: performing attribute-based semantic labeling on the original structured documents of the digital assets to be matched; performing format conversion on the original structured documents after semantic labeling to obtain three-dimensional digital asset data in a pre-set format; storing the three-dimensional digital asset data into a database to obtain the three-dimensional digital asset database.

3. The method of claim 1, wherein, extracting at least one building instance sub-image from original image data of a building to be identified as visual image data, comprising: positioning building instance targets of a pre-set category in the original image data using a pre-constructed target detection model, and obtaining candidate boxes; performing instance mask-based segmentation on the original image data based on the candidate boxes to obtain segmentation results; performing cropping on the original image data based on the segmentation results to obtain at least one building instance sub-image as visual image data.

4. The method of claim 3, wherein, performing instance mask-based segmentation on the original image data based on the candidate boxes to obtain segmentation results, comprising: perform pixel-level segmentation on the instance part and the non-instance part of the original image data in a region delimited by the candidate box to obtain a segmentation result; perform optimization on the segmentation result, and update the segmentation result by using the optimized result.

5. The method of claim 1, wherein, Step S2 includes: based on the large language model, analyze the current candidate building asset in the current candidate building asset list, and extract common features and key difference features of the current candidate building asset as analysis results; based on the large language model, generate a current matching question according to the analysis results, the question and answer history record, and the prompt word.

6. The method of claim 1, wherein, Step S3 includes: based on the visual language model, extract visual features of the visual image data; based on the visual language model, perform semantic analysis on the current matching question to obtain an analysis result; based on the visual language model, perform inference calculation on the matching degree of the visual features and the analysis result to obtain the first inference result as the first answer.

7. The method of claim 1, wherein, Step S6 includes: in the case that the first answer and the second answer are inconsistent, delete the to-be-matched candidate building asset from the current candidate building asset list.

8. The method of claim 1, wherein, The preset end condition includes: only a preset number of candidate building assets are left in the current candidate building asset list, the current candidate building asset list cannot be narrowed down, and / or the second answer includes relevant information of an effective candidate building asset.

9. The method of claim 1, wherein, The prompt word includes: analysis and inference instructions, question generation constraints, historical questioning angle avoidance strategies, and / or questioning strategy examples. 10.A device for matching digital assets based on large models, characterized by, includes: a database module configured to construct a three-dimensional digital asset database according to original structured documents of to-be-matched digital assets; a list module configured to generate a candidate building asset list according to the three-dimensional digital asset database; a subgraph module configured to extract at least one building instance subgraph from original image data of a to-be-identified building as visual image data; a matching module configured to match the visual image data and the candidate building asset list based on a pre-constructed large language model and a visual language model to obtain a target building asset; The matching module is further configured to: S1: take the candidate building asset list as a current candidate building asset list, and initialize a question and answer history record; S2: based on the large language model, generate a current matching question according to the current candidate building asset list, the question and answer history record, and a pre-set prompt word; S3: based on the visual language model, perform inference calculation on the matching degree of the visual image data and the current matching question to obtain a first inference result as a first answer of the current matching question; S4: select a current candidate building asset from the current candidate building asset list as a to-be-matched candidate building asset; S5: based on the large language model, perform inference calculation on the matching degree of the to-be-matched candidate building asset and the current matching question to obtain a second inference result as a second answer of the current matching question; S6: updating the current candidate building asset list according to the first answer and the second answer, and saving the current matching question, the first answer and the second answer to the question and answer history record; S7: repeating steps S4-S6 until the current candidate building asset list is traversed; S8: repeating steps S2-S7 until a preset ending condition is met, and outputting a current candidate building asset in the current candidate building asset list in the last iteration as the target building asset.

11. An electronic device, comprising: Comprise: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method as claimed in any one of claims 1-9.

12. A computer readable storage medium having stored thereon a computer program or instructions, characterized in that, The computer program or instructions implement the method as claimed in any one of claims 1-9 when executed by a processor. The computer program or instructions implement the method as claimed in any one of claims 1-9 when executed by a processor.

Images