Large model-based self-guided knowledge graph construction method and device

Abstract

The application provides a large model-based self-guided knowledge graph construction method and device, wherein the method comprises: constructing a text dataset originating from an open domain and preprocessing to obtain input data; generating a candidate prompt template using a prompt word self-optimization model, forming a general prompt template according to a feedback result, using the general prompt template as the input of an iterative labeling strategy, and using the last layer prompt template to guide a large language model to construct a first graph based on entities and relationships extracted from the input data; querying entities in a pre-acquired standard entity library, eliminating entities that are not queried and abstract semantics to obtain an entity pool, calling an image search interface to query a preset number of image results corresponding to each entity in the entity pool, and structuring and associating the image results with the corresponding entities in the first graph to obtain a second graph; using a double-tower cross-modal matching network to fuse and align the entities and image results of the second graph to generate a semantic matching score, and performing entity disambiguation to obtain a final graph.

Description

Technical Field

[0001] This invention relates to the field of knowledge graph construction technology, and in particular to a self-guided knowledge graph construction method and apparatus based on a large model. Background Technology

[0002] With the increasing prevalence of multimodal data such as images, videos, and audio, how to effectively integrate multimodal information into knowledge graphs and improve the quality of entity recognition, relation extraction, and entity alignment has become a major challenge in the field of knowledge graph construction.

[0003] In related technologies, traditional knowledge graph construction schemes typically rely on manually predefined ontology to regulate entity extraction, relation identification, and modeling processes. This approach offers high accuracy and controllability within specific closed domains, but its applicability is significantly limited and inefficient in open domains. Large-scale language models excel in semantic understanding and generation in open domains, automatically generating entities and relations through soft-coded prompts without requiring manually predefined ontology schemas. However, existing prompt templates are often generated using static techniques, with models using templates generating templates in a single round or randomly combining them from a predefined template library. This results in insufficient semantic depth understanding and a lack of real-time correction feedback, leading to severely inaccurate entity and relation extraction. Furthermore, in the process of cross-modal alignment of text with other types, similarity calculations are based solely on text lexical units, failing to capture higher-level features from isolated units and exhibiting insufficient ability to handle ambiguous entities.

[0004] Based on the above analysis of the development status of this technology field, the existing technologies lack a solution that uses soft coding prompts to guide large language models to automatically infer and generate candidate entities and relationships, and introduces joint processing between images and text. Summary of the Invention

[0005] The purpose of this invention is to provide a self-guided knowledge graph construction method and apparatus based on a large model, aiming to solve the above-mentioned problems in the prior art.

[0006] According to a first aspect of the present invention, a self-guided knowledge graph construction method based on a large model is provided, comprising: We construct and preprocess a text dataset derived from the open domain to obtain the input data; Candidate prompt templates are generated using a prompt word self-optimization model. A general prompt template is formed based on the feedback results. The general prompt template is used as the input for the iterative annotation strategy. The final prompt template is used to guide the large language model to construct the first graph from the entities and relationships extracted from the input data. Query entities in the pre-acquired standard entity library, remove entities that are not found or have abstract semantics to obtain an entity pool, call the image search interface to query the preset number of image results corresponding to each entity in the entity pool, and structurally associate the image results with the corresponding entities in the first map to obtain the second map. A dual-tower cross-modal matching network is used to fuse and align entities in the second atlas with the image results, generating semantic matching scores. Entity disambiguation is then performed based on the semantic matching scores to obtain the final atlas.

[0007] According to a second aspect of the present invention, a self-guided knowledge graph construction apparatus based on a large model is provided, comprising: The initialization module is used to construct and preprocess a text dataset derived from the open domain to obtain input data; The iterative annotation module is used to generate candidate prompt templates using a prompt word self-optimization model, form a general prompt template based on the feedback results, use the general prompt template as input for the iterative annotation strategy, and use the final prompt template to guide the large language model to construct the first graph from the entities and relationships extracted from the input data. The entity linking module is used to query entities in a pre-acquired standard entity library, remove entities that are not found or have abstract semantics to obtain an entity pool, call the image search interface to query a preset number of image results for each entity in the entity pool, and structurally associate the image results with the corresponding entities in the first graph to obtain the second graph. The alignment and disambiguation module is used to fuse and align entities in the second atlas with the image results using a dual-tower cross-modal matching network, generate semantic matching scores, and perform entity disambiguation based on the semantic matching scores to obtain the final atlas.

[0008] According to a third aspect of the present invention, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the self-guided knowledge graph construction method based on a large model as provided in the first aspect of the present disclosure.

[0009] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which an information transmission implementation program is stored, which, when executed by a processor, implements the steps of the self-guided knowledge graph construction method based on a large model provided in the first aspect of the present disclosure.

[0010] The technical solution provided by the embodiments of the present invention has the following beneficial effects: by utilizing the ability of large language models to understand open-domain languages, the large language model is guided to generate entities and relations through soft-coded prompts in the template, without the need for manual pre-definition of ontology, thereby achieving low-cost and highly adaptable knowledge graph construction; by introducing a joint processing mechanism of images and text, and through multimodal embedding alignment to achieve accurate entity disambiguation and enhanced representation, the accuracy, richness and scalability of knowledge graph construction are greatly improved.

[0011] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a self-guided knowledge graph construction method based on a large model, according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the cross-modal alignment framework according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the self-booting execution architecture according to an embodiment of the present invention; Figure 4 This is a schematic diagram of a self-guided knowledge graph construction device based on a large model according to an embodiment of the present invention; Figure 5 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0015] Method Implementation Examples According to embodiments of the present invention, a self-guided knowledge graph construction method based on a large model is provided. Figure 1This is a flowchart of a self-guided knowledge graph construction method based on a large model, as described in an embodiment of the present invention. Figure 1 As shown, the self-guided knowledge graph construction method based on a large model according to an embodiment of the present invention specifically includes: In step S110, a text dataset derived from the open domain is constructed and preprocessed to obtain input data, specifically including: A text dataset is constructed based on the current task. Data is collected from sources such as social media platforms Twitter and Weibo, official news, and social media. The text dataset is then filtered for advertisements and deduplicated to obtain the input data.

[0016] In step S120, a candidate prompt template is generated using a prompt word self-optimization model. A general prompt template is formed based on the feedback results. This general prompt template is used as input to an iterative annotation strategy. The final-layer prompt template guides the large language model to construct the first graph from the entities and relationships extracted from the input data. Specifically, this includes: Candidate suggestion templates are generated using the MiPro V2 self-optimization model for suggestion words in the DSPy framework; The MiPro V2 self-optimizing prompt model specifies that prompts should include a clear description of the task objective, such as listing all entities and their relationships; an unrestricted list of entity types, such as people, countries, organizations, events, and weapons; and formatted output requirements, such as outputting entities as structured triples with type and description. By combining the feedback results during the training process to self-optimize the suggestion combination, the candidate suggestion templates are used to extract entities and relationships in the input data. The extraction effect of different candidate suggestion templates is evaluated, and a weighted mixing strategy is formed based on the extraction effect. The weighted mixing strategy is used to fuse the candidate suggestion templates to obtain a general suggestion template. The weighted mixing strategy can combine the suggestion words in different candidate suggestion templates. Preferably, an adaptive prompt generation algorithm is set to assist dynamic decision-making. When the entity density of the input data document exceeds the density threshold, the system automatically selects a high-precision prompt template. When information in specific fields such as military and political fields is detected, the corresponding field-customized prompt library is activated. In other cases, a general weighted hybrid method is used.

[0017] A generic suggestion template is optimized using an iterative annotation strategy that includes pre-annotation layers, re-annotation layers, and self-reflection layers. Each layer performs independent suggestion optimization, and each iterative layer maintains an independent suggestion pool and weighted mixing strategy. Optimization gradient information is passed between layers. The suggestion pool is the suggestion word content in the template, and the optimization gradient information refers to the performance feedback signal of the previous layer's task. In the pre-labeling layer, a general prompt template is used for extraction to obtain the first extraction result. This stage guides the large model to quickly generate an initial set of entities and relations. This stage aims for high recall and allows the generation of more redundant candidates. In the re-annotation layer, the error and omission information identified in the first extraction result is fed back to the prompt word self-optimization model, and the re-annotation layer template is reconstructed and the second extraction result is output. In this stage, more context-specific and refined prompts are constructed to correct misidentification, supplement missing information, and reorganize the semantic relationships between entities. In the self-reflection layer, metacognitive cue words are constructed based on the second extraction results to form the final cue template. The final cue template is used to correct errors before the large language model outputs, which is equivalent to requiring the large language model to self-examine, correct, and reconstruct, simulating the expert proofreading process to enhance stability, and obtaining the third extraction results to form the first graph.

[0018] In this embodiment of the invention, each layer can output a structured entity and relation format, and the finally extracted entities and relations are formatted and converted to organize into an initial first graph; the outputs of other layers can also be stored and maintained to facilitate the integration of other tasks. In preferred embodiments of the invention, a dual-objective collaborative optimization function is constructed, and multiple rounds of three-layer optimization are performed. In addition to entity recall and relation precision, a graph coherence evaluation index and a multimodal alignment expectation index are added. The graph coherence evaluation index ensures that the extracted entity-relation triples form a more meaningful subgraph structure semantically. The multimodal alignment expectation index considers the feasibility of subsequent image matching in advance and detects whether the text contains visually unsolvable content. Alternatively, a multi-armed slot machine algorithm in reinforcement learning is adopted, which regards the combination of prompt text as "arms" and uses graph quality as a reward signal.

[0019] In step S130, entities are queried in a pre-acquired standard entity library. Entities that are not found or have abstract semantics are removed to obtain an entity pool. An image search interface is called to query a preset number of image results corresponding to each entity in the entity pool. The image results are then structurally associated with the corresponding entities in the first atlas to obtain a second atlas. Specifically, this includes: Entities in the first graph need to be linked to determine if they are core entities. Wikipedia is used as the standard entity database; entity names are directly used as keywords to search Wikipedia. If a result is found, the entity is retained and added to the candidate entity pool; otherwise, it is removed. Furthermore, entities with semantic value in the image need to be further filtered, for example: Concrete entities, such as "Challenger Tank" and "White House," are retained. Abstract entities: If conflict or foreign policy is involved, such entities are deleted; After combining keywords to construct a search query, the Google / Bing image search API is called to obtain the first N results. The image results containing the image URL, thumbnail and original image path are then obtained and stored. By structurally associating the image URL with the corresponding entity in the first graph, a second graph containing JSON structured information is obtained. The JSON structure is shown below: Json { "entity_name": "Challenger Tank", "image_url": "Image source URL", Image title: "British Challenger main battle tank displayed" } In step S140, a dual-tower cross-modal matching network is used to fuse and align the entities in the second atlas with the image results, generating a semantic matching score. Entity disambiguation is then performed based on the semantic matching score to obtain the final atlas, specifically including: The entity names and image results of the second map are input into the dual-tower cross-modal matching network, which includes an encoding layer, a cross-modal attention layer, and an anchor point recognition layer. The text feature vector corresponding to the entity is obtained through the text encoder in the encoding layer, and the image feature vector corresponding to the image result is obtained through the visual encoder in the encoding layer. Specifically, the text part uses the CLIP text encoder, and the image part uses the CLIP visual encoder. The text feature vector and image feature vector are input into the cross-modal attention layer to calculate the cross-modal attention matrix. Preferably, coarse-grained global alignment and fine-grained local alignment are set sequentially in the cross-modal attention layer to establish an accurate correspondence between token text words and corresponding images. Based on the cross-modal attention matrix, the most relevant position in the image is determined for the token text lexical, which is to find the maximum value in the row of the matrix. An inverse attention matrix is ​​constructed. In the anchor point recognition layer, the most relevant position is verified by the inverse attention matrix to ensure the bidirectional optimal matching relationship. After the bidirectional optimal matching relationship is verified, an alignment anchor point set is generated. Obtain the set of alignment anchor points, extract the local context features within the token word window of the preset number of words before and after the entity name, and the neighboring region of the most relevant position in the image within a preset range, and calculate the semantic consistency between the local context features and the neighboring region through cosine similarity. The attention intensity and semantic consistency in the cross-modal attention matrix are weighted and summed to generate a semantic matching score. In this embodiment of the invention, the weights of attention intensity and semantic consistency are 0.6 and 0.4, respectively. Attention intensity focuses on the original matching confidence, while semantic consistency focuses on context compatibility. The matching is evaluated to determine whether it conforms to the overall semantic logic. The weighted sum score is normalized by softmax, and finally a semantic matching score between [0,1] is output. The semantic matching score can intuitively evaluate the quality of knowledge graph construction. Figure 2 This is a schematic diagram of the cross-modal alignment framework according to an embodiment of the present invention, as shown below. Figure 2 As shown, the basic framework of the dual-tower cross-modal matching network is illustrated. The matching score head includes the anchor point recognition layer and the logic for semantic matching score calculation.

[0020] Preferably, for the original text in the input data containing the entity, a hierarchical semantic matching architecture is used for further verification. The word-level matcher uses a 256-dimensional cross-modal attention mechanism to specifically handle the fine-grained correspondence between words and local regions of the image; the phrase-level matcher uses a 512-dimensional feature space to identify the semantic association between phrase structure and object combination in the image; the sentence-level matcher is based on a 768-dimensional global feature representation to capture the high-level semantic matching between the complete sentence and the overall image content. Multi-scale semantic matching is fused using a two-layer fully connected network structure. The first layer compresses the features from the three scales into a 512-dimensional intermediate representation, which is then activated by ReLU and regularized with a random deactivation ratio of 0.1. The second layer maps to a single similarity score and normalizes it using the Sigmoid function to obtain an overall score that serves as the decision interface for subsequent practical tasks.

[0021] Similarity scores are obtained by comparing text feature vectors and image feature vectors between entities. The similarity scores are used to determine whether they have the same meaning. When the semantic matching scores of each entity exceed a preset threshold, homonymous entities are distinguished and homonymous entities are merged. A low semantic matching score can be indirectly considered as an unreliable result. The preset threshold is dynamically adjusted based on the ratio of the number of entity candidates to the maximum number of candidates. The maximum number of candidates refers to the number of semantics corresponding to the same entity in the current text. For highly ambiguous entities, the threshold is increased by 0.15 from the original threshold to improve strictness and avoid matching errors. For low-ambiguity entities, the threshold is decreased by 0.1 from the original threshold to relax the conditions and improve recall. Disambiguation processing further optimizes the graph structure, identifies and completes potential entity connections, and increases the graph density and quality.

[0022] The above technical solutions of the embodiments of the present invention will be illustrated with reference to the following accompanying drawings.

[0023] Figure 3 This is a schematic diagram of the self-booting execution architecture according to an embodiment of the present invention, as shown below. Figure 3 As shown, a self-guided knowledge graph construction framework based on a large model is demonstrated, including the construction of an open-domain dataset; the extraction of entities and relations based on soft-encoded prompts using a large model; the introduction of a joint processing mechanism for images and text; multimodal fusion and alignment; and finally, entity disambiguation and knowledge graph enhancement.

[0024] In summary, addressing the existing problems, this invention presents a self-guided knowledge graph construction method based on a large-scale model. Leveraging the open-domain language understanding capabilities of a large language model, it guides the model to generate entities and relations through soft-coded hints in templates, eliminating the need for manual ontology pre-definition and achieving low-cost, highly adaptable knowledge graph construction. Building upon the traditional MiPro V2 processing method, a three-layer iterative annotation strategy is introduced, with each layer undergoing independent template hint optimization, improving extraction accuracy and recall. A joint image and text processing mechanism is introduced, considering not only cross-modal attention between text terms and corresponding image regions but also overall contextual compatibility to capture higher-level features. Multimodal embedding alignment achieves precise entity disambiguation and enhanced representation, significantly improving the accuracy, richness, and scalability of knowledge graph construction.

[0025] Device Examples According to embodiments of the present invention, a self-guided knowledge graph construction device based on a large model is provided. Figure 4 This is a schematic diagram of a self-guided knowledge graph construction device based on a large model according to an embodiment of the present invention, such as... Figure 4 As shown, the self-guided knowledge graph construction device based on a large model according to an embodiment of the present invention specifically includes: Initialization module 40 is used to construct and preprocess a text dataset derived from the open domain to obtain input data, specifically for: Build a text dataset based on the current task; The text dataset is filtered for advertisements and deduplicated to obtain the input data.

[0026] The iterative annotation module 42 is used to generate candidate suggestion templates using a suggestion word self-optimization model, form a general suggestion template based on feedback results, and use the general suggestion template as input to the iterative annotation strategy. The final suggestion template is used to guide the large language model in constructing the first graph from the entities and relationships extracted from the input data. Specifically, it is used for: Candidate suggestion templates are generated using the MiPro V2 self-optimization model for suggestion words in the DSPy framework; Entities and relationships are extracted from the input data using candidate suggestion templates. The extraction performance of different candidate suggestion templates is evaluated. A weighted mixing strategy is formed based on the extraction performance. The candidate suggestion templates are then merged using the weighted mixing strategy to obtain a general suggestion template.

[0027] Optimize the generic suggestion template using an iterative annotation strategy that includes a pre-annotation layer, a re-annotation layer, and a self-reflection layer: In the pre-labeled layer, a general prompt template is used for extraction to obtain the first extraction result; In the re-annotation layer, the error and omission information identified in the first extraction result is fed back to the prompt word self-optimization model, and the re-annotation layer template is reconstructed and the second extraction result is output. In the self-reflection layer, metacognitive cue words are constructed based on the second extraction results to form the final layer cue template. The final layer cue template is used to correct errors before the large language model outputs, resulting in the third extraction results and forming the first graph.

[0028] Entity linking module 44 is used to query entities in a pre-acquired standard entity library, remove entities that are not found or have abstract semantics to obtain an entity pool, call the image search interface to query a preset number of image results for each entity in the entity pool, and structurally associate the image results with the corresponding entities in the first atlas to obtain a second atlas. Specifically, it is used for: Retrieves an image containing the image URL, thumbnail, and original image path; By structurally associating the image URL with the corresponding entity in the first graph, a second graph containing JSON structured information is obtained.

[0029] Alignment and disambiguation module 46 is used to fuse and align entities in the second atlas with the image results using a dual-tower cross-modal matching network, generate semantic matching scores, and perform entity disambiguation based on the semantic matching scores to obtain the final atlas. Specifically, it is used for: The entity names and image results of the second graph are input into the dual-tower cross-modal matching network. The image results refer to the image itself located by the image URL. The dual-tower cross-modal matching network includes an encoding layer, a cross-modal attention layer, and an anchor recognition layer. The text feature vector corresponding to the entity is obtained by the text encoder in the encoding layer, and the image feature vector corresponding to the image result is obtained by the visual encoder in the encoding layer. The text feature vector and the image feature vector are input into the cross-modal attention layer to calculate the cross-modal attention matrix. Based on the cross-modal attention matrix, the most relevant position in the image is determined for the token text lexical. An inverse attention matrix is ​​constructed, and in the anchor point recognition layer, the most relevant position is verified by the inverse attention matrix to generate an alignment anchor point set. Obtain the set of alignment anchor points, extract the local context features within the token word window of the preset number of words before and after the entity name, and the neighboring region of the most relevant position in the image within a preset range, and calculate the semantic consistency between the local context features and the neighboring region through cosine similarity. The semantic matching score is generated by weighted summation of attention intensity and semantic consistency in the cross-modal attention matrix.

[0030] Similarity scores are obtained by comparing text feature vectors and image feature vectors between entities. The similarity scores are used to determine whether they have the same meaning. When the semantic matching scores of each entity exceed a preset threshold, entities with the same name but different meanings are distinguished and entities with different names but the same meanings are merged. The preset threshold is dynamically adjusted based on the number of entity candidates and the proportion of the maximum number of candidates.

[0031] In summary, addressing the existing problems, this invention presents a self-guided knowledge graph construction device based on a large-scale model. Leveraging the open-domain language understanding capabilities of a large language model, it guides the model to generate entities and relations through soft-coded prompts in templates, eliminating the need for manual ontology pre-definition and achieving low-cost, highly adaptable knowledge graph construction. Building upon the traditional MiPro V2 processing method, a three-layer iterative annotation strategy is introduced, with each layer undergoing independent template prompt optimization, improving extraction accuracy and recall. A joint image and text processing mechanism is introduced, considering not only cross-modal attention between text terms and corresponding image regions but also overall contextual compatibility to capture higher-level features. Multimodal embedding alignment achieves precise entity disambiguation and enhanced representation, significantly improving the accuracy, richness, and scalability of knowledge graph construction.

[0032] Electronic device examples Figure 5 This is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device 500 may include at least one processor 510 and a memory 520. The processor 510 can execute instructions stored in the memory 520. The processor 510 is communicatively connected to the memory 520 via a data bus. In addition to the memory 520, the processor 510 can also be communicatively connected to an input device 530, an output device 540, and a communication device 550 via the data bus.

[0033] Processor 510 can be any conventional processor, such as a commercially available CPU. Processors may also include graphics processing units (GPUs), field-programmable gate arrays (FPGAs), systems-on-chips (SoCs), application-specific integrated circuits (ASICs), or combinations thereof.

[0034] The memory 520 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0035] In this embodiment of the disclosure, the memory 520 stores executable instructions, and the processor 510 can read the executable instructions from the memory 520 and execute the instructions to implement all or part of the steps of any of the self-guided knowledge graph construction methods based on large models in the exemplary embodiments described above.

[0036] Computer-readable storage medium embodiments In addition to the methods and apparatus described above, exemplary embodiments of this disclosure may also be a computer program product or a computer-readable storage medium storing the computer program product, the computer product including computer program instructions that can be executed by a processor to implement all or part of the steps described in any of the large-model-based self-guided knowledge graph construction methods in the exemplary embodiments described above.

[0037] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. Programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages, and scripting languages ​​(e.g., Python). The program code can be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0038] Computer-readable storage media may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: static random access memory (SRAM) having one or more electrically connected wires; electrically erasable programmable read-only memory (EEPROM); erasable programmable read-only memory (EPROM); programmable read-only memory (PROM); read-only memory (ROM); magnetic storage; flash memory; magnetic disk or optical disk; or any suitable combination thereof.

[0039] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A self-guided knowledge graph construction method based on a large model, characterized in that, include: We construct and preprocess a text dataset derived from the open domain to obtain the input data; Candidate suggestion templates are generated using a suggestion word self-optimization model. A general suggestion template is formed based on feedback results. This general suggestion template is used as input to an iterative annotation strategy. The final suggestion template guides the large language model to construct the first graph from the entities and relationships extracted from the input data. Specifically, this includes: Optimize the generic suggestion template using an iterative annotation strategy that includes a pre-annotation layer, a re-annotation layer, and a self-reflection layer: In the pre-annotation layer, the general prompt template is used for extraction to obtain the first extraction result; In the re-annotation layer, the error information and omission information identified in the first extraction result are fed back to the prompt word self-optimization model, and the re-annotation layer template is reconstructed and the second extraction result is output. In the self-reflection layer, metacognitive prompt words are constructed based on the second extraction result to form the final prompt template. The final prompt template is used to correct errors before the output of the large language model to obtain the third extraction result and form the first graph. The entity is queried in the pre-acquired standard entity library. Entities that are not found or have abstract semantics are removed to obtain an entity pool. The image search interface is called to query a preset number of image results corresponding to each entity in the entity pool. The image results are then structurally associated with the corresponding entities in the first map to obtain the second map. A dual-tower cross-modal matching network is used to fuse and align entities and image results in the second atlas, generating semantic matching scores. Entity disambiguation is then performed based on these semantic matching scores to obtain the final atlas. Specifically, this includes: When calculating the semantic matching score, local context features within a preset number of token word windows before and after the entity name in the sentence are extracted, as well as the neighboring regions within a preset range of the most relevant position in the image. The semantic consistency between the local context features and the neighboring regions is calculated using cosine similarity.

2. The method according to claim 1, characterized in that, The construction of the text dataset derived from the open domain and its preprocessing to obtain the input data specifically includes: Construct the text dataset based on the current task; The text dataset is filtered for advertisements and deduplicated to obtain the input data.

3. The method according to claim 1, characterized in that, The process of generating candidate prompt templates using a prompt word self-optimization model and forming a general prompt template based on feedback results specifically includes: Candidate suggestion templates are generated using the MiPro V2 self-optimization model for suggestion words in the DSPy framework; The candidate suggestion templates are used to extract entities and relationships from the input data. The extraction effect of different candidate suggestion templates is evaluated. A weighted mixing strategy is formed based on the extraction effect. The candidate suggestion templates are then fused using the weighted mixing strategy to obtain a general suggestion template.

4. The method according to claim 1, characterized in that, The step of structurally associating the image result with the corresponding entity in the first atlas to obtain the second atlas specifically includes: Retrieves an image containing the image URL, thumbnail, and original image path; The image URL is structurally associated with the corresponding entity in the first graph to obtain a second graph containing JSON structured information.

5. The method according to claim 1, characterized in that, The step of using a dual-tower cross-modal matching network to fuse and align entities in the second atlas with the image results to generate a semantic matching score specifically includes: The entity names and image results of the second map are input into a dual-tower cross-modal matching network, wherein the dual-tower cross-modal matching network includes an encoding layer, a cross-modal attention layer, and an anchor point recognition layer; The text feature vector corresponding to the entity is obtained through the text encoder in the encoding layer, and the image feature vector corresponding to the image result is obtained through the visual encoder in the encoding layer. The text feature vector and the image feature vector are input into the cross-modal attention layer to calculate the cross-modal attention matrix. Based on the cross-modal attention matrix, the most relevant position in the image is determined for the token text lexical units. An inverse attention matrix is ​​constructed. In the anchor point recognition layer, the most relevant position is verified by the inverse attention matrix, and an alignment anchor point set is generated. Obtain the set of alignment anchor points, extract the local context features within a preset number of token word windows before and after the statement where the entity name is located, and the neighboring region of the most relevant position in the image within a preset range, and calculate the semantic consistency between the local context features and the neighboring region through cosine similarity. The semantic matching score is generated by weighting and summing the attention intensity in the cross-modal attention matrix with the semantic consistency.

6. The method according to claim 1, characterized in that, The step of obtaining the final graph by entity disambiguation based on the semantic matching score specifically includes: Similarity scores are obtained by comparing text feature vector similarity and image feature vector similarity between entities. The similarity scores are used to determine whether they have the same meaning. When the semantic matching scores of each entity exceed a preset threshold, entities with the same name but different meanings are distinguished and merged. The preset threshold is dynamically adjusted by the number of entity candidates and the proportion of the maximum number of candidates.

7. A self-guided knowledge graph construction device based on a large model, characterized in that, include: The initialization module is used to construct and preprocess a text dataset derived from the open domain to obtain input data; The iterative annotation module is used to generate candidate suggestion templates using a suggestion word self-optimization model, form a general suggestion template based on feedback results, and use the general suggestion template as input to the iterative annotation strategy. The final suggestion template is then used to guide the large language model in constructing the first graph from the entities and relationships extracted from the input data. Specifically, it is used for: Optimize the generic suggestion template using an iterative annotation strategy that includes a pre-annotation layer, a re-annotation layer, and a self-reflection layer: In the pre-annotation layer, the general prompt template is used for extraction to obtain the first extraction result; In the re-annotation layer, the error information and omission information identified in the first extraction result are fed back to the prompt word self-optimization model, and the re-annotation layer template is reconstructed and the second extraction result is output. In the self-reflection layer, metacognitive prompt words are constructed based on the second extraction result to form the final prompt template. The final prompt template is used to correct errors before the output of the large language model to obtain the third extraction result and form the first graph. The entity linking module is used to query the entities in the pre-acquired standard entity library, remove entities that are not found or have abstract semantics to obtain an entity pool, call the image search interface to query a preset number of image results corresponding to each entity in the entity pool, and structurally associate the image results with the corresponding entities in the first graph to obtain a second graph. The alignment and disambiguation module is used to fuse and align entities and image results of the second atlas using a dual-tower cross-modal matching network, generate semantic matching scores, and perform entity disambiguation based on the semantic matching scores to obtain the final atlas. Specifically, it is used for: When calculating the semantic matching score, local context features within a preset number of token word windows before and after the entity name in the sentence are extracted, as well as the neighboring regions within a preset range of the most relevant position in the image. The semantic consistency between the local context features and the neighboring regions is calculated using cosine similarity.

8. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the self-guided knowledge graph construction method based on a large model as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an implementation program for information transmission, which, when executed by a processor, implements the steps of the self-guided knowledge graph construction method based on a large model as described in any one of claims 1 to 6.

Images