Retrieval method and device and related equipment

Abstract

The invention provides a retrieval method and device and related equipment, and relates to the technical field of retrieval. The method comprises the steps of receiving a question of a user, and determining a first entity from the question based on a knowledge graph; determining a second entity and an additional prompt based on the question, the knowledge graph and the first entity; retrieving a first text from a text library based on the second entity and the additional prompt, and judging whether the first text can answer the question or not by utilizing a large language model; if yes, feeding back the first text as an answer to the question to the user; and if not, optimizing a historical clue by using a large language model, determining a new first entity according to the historical clue, and repeatedly executing the above steps until an answer is obtained, the historical clue including all determined entities. By means of the technical means, the problems that in the prior art, a retrieval path optimization mechanism is lacked, and the retrieval result is inaccurate are solved.

Description

Technical Field

[0001] This disclosure relates to the field of retrieval technology, and in particular to a retrieval method, apparatus and related equipment. Background Technology

[0002] In existing technologies, knowledge graph-based retrieval methods typically rely on single entity matching and static retrieval strategies. If the knowledge graph is incomplete or lacks coverage, it becomes difficult to accurately link questions and answers. Especially when dealing with complex, multi-hop, or ambiguous questions, performing a single retrieval based solely on the initial entity can easily lead to irrelevant or incomplete recall results due to insufficient information. Furthermore, the lack of dynamic feedback and iterative optimization mechanisms for retrieval failures prevents the effective use of historical reasoning clues to correct the retrieval path, thus limiting the accuracy and robustness of the retrieval process. Summary of the Invention

[0003] This disclosure provides a retrieval method, apparatus, and related equipment to improve the accuracy and robustness of retrieval.

[0004] According to one aspect of this disclosure, a retrieval method is provided, comprising: receiving a user's question; identifying a first entity from the question based on a knowledge graph; identifying a second entity and additional hints based on the question, the knowledge graph, and the first entity; retrieving a first text from a text library based on the second entity and the additional hints; determining whether the first text can answer the question using a large language model; if it can, feeding the first text back to the user as the answer to the question; if it cannot, optimizing historical clues using a large language model, identifying a new first entity according to the historical clues, and repeating the above steps until an answer is obtained, wherein the historical clues include all identified entities.

[0005] In one embodiment, determining a first entity from a question based on a knowledge graph includes: extracting one or more third entities from the question; if multiple third entities exist, determining the centrality of each third entity in the knowledge graph, and taking the third entity with the highest centrality as the first entity; if only one third entity exists, taking that third entity as the first entity.

[0006] In one embodiment, after identifying the first entity from the question based on the knowledge graph, the method further includes: retrieving a second text from a text library based on the first entity, and using a large language model to determine whether the second text can answer the question; if it can, then feeding the second text back to the user as the answer; if it cannot, then determining the second entity and additional hints based on the question, the knowledge graph, and the first entity.

[0007] In one embodiment, determining a second entity and additional hints based on a question, a knowledge graph, and a first entity includes: determining a first relation edge of the first entity from the knowledge graph based on the question; determining a third entity from the knowledge graph based on the first relation edge; determining whether the first entity or the third entity is the second entity based on the question; and generating additional hints based on the triples of the first entity, the first relation edge, and the third entity.

[0008] In one embodiment, determining the first relation edge of the first entity from the knowledge graph based on the question includes: determining one or more second relation edges of the first entity from the knowledge graph; if there are multiple second relation edges, determining the relevance score of each second relation edge to the question, and taking the second relation edge with the highest relevance score as the first relation edge; if there is only one second relation edge, taking that second relation edge as the first relation edge.

[0009] In one embodiment, determining whether to use a first entity or a third entity as a second entity based on the question includes: determining the relevance scores of the first entity and the third entity to the question; and using the entity with the higher relevance score from the first entity and the third entity as the second entity.

[0010] In one embodiment, the method further includes providing the user with triples, historical clues, and answers.

[0011] In one embodiment, the method further includes: receiving a question, determining a first entity from the question based on a knowledge graph, and executing a loop: determining whether i equals N, where i is the number of loop executions, i is initially 0, N is the maximum depth of the loop, and N is a positive integer; if i equals N, the first text retrieved in the (i-1)th loop is fed back to the user as the answer, and the loop exits; if i is less than N, a second entity and additional hints are determined based on the question, the knowledge graph, and the first entity; the first text is retrieved from the text library based on the second entity and additional hints, and a large language model is used to determine whether the first text can answer the question; if it can, the first text is fed back to the user as the answer, and the loop exits; if it cannot, the large language model is used to optimize historical clues, a new first entity is determined according to the historical clues, and i is updated with the value of i+1.

[0012] According to another aspect of this disclosure, a retrieval apparatus is provided, comprising: a determining unit configured to receive a user's question and determine a first entity from the question based on a knowledge graph; a generating unit configured to determine a second entity and additional hints based on the question, the knowledge graph, and the first entity; a judging unit configured to retrieve a first text from a text library based on the second entity and the additional hints, and use a large language model to judge whether the first text can answer the question; a feedback unit configured to, if yes, provide the first text as the answer to the question to the user; and an optimization unit configured to, if no, optimize historical clues using a large language model, determine a new first entity according to the historical clues, and repeat the above steps until an answer is obtained, wherein the historical clues include all determined entities.

[0013] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any of the methods described above by executing the executable instructions.

[0014] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements any of the methods described above.

[0015] According to another aspect of this disclosure, a computer program product is provided, including computer instructions stored in a computer-readable storage medium, which, when executed by a processor, implement operation instructions for any of the methods described above.

[0016] In the embodiments of this disclosure, an initial entity is identified from the user's question, and multi-hop reasoning and text retrieval are performed by combining a knowledge graph and additional hints. If the retrieval fails, a new retrieval starting point is generated by optimizing the large language model based on historical clues. This process is repeated until an accurate answer is obtained. This solves the problems of lack of retrieval path optimization mechanism and inaccurate retrieval results in related technologies, thereby improving the accuracy and robustness of retrieval.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0019] Figure 1A schematic diagram of a retrieval system according to an embodiment of this disclosure is shown.

[0020] Figure 2 A flowchart of a retrieval method according to an embodiment of this disclosure is shown.

[0021] Figure 3 A flowchart illustrating another retrieval method in an embodiment of this disclosure is shown.

[0022] Figure 4 A flowchart of a method for generating additional search tips according to an embodiment of this disclosure is shown.

[0023] Figure 5 A flowchart of a method for determining relation edges in an embodiment of this disclosure is shown.

[0024] Figure 6 A flowchart of another retrieval method in an embodiment of this disclosure is shown.

[0025] Figure 7 A schematic diagram of a retrieval device according to an embodiment of the present disclosure is shown.

[0026] Figure 8 A schematic diagram of an electronic device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0027] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0028] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0029] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0030] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0031] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0032] It should be noted that, unless otherwise specified, the embodiments of this disclosure and the technical features thereof can be combined with each other.

[0033] To facilitate understanding, the following is an explanation of several terms used in this disclosure: KGE (Knowledge Graph Embedding): A technique that maps entities and relations in a knowledge graph to a low-dimensional continuous vector space to support reasoning, link prediction, and knowledge discovery.

[0034] PSR (Product-Service-Resource): A four-layer modeling framework for communication networks, consisting of four layers from bottom to top: resources, network services, business services, and products. It is used to structurally describe the mapping relationship between network capabilities and business services.

[0035] RES (Resource): The underlying infrastructure unit in a communication network, including atomic-level capability components such as physical devices, virtualization instances, and configuration templates.

[0036] RFS (Resource Facing Service): A service unit with specific network functions, assembled from one or more resources, intended for internal orchestration systems and not sold directly to external parties.

[0037] CFS (Customer Facing Service): A service consisting of one or more network services, with a clear business semantics, and which can be subscribed to or billed by customers.

[0038] PRODUCT: A market-oriented commercial package that typically includes multiple business services and is accompanied by commercial attributes such as pricing, contracts, and branding.

[0039] GCA (Graph Centrality Algorithm): A class of graph theory methods used to measure the importance or hub status of nodes in a knowledge graph. By calculating indicators such as the number of connections and path positions of nodes, it identifies entities that occupy a core position in the topology and is often used to determine the starting point of inference in multi-entity problems.

[0040] RAG (Retrieval-Augmented Generation): A technical framework that combines information retrieval with the generation capabilities of large language models. It assists language models in generating more accurate and factually consistent answers by retrieving relevant documents from external knowledge bases as context.

[0041] PSR-KG (PSR Knowledge Graph): A structured knowledge base built on the PSR (Product-Service-Resource) hierarchical model, which explicitly depicts the hierarchical dependencies and networking relationships between products, network services and underlying resources, and is used to support intelligent question answering and reasoning in telecommunications services.

[0042] LLM (Large Language Model): An artificial intelligence language model trained with large-scale parameters and massive amounts of text. It has the ability to understand natural language, generate coherent text, and perform a certain degree of logical reasoning. It is often used in question answering, summarizing, and dialogue systems.

[0043] VPN (Virtual Private Network): A technology that builds private communication channels on public networks. It achieves secure isolation and end-to-end transmission of user data through encryption and tunneling mechanisms, and is widely used in enterprise leased lines and remote access scenarios.

[0044] PON (Passive Optical Network): A fiber optic access technology that uses a point-to-multipoint topology to achieve broadband connections from the central office to multiple end users through passive optical splitters. It is currently the mainstream fiber-to-the-home solution.

[0045] Out-degree centrality: In a directed graph, it refers to the number of edges a node radiates outwards, used to measure the influence of the node as an "initiator" or "upstream"; in PSR-KG, it is often used to evaluate the number of underlying resources directly contained in or controlled by a service entity.

[0046] A-Leaf (Access Leaf): A switching device located at the access layer in the new metropolitan area network architecture, responsible for the aggregation and edge processing of user traffic. It usually works with BNG devices to implement Layer 3 VPN access and VLAN termination functions.

[0047] VXLAN tunnel (Virtual Extensible LAN Tunnel): A network virtualization technology based on UDP encapsulation. It establishes an overlay tunnel over an IP network to achieve high Layer 2 connectivity across physical networks, supporting large-scale tenant isolation and flexible networking.

[0048] VLAN (Virtual Local Area Network): A technology that divides a physical network into logical broadcast domains, using tags to isolate traffic from different services or users, thereby improving network security and management flexibility.

[0049] EBGP (External Border Gateway Protocol) Neighbors: BGP peer relationships that run between different autonomous systems. They are used to exchange routing information between metropolitan area networks and backbone networks, or between devices from different vendors, to achieve cross-domain Layer 3 interconnection.

[0050] BNG (Broadband Network Gateway): A core network element located at the edge of a metropolitan area network, responsible for user authentication, IP address allocation, QoS policy enforcement, and VPN route termination. It is a key exit gateway for home and enterprise broadband services.

[0051] BGP (Border Gateway Protocol) is an external gateway protocol used to exchange routing information between different autonomous systems. It is a core routing protocol of the Internet and is responsible for implementing reachability announcements and path selection between large-scale networks.

[0052] IP (Internet Protocol): A fundamental communication protocol that implements packet addressing and routing at the network layer. It defines how data is transmitted from a source address to a destination address and is a core component of the TCP / IP protocol suite, supporting global Internet communication.

[0053] TCP (Transmission Control Protocol) is a connection-oriented, reliable transport layer communication protocol that uses a three-way handshake to establish a connection, acknowledgement, retransmission mechanism, and flow control to ensure that data is transmitted in order and without errors in unreliable networks. It is widely used in scenarios with high reliability requirements, such as web browsing, file transfer, and email.

[0054] QoS (Quality of Service) policy: A set of rules and mechanisms for managing and regulating network traffic. Through priority scheduling, bandwidth allocation, packet loss control, and other means, it ensures that critical services (such as voice, video, or enterprise leased lines) can still obtain the expected quality of service when the network is congested.

[0055] Figure 1 The diagram shows a retrieval system according to an embodiment of the present disclosure. The retrieval system includes: a data collection module 101, a processing module 102, and a large language model 103.

[0056] The receiving module 101 is used to receive user questions.

[0057] The processing module 102 can process the problem received by the receiving module 101. During the processing, the processing module 102 can call the large language model 103.

[0058] The processing module 102 may be equipped with an application program to perform the following actions: identifying a first entity from the question based on a knowledge graph; identifying a second entity and additional hints based on the question, the knowledge graph, and the first entity; retrieving a first text from a text library based on the second entity and additional hints, and using a large language model to determine whether the first text can answer the question; if it can, feeding the first text back to the user as the answer to the question; if it cannot, optimizing historical clues using a large language model, identifying a new first entity based on the historical clues, and repeating the above steps until an answer is obtained, wherein the historical clues include all identified entities.

[0059] Figure 2 A flowchart of a retrieval method according to an embodiment of this disclosure is shown. The method is as follows: Figure 2 As shown, it includes the following steps: S201, Receive the user's question, and determine the first entity from the question based on the knowledge graph.

[0060] As an example, the user's question is a natural language question derived from the user's input.

[0061] As an example, a knowledge graph is pre-generated, structured data containing entities and the relationships between them. Knowledge graphs can be generated using text libraries.

[0062] As an example, a text library is a database that stores a large amount of information and is used for retrieval.

[0063] As an example, the first entity is the third entity with the highest centrality in the knowledge graph.

[0064] As an example, the third entity is a candidate entity extracted directly from the user's question, and there may be one or more of them.

[0065] S202, based on the question, knowledge graph, and first entity, determine the second entity and additional hints.

[0066] As an example, the second entity is the entity obtained through further reasoning for retrieval, and can be either the first entity or the third entity.

[0067] As an example, additional hints are generated based on triples consisting of the first entity, the first relation edge, and the third entity, to enhance the relevance of text retrieval.

[0068] As an example, the first relation edge is the second relation edge with the highest relevance score to the problem.

[0069] As an example, the second relation edge is all candidate relation edges in the knowledge graph that are directly connected to the first entity.

[0070] S203: Retrieve the first text from the text library based on the second entity and additional prompts, and use a large language model to determine whether the first text can answer the question.

[0071] As an example, the first text is the candidate answer text retrieved from the text library based on the second entity and additional hints.

[0072] S204, if possible, the first text will be returned to the user as the answer to the question.

[0073] S205. If not, optimize the historical clues using a large language model, determine the new first entity according to the historical clues, and repeat the above steps until the answer is obtained. The historical clues include all the determined entities.

[0074] In this embodiment, after receiving a user's question, a first entity is identified from the question based on a knowledge graph. Then, a second entity and additional hints are inferred by combining the question, the knowledge graph, and the first entity, and a first text is retrieved from the text library accordingly. A large language model is used to determine whether the first text is a valid answer. If not, the first entity is optimized and re-identified based on historical clues, and the above process is repeated until a first text that can answer the question is obtained or the maximum loop depth is reached. Through the above technical means, the problems of lacking a retrieval path optimization mechanism and inaccurate retrieval results in related technologies are solved, thereby improving the accuracy and robustness of the retrieval.

[0075] In one embodiment, determining a first entity from a question based on a knowledge graph includes: extracting one or more third entities from the question; if multiple third entities exist, determining the centrality of each third entity in the knowledge graph, and taking the third entity with the highest centrality as the first entity; if only one third entity exists, taking that third entity as the first entity.

[0076] As an example, centrality is a metric used to measure how closely an entity is connected to other entities in a knowledge graph. The higher the centrality, the stronger the structural importance of that entity in the knowledge graph.

[0077] In this embodiment, one or more third entities are extracted from the user's question. Based on the centrality of these third entities in the knowledge graph, the entity with the highest centrality is selected as the first entity. This ensures that the initial retrieval starting point has strong semantic representativeness and graph coverage. Through these technical means, subsequent retrieval paths are avoided from deviating from the question's intent due to improper initial entity selection, thereby improving the relevance of the first-round retrieval and the overall question-answering efficiency.

[0078] In one optional embodiment, determining the first entity from the question based on a knowledge graph includes: extracting one or more third entities from the question; if multiple third entities exist, calculating the semantic similarity between each third entity and the question, and selecting the third entity with the highest semantic similarity as the first entity; if only one third entity exists, selecting that third entity as the first entity. Through these technical means, the selection of the first entity is made more closely aligned with the semantic focus of the user's question, thereby enhancing the consistency between the retrieval path and the user's intent.

[0079] Figure 3 A flowchart of another retrieval method in an embodiment of this disclosure is shown, the method as follows: Figure 3 As shown, it includes the following steps: S301, based on the first entity, retrieve the second text from the text library, and use the large language model to determine whether the second text can answer the question; S302, if possible, then provide the second text as the answer to the user; S303, if not, determine the second entity and additional hints based on the question, knowledge graph and first entity.

[0080] As an example, the second text is a candidate answer text obtained directly from the text library based on the first entity, including multiple text fragments in the text library that are most relevant to the first entity in terms of relevance score.

[0081] In this embodiment, after identifying the first entity, a second text is first retrieved from the text library using the first entity, and then the large language model is used to determine whether it is sufficient to answer the user's question. If the determination is yes, the second text is directly returned as the answer, avoiding unnecessary multi-hop reasoning. If the determination is no, the relationship expansion based on the knowledge graph is continued to identify the second entity and additional hints, entering a deeper level of the retrieval process. Through the above technical means, unnecessary computational overhead is reduced while ensuring the quality of the answer, thereby improving the overall retrieval efficiency.

[0082] In an optional embodiment, after identifying the first entity from the question based on the knowledge graph, the method further includes: generating multiple candidate entity pairs based on the first entity and its set of adjacent entities in the knowledge graph; calculating the joint relevance score of each candidate entity pair in conjunction with the question; selecting one entity from the candidate entity pair with the highest score as the second entity; and generating additional hints based on the semantics of the candidate entity pair and the question; subsequently, retrieving the first text from the text library based on the second entity and the additional hints; and using a large language model to determine whether the first text can answer the question. Through these technical means, the ability to capture implicit semantic relationships in complex questions is enhanced, thereby improving the accuracy of multi-hop retrieval.

[0083] Figure 4 This diagram illustrates a flowchart of a method for generating additional search tips according to an embodiment of the present disclosure. The method is as follows: Figure 4 As shown, it includes the following steps: S401, Based on the question, determine the first relation edge of the first entity from the knowledge graph; S402, determine the third entity from the knowledge graph based on the first relation edge, and determine whether the first entity or the third entity is the second entity based on the question; S403, generate additional hints based on the triples of the first entity, the first relation edge, and the third entity.

[0084] As an example, the first relation edge is the one in the knowledge graph that is connected to the first entity and is most relevant to the semantics of the user's question.

[0085] As an example, a triple is a basic unit of a knowledge graph, consisting of a first entity, a first relation edge, and a third entity, used to express structured semantic information.

[0086] In this embodiment, based on the user's question, a first relation edge with the highest semantic match to the question and related to the first entity is selected from the knowledge graph, and the associated third entity is located through this relation edge. Subsequently, based on the semantics of the question, it is determined whether the first entity or the third entity should be used as the second entity. At the same time, additional hints are generated using a triple consisting of the first entity, the first relation edge, and the third entity to assist the subsequent text retrieval process. Through the above technical means, the additional hints have clear structured semantic basis, thereby improving the directionality and accuracy of text retrieval.

[0087] In one optional embodiment, determining the second entity and additional hints based on the question, knowledge graph, and first entity includes: semantically matching all adjacency relationships of the first entity in the knowledge graph based on the question to obtain multiple candidate relationship edges; expanding each candidate relationship edge to generate a corresponding fourth entity, forming multiple triples; calculating the overall semantic relevance score of each triple in conjunction with the question; selecting the triple with the highest score and using its fourth entity as the second entity; and generating additional hints based on the content of the triple. Through the above technical means, the multi-path semantic scoring mechanism enhances the coverage of potential answer paths, thereby improving retrieval robustness in complex question scenarios.

[0088] Figure 5 This diagram illustrates a flowchart of a method for determining relation edges according to an embodiment of the present disclosure. The method is as follows: Figure 5 As shown, it includes the following steps: S501, Identify one or more second relation edges of the first entity from the knowledge graph; S502, if there are multiple second relation edges, determine the relevance score of each second relation edge to the problem, and take the second relation edge with the highest relevance score as the first relation edge; S503: If there is only one second relation edge, then treat the second relation edge as the first relation edge.

[0089] In this embodiment, firstly, all second relation edges connecting the first entity in the knowledge graph are obtained. When multiple second relation edges exist, the relevance score of each edge to the user's question is calculated, and the edge with the highest score is selected as the first relation edge. If there is only one second relation edge, it is directly used as the first relation edge for subsequent determination of the third entity and generation of additional hints. Through the above technical means, it is ensured that the selected relation edges closely match the semantic intent of the user's question, thereby improving the accuracy of the second entity and additional hints.

[0090] In one optional embodiment, determining the first relation edge of the first entity from the knowledge graph based on the question includes: aligning all second relation edges of the first entity with the question through vector embedding to obtain semantic matching vectors for each second relation edge; calculating the context attention weight of each second relation edge based on the semantic matching vectors; and reordering the second relation edges according to the context attention weights, selecting the second relation edge with the highest weight as the first relation edge. By employing the above techniques, a context-aware attention mechanism is introduced to dynamically weight relation edges, thereby enhancing the adaptability of relation selection to complex question contexts.

[0091] In one embodiment, determining whether to use a first entity or a third entity as a second entity based on the question includes: determining the relevance scores of the first entity and the third entity to the question; and using the entity with the higher relevance score from the first entity and the third entity as the second entity.

[0092] As an example, relevance scores are used to quantify how well an entity matches a user's question at the semantic level; relevance scores can be similarity scores.

[0093] In this embodiment, relevance scores between the first entity and the third entity and the user's question are calculated respectively, and the entity with the higher score is selected as the second entity to ensure that the entity used for subsequent text retrieval is closest to the core semantics of the question. This technique avoids semantic shifts caused by fixed selection of related entities, thereby improving the accuracy of the retrieval target.

[0094] In one optional embodiment, determining whether to use a first entity or a third entity as the second entity based on the question includes: concatenating the first entity and the third entity with the question to form two candidate input sequences; inputting the two candidate input sequences into a large language model to obtain their corresponding semantic consistency probabilities; and selecting the entity with the higher semantic consistency probability as the second entity. Through the above techniques, the deep semantic understanding capability of the large language model is used to dynamically evaluate the fit between the entity and the question, thereby improving the rationality of the second entity selection.

[0095] In one embodiment, the method further includes providing the user with triples, historical clues, and answers.

[0096] In this embodiment, after generating the final answer, the triples used for reasoning, historical clues containing all explored entities, and the answer are all fed back to the user to provide interpretable question-and-answer evidence and a complete search trajectory. Through these technical means, the transparency and traceability of the question-and-answer process are enhanced, enabling users to understand the source of the answer and the reasoning path.

[0097] In an optional embodiment, the method further includes: generating explanatory information based on triples and historical clues using a large language model before returning the answer to the user; and returning the explanatory information to the user along with the answer, wherein the explanatory information includes the key entities involved, relational paths, and excluded candidate paths. Through the above technical means, the efficiency of users' understanding of complex multi-hop reasoning results is improved, thereby enhancing the trust level of human-computer interaction.

[0098] Figure 6 This invention discloses a flowchart of another retrieval method in an embodiment of the present disclosure, the method being as follows: Figure 6 As shown, it includes the following steps: S601, Receive the question, determine the first entity from the question based on the knowledge graph, and execute the next loop: S602, determine whether i is equal to N, where i is the number of times the loop is executed, the initial value of i is 0, N is the maximum depth of the loop, and N is a positive integer; S603, if i equals N, then the first text retrieved in the (i-1)th iteration is returned to the user as the answer, and the loop exits; S604, if i is less than N, determine the second entity and additional hints based on the question, knowledge graph and first entity; S605: Retrieve the first text from the text library based on the second entity and additional prompts, and use a large language model to determine whether the first text can answer the question; S606, If possible, return the first text as the answer to the user and exit the loop; S607, if not, then use the large language model to optimize historical clues, determine the new first entity according to the historical clues, and update i with the value of i+1.

[0099] As an example, the number of times the loop is executed is a counter variable used to record the current iteration round. It is initially set to 0 and increments by 1 after each retrieval and judgment process is completed.

[0100] As an example, the maximum depth of the loop is a preset positive integer N, which limits the maximum number of rounds in the entire iterative retrieval process to prevent infinite loops or over-reasoning.

[0101] In this embodiment, after receiving the user's question and determining the initial first entity, a loop is entered: In each loop, a second entity and additional hints are generated based on the current first entity, the first text is retrieved, and its validity is verified by a large language model; if the verification passes, the answer is returned and the loop terminates; if it fails, the historical clues are optimized using the large language model, and the first entity is updated accordingly, continuing the next round of retrieval; when the number of loops reaches the maximum depth N, even if the ideal answer is not found, the first text retrieved in the previous round is returned as the answer. Through the above technical means, while ensuring retrieval efficiency, the unsolvable state caused by the incomplete knowledge graph is avoided, thereby improving the robustness of the system in open-domain question answering.

[0102] In an optional embodiment, the method further includes: receiving a question; determining a first entity from the question based on a knowledge graph; and executing a loop: determining whether i equals N, where i is the number of loop executions, i is initially 0, N is the maximum depth of the loop, and N is a positive integer; if i equals N, then based on all retrieved first texts in the historical clues, using a large language model for fusion and reordering, selecting the first text with the highest overall confidence as the answer to be fed back to the user, and exiting the loop; if i is less than N, determining a second entity and additional hints based on the question, knowledge graph, and first entity; retrieving the first text from the text library based on the second entity and additional hints, and using a large language model to determine whether the first text can answer the question; if it can, then feeding back the first text as the answer to the user, and exiting the loop; if it cannot, then using a large language model to semantically expand the historical clues, introducing potential related entities to generate a new set of candidate first entities, selecting the entity with the highest relevance to the question as the new first entity, and updating i with the value of i+1. By employing the aforementioned technical means, the quality of the fallback answer is improved through multi-result fusion when the maximum depth is reached. At the same time, a semantic expansion mechanism is introduced during the iteration process to enhance the path exploration capability, thereby improving the overall accuracy of answers to complex questions.

[0103] In the PSR scenario, the user's question is: "In a new metropolitan area network (MAN) scenario with PON access, what device network elements (RES) are there under the MAN Layer 3 VPN access RFS? How do they achieve network connectivity?" First, one or more third entities are extracted from this question, including "PON access" and "MAN Layer 3 VPN access RFS". Using entity linking technology, these third entities are aligned with nodes in the PSR Service Knowledge Graph (PSR-KG). Since there are multiple third entities, the centrality (e.g., out-degree centrality) of each third entity in the PSR-KG is further calculated. It is found that "MAN Layer 3 VPN access RFS" has the most neighboring nodes, i.e., the highest centrality, and is therefore identified as the first entity.

[0104] Subsequently, based on the first entity, a second text was retrieved from the text corpus, and a large language model was used to determine whether the second text could answer the question. The retrieval results included descriptions such as the RFS interfacing with A-Leaf devices and relying on PON links, but the large language model evaluation concluded that the current text did not fully list all the lower-layer device network elements (RES) nor explain the specific mechanism of network connectivity, and the information was insufficient to generate an answer, thus entering the iterative process.

[0105] Next, based on the question, PSR-KG, and the first entity, the second entity and additional hints are determined. Specifically, one or more second relationship edges connected to the first entity are first obtained from the PSR-KG, including "lower layer → A-Leaf device", "lower layer → PON link RFS", "upper layer → fixed network access line CFS", and "belonging network domain → new metropolitan area network". Based on the question's intent, the list of second relationship edges is input into the large language model along with the question, which scores the relevance of each relationship, resulting in "lower layer" scoring 0.9, "belonging network domain" scoring 0.2, and "upper layer" scoring 0.1. A threshold of 0.5 is set, and only the "lower layer" edge with the highest score is retained as the first relationship edge.

[0106] Based on this first relation edge, the corresponding third entities are identified from the PSR-KG, including "A-Leaf device" and "PON link RFS". Subsequently, the large language model evaluates the relevance of the first entity (i.e., "metropolitan area network layer 3 VPN access RFS") to each third entity and the problem, and determines that "A-Leaf device" as a device network element RES more directly responds to the core of the problem, and therefore identifies it as the new second entity.

[0107] At the same time, based on the triple "Metropolitan Area Network Layer 3 VPN Access RFS → Lower Layer → A-Leaf Device" consisting of the first entity, the first relation edge, and the third entity (A-Leaf device), an additional prompt in natural language form is generated: "The lower layer of Metropolitan Area Network Layer 3 VPN Access RFS is A-Leaf device".

[0108] Subsequently, using the second entity (A-Leaf device) and additional hints as a joint query, the first text was retrieved from the text corpus, including details such as the VLAN sub-interface configuration of the A-Leaf device, the establishment of EBGP neighbor with the BNG device, and the implementation of Layer 3 interconnection through VXLAN tunnel.

[0109] The large language model, considering the triple path, the first text, and historical clues (including explored entities such as "Metropolitan Area Network Layer 3 VPN Access RFS" and "A-Leaf Device," as well as excluded relationships), deems the current information insufficient: the existence of other device network elements (such as BNG devices) has not yet been confirmed, and the network connectivity mechanism is not fully closed-loop. Therefore, the large language model outputs optimized historical clues, suggesting "BNG Device" as a candidate entity and updating the first entity for the next round to "BNG Device" (if it is verified as a lower layer of RFS in the KG), and continues the iteration.

[0110] The process continuously loops until a preset maximum iteration depth N (e.g., N=5) is reached: each time, a new second entity and additional hints are determined based on the current first entity, the first text is retrieved, and the large language model determines whether it is a valid answer; if valid, the first text is returned to the user as the answer; if not, the historical clues are updated and a new first entity is determined, until an answer is obtained or the loop terminates. Finally, all triples used, historical clues, and answers can be returned together, providing an interpretable reasoning path.

[0111] This solution proposes a hybrid retrieval enhancement generation (RAG) framework suitable for PSR models. By deeply fusing the PSR business knowledge graph (PSR-KG) with text corpora, it achieves deep and comprehensive knowledge reasoning. This method uses the network topology and entity relationships in the PSR-KG (such as the hierarchical structure between CFS, RFS, and RES) to guide the text retrieval path, overcoming the limitations of traditional semantic retrieval that relies solely on superficial similarity. Simultaneously, it dynamically supplements the missing entity details in the knowledge graph (such as the configuration specifications and interface parameters of network element devices RES) using text corpora, forming a bidirectional enhancement mechanism between the knowledge graph and text.

[0112] Its core lies in an iterative retrieval method that simulates the progressive reasoning pattern of humans: in each round, the system explores potential paths based on the relational topology of the current entity in the Knowledge Base (KG), and combines this with contextual information retrieved from the text corpus for comprehensive judgment by a Large Language Model (LLM); if the information is insufficient, optimization clues are generated to guide the next round of joint exploration of the KG and text. Through the alternation of KG structure guidance and text semantic verification, the system gradually focuses on key clues related to the question, ultimately achieving accurate knowledge discovery.

[0113] This framework achieves a deep integration of structure and semantics: on the one hand, it overcomes the problems of weak relevance and broken logical chains in plain text RAGs under complex business scenarios; on the other hand, it compensates for the information fragmentation defects caused by the lack of specific networking details between network elements in PSR business knowledge graphs. As a result, the reasoning process of LLM has both logical rigor (derived from the explicit relationships of KG) and contextual completeness (derived from the rich semantics of text), significantly improving the accuracy and interpretability of question answering in telecom cloud network business scenarios.

[0114] Based on the same inventive concept, this disclosure also provides a retrieval device, as shown in the following embodiments. Since the principle by which the retrieval device solves the problem is similar to that of the above method embodiments, the implementation of the retrieval device can refer to the implementation of the above method embodiments, and repeated details will not be elaborated further.

[0115] Figure 7 An embodiment of the present disclosure shows a retrieval device, such as... Figure 7 As shown, the retrieval device may include: Unit 701 is configured to receive user questions and determine the first entity from the questions based on the knowledge graph; Generation unit 702 is configured to determine a second entity and additional hints based on the question, knowledge graph and first entity; The judgment unit 703 is configured to retrieve the first text from the text library based on the second entity and additional prompts, and use a large language model to determine whether the first text can answer the question; Feedback unit 704 is configured to provide the first text as the answer to the question to the user if possible. The optimization unit 705 is configured to optimize the historical clues using a large language model if the problem cannot be solved, and to determine a new first entity based on the historical clues, so as to repeat the above steps until the answer is obtained, wherein the historical clues include all the determined entities.

[0116] In some embodiments, the determining unit 701 is further configured to extract one or more third entities from the problem; if there are multiple third entities, the centrality of each third entity in the knowledge graph is determined, and the third entity with the highest centrality is taken as the first entity; if there is only one third entity, the third entity is taken as the first entity.

[0117] In some embodiments, the generation unit 702 is further configured to retrieve a second text from a text library based on a first entity, and use a large language model to determine whether the second text can answer the question; if it can, the second text is fed back to the user as the answer; if it cannot, the second entity and additional hints are determined based on the question, the knowledge graph, and the first entity.

[0118] In some embodiments, the generation unit 702 is further configured to determine a first relation edge of a first entity from the knowledge graph based on a question; determine a third entity from the knowledge graph based on the first relation edge; determine whether the first entity or the third entity is a second entity based on the question; and generate additional hints based on the triples of the first entity, the first relation edge, and the third entity.

[0119] In some embodiments, the generation unit 702 is further configured to determine one or more second relation edges of the first entity from the knowledge graph; if there are multiple second relation edges, the relevance score of each second relation edge to the question is determined, and the second relation edge with the highest relevance score is taken as the first relation edge; if there is only one second relation edge, the second relation edge is taken as the first relation edge.

[0120] In some embodiments, the generation unit 702 is further configured to determine the relevance scores of the first entity and the third entity to the problem; and to designate the entity with the higher relevance score among the first entity and the third entity as the second entity.

[0121] In some embodiments, the feedback unit 704 is also configured to provide the user with triples, historical clues, and answers.

[0122] In some embodiments, the optimization unit 705 is further configured to receive a question, determine a first entity from the question based on a knowledge graph, and execute the following loop: determine whether i is equal to N, where i is the number of loop executions, i is initially 0, N is the maximum depth of the loop, and N is a positive integer; if i is equal to N, the first text retrieved in the (i-1)th loop is fed back to the user as the answer, and the loop exits; if i is less than N, a second entity and additional hints are determined based on the question, the knowledge graph, and the first entity; the first text is retrieved from the text library based on the second entity and additional hints, and a large language model is used to determine whether the first text can answer the question; if it can, the first text is fed back to the user as the answer, and the loop exits; if it cannot, the large language model is used to optimize historical clues, a new first entity is determined according to the historical clues, and i is updated with the value of i+1.

[0123] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0124] The following reference Figure 8 To describe an electronic device 800 according to such an embodiment of the present disclosure. Figure 8The electronic device 800 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0125] like Figure 8 As shown, the electronic device 800 is presented in the form of a general-purpose computing device. The components of the electronic device 800 may include, but are not limited to: at least one processor 810, at least one memory 820, and a bus 830 connecting different system components (including memory 820 and processor 810).

[0126] The memory stores program code that can be executed by the processor 810, causing the processor 810 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processor 810 can perform the following steps of the above method embodiments: receiving a user's question; determining a first entity from the question based on a knowledge graph; determining a second entity and additional hints based on the question, the knowledge graph, and the first entity; retrieving a first text from a text library based on the second entity and additional hints; using a large language model to determine whether the first text can answer the question; if it can, feeding the first text back to the user as the answer to the question; if it cannot, optimizing historical clues using a large language model, determining a new first entity according to the historical clues, and repeating the above steps until an answer is obtained, wherein the historical clues include all determined entities.

[0127] The memory 820 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 8201 and / or cache memory 8202, and may further include read-only memory (ROM) 8203.

[0128] The memory 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0129] Bus 830 can represent one or more of several types of bus structures, including a memory bus or memory controller, peripheral bus, graphics acceleration port, processor, or a local bus using any of the various bus structures.

[0130] Electronic device 800 can also communicate with one or more external devices 840 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more retrieval systems that enable user interaction with electronic device 800, and / or with any device that enables electronic device 800 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 850. Furthermore, electronic device 800 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 860. As shown, network adapter 860 communicates with other modules of electronic device 800 via bus 830. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 800, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0131] In the disclosed exemplary embodiments, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium.

[0132] In some possible implementations, various aspects of this disclosure may also be implemented as a program product comprising program code that, when run on a terminal device, causes the terminal device to perform the steps described in the foregoing “Detailed Description” section of this specification according to various exemplary embodiments of this disclosure.

[0133] More specific examples of computer-readable storage media in this disclosure may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0134] In this disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device.

[0135] Optionally, the program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0136] In practical implementation, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on a terminal device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0137] This disclosure provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a retrieval method provided in various alternative embodiments of this disclosure.

[0138] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0139] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0140] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0141] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope of this disclosure is indicated by the appended claims.

Claims

1. A retrieval method characterized by, The method comprises the following steps: receiving a user's question, determining a first entity from the question based on a knowledge graph; determining a second entity and an additional prompt based on the question, the knowledge graph and the first entity; retrieving a first text from a text library based on the second entity and the additional prompt, and determining whether the first text can answer the question by using a large language model; if yes, feeding back the first text as the answer to the question to the user; if no, optimizing a historical clue by using the large language model, determining a new first entity according to the historical clue, and repeating the above steps until the answer is obtained, wherein the historical clue contains all the determined entities.

2. The method of claim 1, wherein, The method comprises the following steps: extracting one or more third entities from the question; if there are multiple third entities, determining the centrality of each third entity in the knowledge graph, and taking the third entity with the highest centrality as the first entity; if there is only one third entity, taking the third entity as the first entity.

3. The method of claim 1, wherein, After determining the first entity based on the knowledge graph, the method further comprises the following steps: retrieving a second text from the text library based on the first entity, and determining whether the second text can answer the question by using the large language model; if yes, feeding back the second text as the answer to the user; if no, determining the second entity and the additional prompt based on the question, the knowledge graph and the first entity.

4. The method of claim 1, wherein, The method comprises the following steps: determining a first relationship edge of the first entity from the knowledge graph based on the question; determining a third entity from the knowledge graph based on the first relationship edge, and determining the first entity or the third entity as the second entity based on the question; generating the additional prompt based on the triple of the first entity, the first relationship edge and the third entity.

5. The method of claim 4, wherein, The method comprises the following steps: determining one or more second relationship edges of the first entity from the knowledge graph; if there are multiple second relationship edges, determining the relevance score of each second relationship edge with the question, and taking the second relationship edge with the highest relevance score as the first relationship edge; if there is only one second relationship edge, taking the second relationship edge as the first relationship edge.

6. The method of claim 4, wherein, The method comprises the following steps: determining the relevance score of the first entity and the third entity with the question respectively; taking the entity with higher relevance score between the first entity and the third entity as the second entity.

7. The method of claim 4, wherein, The method further comprises the following steps: feeding back the triple, the historical clue and the answer to the user.

8. The method of claim 1, wherein, The method further comprises the following steps: receiving the question, determining the first entity from the question based on the knowledge graph, and executing the following loop: determining whether i is equal to N, where i is a number of times of execution of the loop, an initial value of i is 0, N is a maximum depth of the loop, and N is a positive integer; if i is equal to N, feeding back the first text retrieved by the i-1th loop as the answer to the user, and exiting the loop; if i is less than N, determining the second entity and the additional prompt based on the question, the knowledge graph, and the first entity; retrieving the first text from the text library based on the second entity and the additional prompt, and determining whether the first text can answer the question by using the large language model; if yes, feeding back the first text as the answer to the user, and exiting the loop; if no, optimizing the historical clue by using the large language model, determining a new first entity according to the historical clue, and updating i with a value of i+1.

9. A retrieval device characterized by, comprising: a determining unit configured to receive a question of a user, and determine a first entity from the question based on a knowledge graph; a generating unit configured to determine a second entity and an additional prompt based on the question, the knowledge graph, and the first entity; a judging unit configured to retrieve a first text from a text library based on the second entity and the additional prompt, and determine whether the first text can answer the question by using a large language model; a feeding back unit configured to feed back the first text as the answer to the question to the user if the first text can answer the question; an optimizing unit configured to optimize a historical clue by using the large language model if the first text cannot answer the question, determine a new first entity according to the historical clue, and repeat the above steps until the answer is obtained, wherein the historical clue contains all determined entities.

10. An electronic device, comprising: comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the method of any one of claims 1-8 by executing the executable instructions.

11. A computer readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the method of any one of claims 1-8.

12. A computer program product comprising computer instructions stored in a computer readable storage medium, the computer instructions being executed by a processor to implement the operation instructions of the method of any one of claims 1-8.

Images