Data processing method and system

Abstract

The embodiment of the invention provides a data processing method and system. The method comprises the steps of obtaining a target question sent by a user; a knowledge graph is obtained, the knowledge graph comprises a plurality of nodes and edges among the nodes, the knowledge graph is generated based on documents in a document library, and the knowledge graph is divided into a plurality of communities; the correlation degree between the target question and each community in the knowledge graph is analyzed from multiple correlation dimensions, so that at least one target community related to the target question is determined from the knowledge graph, and the multiple correlation dimensions comprise at least two of a text correlation dimension, a semantic correlation dimension and a context correlation dimension; and community abstracts corresponding to the at least one target community are obtained, answer information corresponding to the target question is generated based on the community abstracts corresponding to the at least one target community, and the community abstract corresponding to each target community is a general description of nodes in the target community and relationships thereof.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method and system. Background Art

[0002] With the advent of the big data era, the explosive growth of data volume has made information retrieval technology an indispensable part of modern society. Whether in academic research or personal life, people rely on information retrieval technology to obtain the information they need. For example, some retrieval platforms provide users with a retrieval method for information retrieval in a document library. Among them, the document library can include multiple documents. Users can enter retrieval requirements into the retrieval platform, such as retrieval keywords or questions. The retrieval platform retrieves relevant content in the document library based on the above retrieval requirements and generates retrieval results.

[0003] However, as the amount of data in document libraries continues to increase, users' demand for information retrieval is also increasing. More and more users expect to quickly and accurately obtain the required information from massive text data and generate summaries or decision support information for specific queries.

[0004] The content of the background technology section is only the information known to the inventor personally, and does not mean that the above information has entered the public domain before the application date of this disclosure, nor does it mean that it can become the prior art of the present disclosure. Summary of the invention

[0005] This specification provides a data processing method and system, which can quickly and accurately generate answer information for a target question and make the answer information have a certain summary depth.

[0006] In a first aspect, the present specification provides a data processing method, comprising: obtaining a target question sent by a user; obtaining a knowledge graph, the knowledge graph comprising a plurality of nodes and edges between the plurality of nodes, the knowledge graph being generated based on documents in a document library, the nodes corresponding to entities in the documents, the edges corresponding to relationships between different entities in the documents, the knowledge graph being divided into a plurality of communities, the closeness between nodes within the same community being greater than the closeness between nodes in different communities; analyzing the relevance between the target question and each community in the knowledge graph from a plurality of relevant dimensions to determine at least one target community related to the target question from the knowledge graph, wherein the plurality of relevant dimensions comprise at least two of a text-related dimension, a semantic-related dimension, and a context-related dimension; and obtaining a community summary corresponding to each of the at least one target communities, and generating answer information corresponding to the target question based on the community summary corresponding to the at least one target community, wherein the community summary corresponding to each target community is a general description of the nodes and their relationships in the target community.

[0007] In some embodiments, the analysis of the relevance between the target question and each community in the knowledge graph from multiple relevance dimensions to determine at least one target community related to the target question from the knowledge graph includes: from the text relevance dimension, based on the keywords in the target question, determining the text relevance between each community in the knowledge graph and the target question respectively; from the semantic relevance dimension, based on the semantic information of the target question, determining the semantic relevance between each community in the knowledge graph and the target question respectively; from the context relevance dimension, based on the most recent N historical questions before the target question, determining the context relevance between each community in the knowledge graph and the target question respectively, where N is an integer greater than 1; and determining the at least one target community in the knowledge graph based on the text relevance, the semantic relevance and the context relevance corresponding to each community in the knowledge graph respectively.

[0008] In some embodiments, each element in the community corresponds to an element summary, each element corresponds to a node or an edge, and based on the keywords in the target question, determining the text relevance between each community in the knowledge graph and the target question, includes: for each community in the knowledge graph, searching for target elements in the community, and determining the text relevance between the community and the target question based on the number of target elements searched, and the element summary corresponding to the target element contains the keyword.

[0009] In some embodiments, based on the semantic information of the target question, determining the semantic relevance between each community in the knowledge graph and the target question, includes: for each community in the knowledge graph, obtaining the community summary corresponding to the community, and determining the semantic relevance between the community and the target question based on the semantic similarity between the semantic information of the target question and the community summary.

[0010] In some embodiments, the context relevance between each community in the knowledge graph and the target question is determined from the context relevance dimension based on the N most recent historical questions before the target question, including: for each community in the knowledge graph, K target historical questions are screened out from the N historical questions, wherein the community is selected as the target community in the process of answering the target historical question, and K is an integer greater than or equal to 1, determining the similarities between the K target historical questions and the target question, and performing weighted summation of the similarities between the K target historical questions and the target question to obtain the context relevance between the community and the target question, wherein the weights corresponding to each of the K target historical questions gradually increase with the order of the K target historical questions.

[0011] In some embodiments, the determining of the at least one target community in the knowledge graph based on the text relevance, the semantic relevance, and the context relevance respectively corresponding to each community in the knowledge graph includes: for each community in the knowledge graph, based on the weights corresponding to each of the multiple related dimensions, taking a weighted sum of the text relevance, the semantic relevance, and the context relevance to determine the comprehensive relevance between the target problem and the community; based on the comprehensive relevance respectively corresponding to each community in the knowledge graph, selecting k communities with the highest comprehensive relevance as the at least one target community, wherein k is an integer greater than or equal to 1.

[0012] In some embodiments, the method further includes: determining the weights corresponding to each of the multiple related dimensions according to the type of the target question or at least one of the preference information of the user for the multiple related dimensions.

[0013] In some embodiments, generating answer information corresponding to the target question based on the community summary corresponding to the at least one target community includes: providing the community summary corresponding to each target community and the target question to a large model, and guiding the large model to generate candidate answers to the target question, thereby obtaining at least one candidate answer; and summarizing the at least one candidate answer to obtain answer information corresponding to the target question.

[0014] In some embodiments, the aggregating the at least one candidate answer to obtain answer information corresponding to the target question includes: providing the target question, the at least one candidate answer, historical questions before the target question, and historical answers to the historical questions to a large model, and guiding the large model to aggregate the at least one candidate answer with reference to the historical questions and the historical answers to generate answer information corresponding to the target question.

[0015] In some embodiments, the method also includes: determining the type of the target question; obtaining the community summary corresponding to each of the at least one target communities, and generating answer information corresponding to the target question based on the community summary corresponding to the at least one target community, including: when the type of the target question represents a query for global information, obtaining the community summary corresponding to each of the at least one target communities, and generating answer information corresponding to the target question based on the community summary corresponding to the at least one target community.

[0016] In some embodiments, the method further includes: when the type representation of the target question inquires about local information, obtaining element summaries corresponding to each element within the at least one target community, wherein each element corresponds to a node or an edge, and the element summary corresponding to each element is a general description of the element; and generating answer information corresponding to the target question based on the element summaries corresponding to each element within the at least one target community.

[0017] In some embodiments, the method also includes: extracting multiple triples from the documents in the document library, each triple including two entities and the relationship between the two entities; generating the knowledge graph based on the multiple triples; and dividing the knowledge graph into the multiple communities, and generating community summaries corresponding to the multiple communities respectively.

[0018] In some embodiments, the extracting multiple triples from the document in the document library includes: providing the document to a large model, and guiding the large model to extract at least one triple from the document, and adding the at least one triple to a triple set; repeating the following extraction process until no new triples are identified, or until an upper limit on the number of extractions is reached: providing at least part of the triples in the triple set to the large model, and guiding the large model to extract new triples from the document using the at least part of the triples as examples, and adding the new triples to the triple set.

[0019] In some embodiments, the dividing the knowledge graph into the multiple communities and generating community summaries corresponding to the multiple communities respectively include: generating element summaries corresponding to each element in the knowledge graph based on the entities in the knowledge graph and the relationships between different entities, wherein each element corresponds to a node or an edge, and the element summary corresponding to each element is a general description of the element; dividing the knowledge graph into the multiple communities based on the correlation between different elements in the knowledge graph; and generating, for each community in the multiple communities, a community summary corresponding to the community based on the element summaries corresponding to each element in the community.

[0020] In a second aspect, this specification also provides a data processing system, including:

[0021] at least one storage medium storing at least one instruction set for performing data processing; and

[0022] At least one processor is communicatively connected to the at least one storage medium, wherein when the data processing system is running, the at least one processor reads the at least one instruction set and executes the data processing method as described in any one of the first aspects according to the instructions of the at least one instruction set.

[0023] It can be seen from the above technical solutions that in the data processing method and system provided in this specification, a target question sent by a user is obtained; a knowledge graph is obtained, the knowledge graph is divided into multiple communities, and the closeness between nodes within the same community is greater than the closeness between nodes in different communities; the relevance between the target question and each community in the knowledge graph is analyzed from multiple relevant dimensions to determine at least one target community related to the target question from the knowledge graph, wherein the multiple relevant dimensions include: at least two of text-related dimensions, semantic-related dimensions, and context-related dimensions; and a community summary corresponding to each of the at least one target community is obtained, and based on the community summary corresponding to the at least one target community, the answer information corresponding to the target question is generated. The above scheme searches for target communities related to the target question from multiple different relevant dimensions, so that the target community found is more accurate, and then the answer information generated based on the community summary corresponding to the target community is more comprehensive and accurate. In addition, using the community as the granularity, finding the target community related to the target question in the knowledge graph can improve the retrieval speed of the knowledge graph, and thus improve the speed of answering the target question. Since the community summary is a general description of the nodes and their relationships in the community, answer information is generated based on the retrieved community summary of the target community, so that the answer information has a certain summary depth.

[0024] Other functions of the data processing method and system provided in this specification will be partially listed in the following description. The creative aspects of the data processing method and system provided in this specification can be fully explained by practicing or using the methods, devices and combinations described in the following detailed examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 A schematic diagram of an application scenario of data processing provided according to an embodiment of this specification is shown;

[0027] Figure 2 A data processing hardware structure diagram provided according to an embodiment of this specification is shown;

[0028] Figure 3 A schematic diagram of a construction process of constructing a knowledge graph provided according to an embodiment of this specification is shown;

[0029] Figure 4 A flow chart of a data processing method provided according to an embodiment of this specification is shown. DETAILED DESCRIPTION

[0030] The following description provides specific application scenarios and requirements of this specification, with the purpose of enabling those skilled in the art to make and use the contents of this specification. Various local modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but to the widest scope consistent with the claims.

[0031] The terms used herein are only used for the purpose of describing specific example embodiments and are not restrictive. For example, unless the context clearly indicates otherwise, as used herein, the singular forms "a", "an" and "the" may also include plural forms. When used in this specification, the terms "include", "comprise" and / or "contain" mean that the associated integers, steps, operations, elements and / or components exist, but do not exclude the existence of one or more other features, integers, steps, operations, elements, components and / or groups or that other features, integers, steps, operations, elements, components and / or groups may be added in the system / method.

[0032] In view of the following description, these and other features of the present specification, as well as the operation and function of the related elements of the structure, and the economy of the combination and manufacture of the parts can be significantly improved. Reference is made to the accompanying drawings, all of which form a part of this specification. However, it should be clearly understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0033] The flowcharts used in this specification illustrate the operations implemented by the system according to some embodiments in this specification. It should be clearly understood that the operations of the flowcharts may not be implemented in sequence. On the contrary, the operations may be implemented in reverse order or simultaneously. In addition, one or more other operations may be added to the flowchart. One or more operations may be removed from the flowchart.

[0034] For the convenience of description, the terms that will appear in the following text of this specification are first explained.

[0035] Element: Each element corresponds to a node or an edge in the knowledge graph;

[0036] Element Summary: The element summary corresponding to each element is a general description of the element.

[0037] Community: Each community represents a specific topic or concept in the text, reflecting the natural aggregation of entities and relations.

[0038] Community summary: The community summary corresponding to each target community is a general description of the nodes and their relationships in the target community.

[0039] Text-related dimension: Match the keywords in the target question with the text content of each community in the knowledge graph and evaluate their similarity.

[0040] Semantic relevance dimension: Based on analyzing the semantic information of the target question (such as synonyms, hyponyms, and hyponyms), the semantic match with each community is evaluated.

[0041] Contextual relevance dimension: Use the N historical questions before the target question to evaluate the contextual relevance between each community in the knowledge graph.

[0042] The following is an introduction to the application scenarios of this manual.

[0043] The embodiments of this specification can be applied to summary generation scenarios in data processing. Specifically, the embodiments of this specification can be based on the historical interaction sequence formed by multiple interaction data that the target user interacts with successively in the historical time period, and generate answer information corresponding to the target question to help users quickly understand the key points of the information. It should be noted that the embodiments of this specification are not limited to the specific data processing field, that is, they can be used for any type of scenario. For example, the embodiments of this specification can be used for question and answer generation scenarios. In this scenario, the above-mentioned interaction data can refer to the target question queried by the user and the page content corresponding to the question, and the answer information corresponding to the target question is generated by extracting information related to the query and the target question. For another example, the embodiments of this specification can be used for query scenarios in the field of professional knowledge. In this scenario, the above-mentioned interaction data can be a query question of the user in the technical or academic field. The system generates an accurate knowledge summary by integrating the contents of multiple professional documents or technical reports to help users quickly understand the core ideas. In addition, the embodiments of this specification can also be applied to the summary extraction scenario of interaction records. By extracting key information from a large number of interaction records (problems that have occurred on an electrical appliance before and the corresponding solutions), solutions to common user problems (such as the user's use of the functions of an electrical appliance) are generated to provide users with quick reference and support.

[0044] It should be noted that the above summary generation scenario is only one of the multiple usage scenarios provided in this specification. The data processing method provided in this specification can be applied not only to the summary generation scenario, but also to all scenarios in which answer information corresponding to the target question needs to be generated in data processing. For example, it can be applied to data processing fields such as text analysis, information retrieval, knowledge management and automatic summary generation, such as search engines, academic research tools, automated report generation, enterprise knowledge management, online education content aggregation, news summary services, etc., which have high market application value and social benefits. Those skilled in the art should understand that when the data processing provided in this specification is applied to other usage scenarios, its implementation method and technical effect are similar.

[0045] Figure 1 FIG. 1 shows a schematic diagram of an application scenario of data processing provided according to an embodiment of this specification. Figure 1 As shown, the application scenario 100 may include a target user 110 , a client 120 , a server 130 , and a network 140 .

[0046] The target user 110 may be a user who triggers the processing of the target problem, and the target user 110 may perform data processing operations on the client 120 .

[0047] The client 120 may be a device that processes the target problem in response to the data processing operation of the target user 110. In some embodiments, the data processing method may be executed on the client 120. At this time, the client 120 may store data or instructions for executing the data processing method described in this specification, and may execute or be used to execute the data or instructions. In some embodiments, the client 120 may include a hardware device with a data information processing function and a necessary program required to drive the hardware device to work.

[0048] In some embodiments, the client 120 may include a mobile device, a tablet computer, a laptop computer, a built-in device of a motor vehicle, or the like, or any combination thereof. In some embodiments, the mobile device may include a smart home device, a smart mobile device, a virtual reality device, an augmented reality device, or the like, or any combination thereof. In some embodiments, the smart home device may include a smart TV, a desktop computer, or the like, or any combination thereof. In some embodiments, the smart mobile device may include a smart phone, a personal digital assistant, a gaming device, a navigation device, or the like, or any combination thereof. In some embodiments, the virtual reality device or the augmented reality device may include a virtual reality helmet, virtual reality glasses, a virtual reality patch, an augmented reality helmet, augmented reality glasses, an augmented reality patch, or the like, or any combination thereof. For example, the virtual reality device or the augmented reality device may include Google Glass, a head-mounted display, VR, or the like. In some embodiments, the built-in device in the motor vehicle may include an onboard computer, an onboard TV, or the like. In some embodiments, the client 120 may include a text or voice acquisition device for acquiring text or voice information in a target space, thereby obtaining a target problem. In some embodiments, the client 120 may be a device with positioning technology for locating the position of the client 120.

[0049] In some embodiments, the client 120 may be installed with one or more application programs (APPs). The APP can provide the target user 110 with the ability and interface to interact with the outside world through the network 140. The APP includes but is not limited to: web browser APP programs, search APP programs, chat APP programs, shopping APP programs, video APP programs, financial management APP programs, instant messaging tools, email clients, social platform software, etc. In some embodiments, a target APP may be installed on the client 120. The target APP can collect text or voice collection devices in the target space for the client 120 to obtain the target question. In some embodiments, the target user 110 can also trigger a data processing request through the target APP. The target APP can respond to the data processing request and execute the data processing method described in this specification.

[0050] The server 130 may be a server that provides various services, such as a background server that provides support for the target questions collected on the client 120. In some embodiments, the data processing method may be executed on the server 130. At this time, the server 130 may store data or instructions for executing the data processing method described in this specification, and may execute or be used to execute the data or instructions. In some embodiments, the server 130 may include a hardware device with a data information processing function and a necessary program to drive the hardware device to work. The server 130 may be connected to multiple clients 120 for communication, and receive data sent by the client 120.

[0051] The network 140 is used to provide a medium for communication connection between the client 120 and the server 130. The network 140 can facilitate the exchange of information or data. Figure 1 As shown, the client 120 and the server 130 can be connected to the network 140 and transmit information or data to each other through the network 140. In some embodiments, the network 140 can be any type of wired or wireless network, or a combination thereof. For example, the network 140 can include a cable network, a wired network, an optical fiber network, a telecommunications network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network, a near field communication (NFC) network, or a similar network. In some embodiments, the network 140 can include one or more network access points. For example, the network 140 can include a wired or wireless network access point, such as a base station or an Internet exchange point, through which one or more components of the client 120 and the server 130 can be connected to the network 140 to exchange data or information.

[0052] It should be understood that Figure 1 The number of clients 120, servers 130 and networks 140 in the embodiment is only illustrative. Any number of clients 120, servers 130 and networks 140 may be provided according to implementation requirements.

[0053] It should be noted that the data processing method may be completely executed on the client 120 , may be completely executed on the server 130 , or may be partially executed on the client 120 and partially executed on the server 130 .

[0054] It should be noted that the user data obtained in this manual has been authorized by the user and does not involve user privacy.

[0055] Figure 2 FIG. 2 shows a hardware structure diagram of a computing device 200 provided according to an embodiment of the present specification. The computing device 200 can be used as Figure 1 The client 120 or server 130 in the embodiment executes the data processing method described in this specification. The computing device 200 can also be used as a data processing system provided in this specification to execute the data processing method provided in this specification.

[0056] like Figure 2 As shown, computing device 200 may include at least one storage medium 230 and at least one processor 220. In some embodiments, computing device 200 may also include communication port 250 and internal communication bus 210. Computing device 200 may also include I / O component 260.

[0057] The internal communication bus 210 can connect different system components. For example, the internal communication bus 210 can connect the storage medium 230, the processor 220, the communication port 250, and the I / O component 260, etc.

[0058] I / O components 260 support input / output between computing device 200 and other components.

[0059] The communication port 250 is used for data communication between the computing device 200 and the outside world. For example, the communication port 250 can be used for data communication between the computing device 200 and the network 140. The communication port 250 can be a wired communication port or a wireless communication port.

[0060] The storage medium 230 may include a data storage device. The data storage device may be a non-temporary storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 232, a read-only storage medium (ROM) 234, or a random access storage medium (RAM) 235. The storage medium 230 also includes at least one instruction set stored in the data storage device. The instruction set may include computer program code, and the computer program code may include a program, a routine, an object, a component, a data structure, a process, a module, and the like.

[0061] At least one processor 220 may be communicatively connected to at least one storage medium 230. When the computing device 200 is running, at least one processor 220 reads the at least one instruction set and executes the data processing method provided in this specification according to the instructions of the at least one instruction set. The processor 220 may execute the steps included in the data processing method. The processor 220 may be in the form of one or more processors. In some embodiments, the processor 220 may include one or more hardware processors, such as a microcontroller, a microprocessor, a reduced instruction set computer (RISC), an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a central processing unit (CPU), a graphics processing unit (GPU), a physical processing unit (PPU), a microcontroller unit, a digital signal processor (DSP), a field programmable gate array (FPGA), an advanced RISC machine (ARM), a programmable logic device (PLD), any circuit or processor capable of performing one or more functions, etc., or any combination thereof.

[0062] For illustration purposes only, the computing device 200 in the figure shows only one processor 220. However, it should be noted that the computing device 200 in this specification may also include multiple processors, and therefore, the operations and / or method steps disclosed in this specification may be performed by one processor or by multiple processors in combination. For example, if the processor 220 of the computing device 200 is described in this specification as performing step A and step B, it should be understood that step A and step B may also be performed jointly or separately by two different processors 220 (e.g., the first processor performs step A, the second processor performs step B, or the first and second processors perform steps A and B together).

[0063] In some embodiments, information retrieval can be performed using retrieval-augmented generation (RAG) technology. The RAG method usually relies on local text retrieval to generate answers, and it is difficult to meet the user's needs for comprehensive and in-depth information when facing complex global problems. This specification discloses an innovative technology that combines graph theory principles and large language models (LLMs) to construct a knowledge graph and apply a community detection algorithm to provide a new data processing method-Graph RAG. This method can be used to process and summarize large-scale text data sets. The Graph RAG solution disclosed in the present invention can solve the limitations of RAG in processing complex queries and large-scale data sets. By dividing the modular processing of the community and generating community summaries and element summaries, it can generate a more comprehensive, accurate and highly relevant summary to the user query. Compared with the RAG solution, the Graph RAG solution has significant advantages in processing speed, summary quality and user customization requirements. It can significantly improve the accuracy and efficiency of information retrieval and meet the user's demand for high-quality summaries.

[0064] It is worth noting that Graph RAG needs to build a graph-based knowledge graph to effectively organize and summarize information, and then generate comprehensive, diverse and informative answers. Figure 3 Explain how to construct a knowledge graph and generate a community summary.

[0065] Figure 3 A schematic diagram of a knowledge graph construction process provided according to some embodiments of the present specification is shown.

[0066] In some embodiments of the present specification, multiple triples can be extracted from documents in a document library, each triple including two entities and a relationship between the two entities; the knowledge graph is generated based on the multiple triples; and the knowledge graph is divided into the multiple communities, and community summaries corresponding to the multiple communities are generated respectively.

[0067] In some embodiments, the documents in the document library can come from public databases, internal enterprise systems, web pages crawled from the Internet, etc. It can also be selected based on different usage scenarios. For example, in the scenario of the question-answering system, the documents in the document library can include all documents that can be searched on the Internet; in the query scenario of the professional knowledge field, the documents in the document library can include all documents and technical materials related to the professional knowledge field that can be searched on the Internet. Of course, the scope of documents in the document library can also be personalized according to different usage scenarios. For example, in the query scenario of the professional knowledge field, the documents in the document library can be limited to all documents and technical materials related to the professional knowledge field in a company's intranet, so that the generated answer information will be more targeted. The documents in the document library in the above different scenarios are different, and the generated knowledge graph content is also different. Those skilled in the art can understand that in different usage scenarios, the content referred to by the documents in the document library may be different.

[0068] In some embodiments, the information in the knowledge graph is represented in the form of triples. Therefore, before constructing the knowledge graph, the computing device 200 needs to extract triples from the documents in the document library. Before extracting triples, the computing device 200 needs to pre-process the documents in the document library. Specifically: first, extract text from the documents in the document library. Secondly, documents from different channels may contain noise, so after extracting the text, data cleaning is required to delete duplicate, erroneous or missing information to ensure the consistency and accuracy of the data. The text also needs to be formatted and the data converted into a suitable structure for further processing. Finally, it is divided into smaller text blocks (for example, each text block does not exceed 4096 tokens) to adapt to the window processing limitations of the large model.

[0069] In some embodiments, a triple contains (entity 1, relationship, entity 2), for example (Li Si, works at, company A). Different triples form nodes and edges in the knowledge graph by extracting entities and relationships. Optionally, models such as LLMs can be used to extract entities and relationships. Specifically, when extracting triples, the above-mentioned segmented text blocks are provided to the LLMs model respectively, and each text block is analyzed using LLMs to identify and extract entities (such as names, places, organizations, etc.) and their attributes (names, types, descriptions). After identifying the entities, it is also necessary to distinguish between entities with the same name but different meanings, such as distinguishing whether "apple" refers to fruit or company. At the same time, the relationships between entities in the text block are identified, and descriptions of these relationships are extracted. Relationship extraction is to extract the relationships between entities from the text block. For example, from "Li Si is Zhang San's colleague", it can be extracted that there is a "colleague" relationship between "Li Si" and "Zhang San".

[0070] After extracting multiple triples, a graph-based knowledge graph can also be constructed based on each triple. Specifically, the extracted entities are used as nodes and the relationships are used as edges to construct a graph-based knowledge graph, where the information carried by the nodes and edges is extracted through LLMs, including entity descriptions and relationship descriptions. By constructing a knowledge graph, when performing subsequent information retrieval for the target question, relevant triples can be queried from the knowledge graph, which facilitates the rapid retrieval of entities and their associated information.

[0071] It is worth mentioning that please refer to Figure 3 In some embodiments of the present specification, the above process from segmentation to triple extraction can be operated by a large model, and the computing device 200 can also perform multiple rounds of triple refinement in the following manner to ensure that all relevant entities and relationships are identified and extracted: providing the document to the large model, and guiding the large model to extract at least one triple from the document, and adding the at least one triple to the triple set; repeating the following extraction process until no new triples are identified, or until the upper limit of the number of extractions is reached: providing at least part of the triples in the triple set to the large model, and guiding the large model to extract new triples from the document using the at least part of the triples as examples, and adding the new triples to the triple set.

[0072] In some embodiments, in each round of refinement, an Adaptive Few-Shot Prompting Strategy can be used in the LLMs model to select examples from the triples (entity-relationship-entity) that have good results extracted in the previous round, to guide the LLM to more accurately identify missed entities and relationships. The core of the Adaptive Few-Shot prompting strategy lies in dynamic example selection, so the accuracy and comprehensiveness of each round of extraction can also be improved by continuously optimizing prompts and utilizing the feedback mechanism of each round mentioned above. The multiple rounds of refinement process in the above embodiments can gradually improve the extraction accuracy of entities and relationships, and more comprehensively identify and extract relevant information.

[0073] In some embodiments of the present specification, the construction of the knowledge graph is completed by extracting the above triples. Further, the computing device 200 can divide the knowledge graph into the multiple communities and generate community summaries corresponding to the multiple communities in the following manner: based on the entities in the knowledge graph and the relationship between different entities, generate element summaries corresponding to each element in the knowledge graph, wherein each element corresponds to a node or an edge, and the element summary corresponding to each element is a general description of the element; based on the correlation between different elements in the knowledge graph, divide the knowledge graph into the multiple communities; and for each of the multiple communities, generate a community summary corresponding to the community based on the element summaries corresponding to each element in the community.

[0074] In some embodiments, the computing device 200 may provide the above-extracted triples to the large model in sequence, or construct a parallel analysis model (which may be embedded in the large model) to perform parallel analysis on different triples to generate multiple element summaries corresponding to the same element. The large model then analyzes and integrates the above-mentioned multiple element summaries to generate a final element summary about the element. In other embodiments, the above-extracted triples may be input in batches, that is, the knowledge graph is decomposed into multiple parts (or several subgraphs), and each part (such as several triples) is used as an independent input and input into the large model in sequence. This can avoid exceeding the limit of the context window, and the model can gradually understand the information of each part and then perform global reasoning. In other embodiments, the knowledge graph constructed by triples can also be directly provided to the large model, and the large model performs a comprehensive analysis based on all nodes (entities) and edges (relationships) corresponding to the same element in the knowledge graph to generate an element summary about the element.

[0075] In the above embodiment, in the way that the large model analyzes different triples sequentially or in parallel, since a group of triples contains less information, the model processes a single group of triples at a faster rate, and a relatively complete element summary of an element can be obtained through the final integration. In the way of obtaining element summaries through knowledge graph analysis, the large model uniformly analyzes the global information contained in the knowledge graph, and the generated element summary content will be richer and more accurate. However, due to the large amount of information, the processing efficiency of the large model may be reduced, especially when the knowledge graph contains massive data, the processing capacity will be greatly weakened. Therefore, the method of generating element summaries is not limited in this specification, and those skilled in the art can make a choice based on actual usage scenarios and needs, or choose other possible generation methods.

[0076] In some embodiments, the degree of closeness (correlation) of the relationship between each triple can be preset according to the factual meaning represented by the triple. According to the correlation between the graph triples, the knowledge graph is divided into communities, and the knowledge graph is divided into the multiple communities. The closeness between nodes within the same community is greater than the closeness between nodes in different communities. In some possible embodiments, some rules can be used to correct the output of the graph community detection algorithm (such as the Leiden algorithm) to ensure that nodes and edges are accurately grouped according to the preset relationship closeness, wherein the Leiden algorithm divides network nodes into multiple closely connected communities by maximizing modularity.

[0077] In some embodiments, for each community in the multiple communities, element summaries of all elements in the community may be provided to the big model to guide the big model to generate a community summary corresponding to the community, where the community summaries can summarize the main entities and relationships within the community.

[0078] To more intuitively illustrate the above construction process, please refer to the following process S300 of extracting a text from "Dream of Red Mansions" and constructing a knowledge graph:

[0079] S310: Text extraction and segmentation Source document: Extract a text from "Dream of Red Mansions", such as part of the first chapter, and segment the text into small blocks of no more than 4096 tokens. For example: [The first block: contains the first half of the first chapter], [The second block: contains the second half of the first chapter].

[0080] S311: Extract triples: Use LLM to analyze each text block, identify and extract entities and their attributes. For example, entities can be names (such as Jia Baoyu, Lin Daiyu), places (such as Rongguo Mansion, Grand View Garden), organizations (such as Jia Family), etc. Attributes can be names, types, descriptions, etc. Relationships such as "Jia Baoyu is Lin Daiyu's cousin", "Jia Baoyu lives in Rongguo Mansion". The triples obtained through the above extraction process can be (Jia Baoyu, Lin Daiyu, cousin), (Jia Baoyu, Rongguo Mansion, lives in), etc.

[0081] S312: Construct a knowledge graph based on triples: Entities are nodes in the knowledge graph, for example: Node 1: Jia Baoyu (character); Node 2: Lin Daiyu (character); Node 3: Rongguo Mansion (place). Relationships are edges in the knowledge graph, for example: Edge 1 points from Node 1 (Jia Baoyu) to Node 2 (Lin Daiyu), and the description of Edge 1 is "cousin". Edge 2 points from Node 1 (Jia Baoyu) to Node 3 (Rongguo Mansion), and the description of Edge 2 is "lives in".

[0082] S313: Multiple rounds of refinement: Select the triples that worked well in the previous round as examples to guide LLM to more accurately identify missed entities and relationships. For example, if the previous round successfully identified (Jia Baoyu, Lin Daiyu, cousin), this triple can be used as an example in the next round of prompts. Optimize prompts: Continuously optimize prompts and use feedback mechanisms to improve the accuracy and comprehensiveness of each round of extraction. For example, after several rounds of refinement, it may be found that "Jia Mu" is also an important person, and she has a "grandmother" relationship with Jia Baoyu, thus obtaining the triple (Jia Mu, Jia Baoyu, grandmother).

[0083] S314: Generate element summary: Generate element summary for each node. For example, the element summary of node 1 (Jia Baoyu) can be "Jia Baoyu is one of the main characters in A Dream of Red Mansions and is the son of Rongguo Mansion". Generate element summary for each edge. For example, the element summary of edge 1 (the edge from node 1 to node 2) can be: "Jia Baoyu and Lin Daiyu are cousins".

[0084] S315: Community division: The community is divided according to the closeness of the relationship between different triples. For example, the closeness of the relationship between the triples (Jia Baoyu, Lin Daiyu, cousin) and (Jia Mu, Jia Baoyu, grandmother) is relatively high, so these two triples are divided into one community. When performing community division, the Leiden algorithm can be used for community division, and the result can be corrected according to the preset rules. For example, Jia Baoyu, Lin Daiyu, and Jia Mu are divided into the same community, indicating that they belong to the same family.

[0085] S316: Generate community summary: For each community, generate a community summary corresponding to the community based on the element summary of each element in the community. For example, the community summary of Jia Baoyu can be: "This community includes Jia Baoyu and his family, such as Lin Daiyu and Jia Mu. These characters are the core characters in "Dream of Red Mansions".

[0086] Based on the above construction process, we can build a knowledge graph based on some fragments of "Dream of Red Mansions" and multiple communities corresponding to the knowledge graph.

[0087] Figure 4 1 shows a flow chart of a data processing method P400 provided according to an embodiment of the present specification. As before, the computing device 200 can execute the data processing method P400 of the present specification. Specifically, the processor 220 in the computing device 200 can read the instruction set stored in its local storage medium, and then execute the data processing method P400 of the present specification according to the provisions of the instruction set. Figure 4 As shown, method P400 may include:

[0088] S410: Obtain the target question sent by the user.

[0089] In some embodiments of the present specification, the user can send the target question in the form of text or voice. When the target question is sent in the form of voice, the target question can be recognized with the help of the voice recognition model embedded in the computing device 200 to parse the content of the target question. For example, in an automatic question-and-answer scenario, in addition to text, the user can also describe the question by voice and send it to the question-and-answer system, and the question-and-answer system can convert the voice into text. Of course, in some possible ways, the user can also upload pictures or screenshots to describe the target question. In other possible ways, the user can also select the target question on the wearable device by gesture operation or touch click. Any existing supported conversion method can be used to convert target questions of different formats into the format supported by the computing device 200 in this specification, and this specification does not limit this.

[0090] S420: Obtain a knowledge graph, wherein the knowledge graph includes a plurality of nodes and edges between the plurality of nodes, wherein the knowledge graph is generated based on documents in a document library, wherein the nodes correspond to entities in the documents, and the edges correspond to relationships between different entities in the documents, and wherein the knowledge graph is divided into a plurality of communities, wherein the closeness between nodes within the same community is greater than the closeness between nodes in different communities.

[0091] In some embodiments, the knowledge graph is pre-built, that is, a complete knowledge graph structure has been created based on fixed data sources and domain knowledge before the system is run. This type of pre-built knowledge graph can be widely used in most information retrieval and analysis scenarios. Of course, in other embodiments, the knowledge graph can also be temporarily built after receiving the target question, and the construction of the dynamic knowledge graph is suitable for scenarios that require real-time integration and analysis of information from multiple data sources. For example, in the news field, when a user asks about a specific hot event, the system can extract entities and relationships from multiple news sources in real time, and build a temporary knowledge graph for the event to answer the user's question. This way of dynamically building a knowledge graph is more flexible and suitable for rapidly changing information environments and dynamic needs for the latest information. Based on the different construction methods in the above embodiments, the knowledge graph can take into account the needs of long-term stable fields and real-time dynamic scenarios, and provide users with flexible and efficient information support. Please refer to the above embodiments for the specific construction process of the knowledge graph, which will not be repeated here.

[0092] S430: Analyze the relevance between the target problem and each community in the knowledge graph from multiple relevant dimensions to determine at least one target community related to the target problem from the knowledge graph, wherein the multiple relevant dimensions include: at least two of: text-related dimensions, semantic-related dimensions, and context-related dimensions.

[0093] In some embodiments, the computing device 200 may analyze the relevance between the target question and the communities in the knowledge graph from multiple relevant dimensions by executing S431-S434 to determine at least one target community related to the target question from the knowledge graph.

[0094] S431: From the text relevance dimension, based on the keywords in the target question, determine the text relevance between each community in the knowledge graph and the target question.

[0095] In some embodiments, text relevance is applicable to situations where there are significant keywords in the explicit target question.

[0096] In some embodiments, each element in the community corresponds to an element summary, and each element corresponds to a node or an edge. The computing device 200 can determine the text relevance between each community in the knowledge graph and the target question in the following manner: for each community in the knowledge graph, search for target elements in the community, and determine the text relevance between the community and the target question based on the number of target elements searched, and the element summary corresponding to the target element contains the keyword.

[0097] In some embodiments, by counting the number of target elements found in each community, the computing device 200 can quantify the frequency of occurrence of these elements, thereby determining the text relevance between each community and the target question. In short, the more target elements there are, the higher the text relevance between the community and the target question. In some embodiments, the computing device 200 can also extract keywords in the query based on pre-trained models such as term frequency-inverse document frequency (TF-IDF) or bidirectional encoder representation Transformer (BERT), or text analysis methods such as word vector similarity, to calculate the degree of matching of keywords in each community.

[0098] S432: From the semantic relevance dimension, based on the semantic information of the target question, determine the semantic relevance between each community in the knowledge graph and the target question.

[0099] In some embodiments, the computing device 200 can determine the semantic relevance between each community in the knowledge graph and the target question in the following manner: for each community in the knowledge graph, obtain the community summary corresponding to the community, and determine the semantic relevance between the community and the target question based on the semantic similarity between the semantic information of the target question and the community summary.

[0100] In some embodiments, the computing device 200 can use the pre-trained language model RoBERTa to semantically embed the target question and the community summary, map them into vector representations, and then calculate the similarity between the semantic vector of the target question and the semantic vector of the community summary. For example, cosine similarity is used to measure the degree of semantic matching between the two. The higher the similarity, the stronger the correlation between the content of the community and the target question at the semantic level, thereby determining the semantic relevance between the community and the target question. Through the above embodiments, not only can the content with similar surface text matching be identified, but also those communities with the same or similar semantics can be captured, so that the computing system can still find the community that best matches the target question when faced with diversified expressions, thereby improving the intelligence and response accuracy of the question-answering system.

[0101] S433: From the context-related dimension, based on the N most recent historical questions before the target question, determine the context-related relevance between each community in the knowledge graph and the target question, where N is an integer greater than 1.

[0102] In some embodiments, the computing device 200 can determine the contextual relevance between each community in the knowledge graph and the target question in the following manner: for each community in the knowledge graph, K target historical questions are screened out from the N historical questions, wherein the community is selected as the target community in the process of answering the target historical question, and K is an integer greater than or equal to 1. Then, the similarity between the K target historical questions and the target question is determined, and the similarity between the K target historical questions and the target question is weighted and summed to obtain the contextual relevance between the community and the target question. The weights corresponding to the K target historical questions increase gradually with the order of the K target historical questions.

[0103] In some embodiments, the computing device 200 evaluates the degree of association between the K target historical questions and the current target question by analyzing the similarity between them. Specifically, it may include calculating textual or semantic similarity, such as cosine similarity, word embedding vector similarity, etc. In order to further enhance the coherence of the context, the computing device 200 performs a weighted summation of these similarity values ​​to determine the contextual relevance between the community and the target question. When weighted summation, the weight gradually increases with the order of the historical questions, that is, the closer the historical questions are to the target question, the greater the similarity weight. This processing method ensures that the most recent context has a stronger influence on the target question and maintains semantic coherence between questions. For example, if a community has a high correlation in the historical questions closest to the current target question, then the contextual relevance of the community will be higher, and it may be more in line with the needs of the current question.

[0104] For ease of understanding, please refer to the following example to explain how to calculate the contextual relevance to the target question:

[0105] For example, suppose N=10, that is, the most recent 10 historical questions before the target question are obtained. Among these 10 historical questions, community 1 is selected as the target community during the answering process of question 4, question 9, and question 10. Then the K target historical questions include: question 4, question 9, and question 10.

[0106] The similarity between question 4 and the target question is 50%, the similarity between question 9 and the target question is 80%, and the similarity between question 10 and the target question is 80%.

[0107] Question 4 is far from the target question and has a weight of 0.1. Questions 9 and 10 are closer to the target question and have weights of 0.4 and 0.5 respectively.

[0108] The weighting is performed as follows: 0.1*50%+0.4*80%+0.5*80%=0.77, that is, the contextual relevance between community 1 and the target question is 0.77.

[0109] Through the above embodiments, the logical relationship between the current question and the user's previous query can be better captured, ensuring that the selected community is not only relevant to the target question in terms of content, but also consistent with the user's overall query intention, thereby improving the accuracy and coherence of the response and helping to provide users with more natural and contextual answers.

[0110] S434: Determine the at least one target community in the knowledge graph based on the text relevance, the semantic relevance, and the context relevance respectively corresponding to each community in the knowledge graph.

[0111] In some embodiments, the computing device 200 may determine the at least one target community in the knowledge graph in the following manner: for each community in the knowledge graph, based on the weights corresponding to each of the multiple related dimensions, weighted sum the text relevance, the semantic relevance and the context relevance to determine the comprehensive relevance between the target question and the community; based on the comprehensive relevance corresponding to each community in the knowledge graph, select the k communities with the highest comprehensive relevance as the at least one target community, where k is an integer greater than or equal to 1.

[0112] In some embodiments, the computing device 200 may determine the weights corresponding to each of the multiple related dimensions in the following manner: determine the weights corresponding to each of the multiple related dimensions based on the type of the target question or at least one of the user's preference information for the multiple related dimensions.

[0113] In some embodiments, a fixed weight can be set in advance for each dimension. In other embodiments, the weights of different dimensions can also be adjusted according to the type of the target question, or at least one of the user's preference information for the multiple related dimensions. Specifically: In some embodiments, the weights of each dimension can also be determined according to the type of the target question. For example, if the target question belongs to a factual query, a higher weight can be given to the text relevance in order to more accurately match specific keywords; and if the target question is a type that requires conceptual understanding or reasoning, the weight of the semantic relevance can be increased to better capture deep semantic information.

[0114] In some embodiments, the weights may be adjusted based on the user's preference information for various related dimensions. For example, if the user tends to choose answers that focus more on historical context in previous interactions, the weight of contextual relevance may be increased; conversely, if the user prefers quick and direct answers, the weight of textual relevance may be increased.

[0115] In some embodiments, the determination of weights can also be dynamically adjusted in combination with user behavior data. For example, the user's preferences can be inferred through historical queries and feedback, and the weights of each dimension can be gradually optimized to provide more personalized recommendations. This flexible weight determination method can adapt to diverse user needs, ensure that the answers generated by the system are consistent with both the task scenario and the user's specific preferences, and improve user satisfaction and interactive experience.

[0116] For ease of understanding, please refer to the following example to explain how to select a target community:

[0117] Assume that in the current task, based on the weights set by user preferences, the weights of text relevance, semantic relevance, and context relevance are 0.3, 0.4, and 0.3 respectively.

[0118] For community A, the text, semantic and context relevance scores are 3, 4 and 2 respectively, so the weighted comprehensive relevance is: 0.3*3+0.4*4+0.3*2=0.9+1.6+0.6=3.1.

[0119] For community B, the text, semantic and context relevance scores are 2, 5 and 3, and the weighted comprehensive relevance is: 0.3*2+0.4*5+0.3*3=0.6+2+0.9=3.5.

[0120] Based on the comprehensive relevance of each community in the knowledge graph, the computing device 200 selects k communities with the highest comprehensive relevance as target communities, where k is an integer greater than or equal to 1. For example, if k=1, community B is selected as the final target community; if k=2, communities A and B are selected.

[0121] The above embodiment can ensure that the community that best meets the target question is selected based on the analysis combined with multiple relevant dimensions, thereby improving the accuracy of community division and the quality of answers.

[0122] In some embodiments, multiple candidate communities with the highest relevance to the target problem can be selected for each dimension separately, and then the final target community can be determined by union or intersection. That is, multiple candidate communities that are most relevant to the target problem are screened based on text, semantics, and context relevance. The union method means that the communities selected as relevant in all dimensions are merged to eventually include more possible candidate communities. The intersection method means that only communities that are relevant in all dimensions are selected to ensure that the final community performs well in all dimensions.

[0123] For ease of understanding, please refer to the following examples to explain how to select a target community by intersection or union:

[0124] Union: The text-related dimension selects communities A and B, the semantic-related dimension selects communities B and C, and the context-related dimension selects communities A and D. The final union is communities A, B, C, and D.

[0125] Intersection: The text-related dimension selects community A and community B, the semantic-related dimension selects community B and community C, and the context-related dimension selects community B and community D. The final intersection is community B.

[0126] It is worth noting that a combination of unions and intersections can also be selected between different dimensions, for example, a candidate union of text-related dimensions and semantic-related dimensions is selected, and then the intersection of the candidate union and context-related dimensions is selected. In the above embodiments, the union method can ensure maximum coverage of potentially related communities and increase the capture of diverse content, while the intersection method can ensure that the selected communities have high correlation in all dimensions, thereby improving the accuracy of the selection. The combination of union and intersection can increase the fault tolerance rate. The specific selection strategy can be set by those skilled in the art according to the actual scenario and usage requirements, and is not limited in this specification.

[0127] In some embodiments, a total relevance score may be obtained by calculating the sum of the relevance of each community in the three dimensions of text, semantics, and context. Then, a target community is selected based on the total relevance score, and the communities with the highest total relevance scores are considered to be most relevant to the target problem in general.

[0128] For ease of understanding, please refer to the following example to explain how to select a target community by using the total relevance score:

[0129] Community A has a text relevance score of 3, a semantic relevance score of 4, a context relevance score of 2, and a total relevance score of 9.

[0130] Community B has a text relevance score of 2, a semantic relevance score of 5, a contextual relevance score of 3, and a total relevance score of 10.

[0131] If a community with the highest total relevance score is selected, since community B has the highest total relevance score, community B is selected as the target community.

[0132] In the above embodiment, the overall relevance of the community is measured by the total relevance score, which is suitable for rapid evaluation and thus ensures the balance of different dimensions.

[0133] In some embodiments, the mean values ​​of different dimensions can be calculated and selected based on the mean values. For each community, the mean value of its relevance in the three dimensions of text, semantics and context is calculated, and then the target community is determined based on the mean value. This method balances the differences between the dimensions to reduce the impact of a high or low score on a certain dimension on the final result.

[0134] The text relevance score of community A is 3, the semantic relevance score is 4, and the context relevance score is 2, with an average of (3+4+2) / 3=3.

[0135] The text relevance score of community B is 5, the semantic relevance score is 5, and the context relevance score is 0, with an average of (5+5+0) / 3≈3.33.

[0136] If a community with the highest score is selected, community B is selected as the target community because its mean is higher.

[0137] In the above embodiment, the influence of extreme scores can be eliminated by selecting the mean, and the influence of each dimension can be balanced by the mean, so that the result is more stable and not affected by the fluctuation of the score of a single dimension.

[0138] It is worth noting that the above-mentioned different target community selection methods can be implemented in combination, or can be selected and adjusted according to the needs of actual application scenarios, so as to better determine the community most relevant to the target question and improve the accuracy and effectiveness of answering the question.

[0139] S440: Obtain a community summary corresponding to each of the at least one target community, and generate answer information corresponding to the target question based on the community summary corresponding to the at least one target community, wherein the community summary corresponding to each target community is a general description of nodes and their relationships in the target community.

[0140] In some embodiments, after determining the target community, the computing device 200 can obtain the community summary corresponding to at least one target community, and generate the answer information of the target question based on these summaries. In some embodiments, the community summary can be pre-generated and stored before receiving the question. In the construction and generation stage of the knowledge graph, the computing device 200 will pre-generate a community summary for each community, and the generated community summary will be stored in a special database so that it can be quickly retrieved when the user question needs to be answered later. This pre-generated community summary can significantly improve the response speed, because when generating the answer, the computing device 200 only needs to read the existing community summary from the database, avoiding the overhead of online real-time calculation. In other embodiments, the community summary can also be generated in real time when answering questions: the computing device 200 will generate a summary of the entire community based on the summary of each element (including nodes and edges) in the community after determining the target community. The advantage of this method is that it can ensure that the community summary is up to date, which is particularly suitable for scenarios where knowledge is frequently updated or dynamically changed. For example, in the news knowledge graph, events and related entities change very quickly, and the community summary generated in real time can better reflect the current state of knowledge, thereby providing users with the most relevant answer information.

[0141] In some embodiments, the computing device 200 may generate answer information corresponding to the target question in the following manner: providing the community summary corresponding to each target community and the target question to a large model, and guiding the large model to generate candidate answers to the target question, thereby obtaining at least one candidate answer; and summarizing the at least one candidate answer to obtain answer information corresponding to the target question.

[0142] In some embodiments, different large models can be selected based on different construction requirements of the system. In some embodiments, through the community summary and the target question, the large model can generate candidate answers that match the target question. The summary of each target community will guide the model to generate a candidate answer from different angles to ensure that all aspects of the target question are covered. Next, the computing device 200 will summarize at least one candidate answer generated. This summarization process includes combining key information from multiple candidate answers to ensure that the final generated answer information has high completeness, accuracy and coherence.

[0143] In some embodiments, the computing device 200 may provide the target question, the at least one candidate answer, historical questions before the target question, and historical answers to the historical questions to the big model, and guide the big model to summarize the at least one candidate answer with reference to the historical questions and the historical answers to generate answer information corresponding to the target question.

[0144] In some embodiments, during the aggregation process, the large model will refer to the content of historical questions and historical answers, and process and synthesize the candidate answers to ensure that the final generated answer not only accurately reflects the core information of the target question, but also maintains logical coherence with the previous conversation content, generating a more coherent and contextually consistent summary answer. By guiding the large model to combine the context of historical conversations, the large model can more accurately understand the user's intentions, and then optimize and aggregate the candidate answers to improve the overall quality of the answers. The above embodiments can effectively utilize historical interaction data so that the generated answers have both contextual coherence and can learn from multiple candidate answers to generate a more complete answer that meets user expectations, thereby improving the user experience.

[0145] In some embodiments, the computing device 200 first determines the type of the target question, and selects to generate an answer using a community summary or an element summary based on the type of the target question.

[0146] In some embodiments, the computing device 200 generates answer information corresponding to the target question in the following manner: when the type of the target question represents an inquiry for global information, a community summary corresponding to each of the at least one target community is obtained, and based on the community summary corresponding to the at least one target community, the answer information corresponding to the target question is generated.

[0147] In the above embodiment, the community summary represents a comprehensive description of the nodes and their relationships in the entire community, including global summary information. When the user asks a global question, the community summary can be used to generate an answer. Based on these community summaries, the system can generate answer information related to the target question. This method is particularly suitable for handling situations involving macro overviews, general trends or integrated answers, such as questions about the overall situation in a certain field.

[0148] In some embodiments, the computing device 200 generates an answer in the following manner: when the type representation of the target question inquires about local information, obtaining element summaries corresponding to each element within the at least one target community, wherein each element corresponds to a node or an edge, and the element summary corresponding to each element is a general description of the element; and generating answer information corresponding to the target question based on the element summaries corresponding to each element within the at least one target community.

[0149] In the above embodiment, the elements include each node or edge in the community, and the element summary is a concise description of the specific attributes and relationships of the node or edge. Through these element-level summaries, the system can generate more detailed answer information to accurately answer the user's local questions. For example, "What does xx represent?" or "What is the specific relationship between two nodes?" This method ensures the accuracy and pertinence of the answer, and is suitable for scenarios where users explore and gain in-depth understanding of specific details.

[0150] Still taking the knowledge graph of some paragraphs of "Dream of Red Mansions" constructed above as an example, the computing device 200 processes the user's query question. For example, the user inputs the target question "Who are Jia Baoyu's family members?" When processing, the computing device 200 first identifies the community related to "Jia Baoyu", and then analyzes the summary of the relevant community to generate a partial answer. For example, two corresponding answers are generated based on two communities: "Jia Baoyu's family members include Lin Daiyu" and "Jia Baoyu's family members include Jia Mu". Then, all partial answers related to the target question are summarized. The summary result is: "Jia Baoyu's family members include Lin Daiyu, Jia Mu, etc." Finally, a large model is used to combine the user's query history and context information to generate a more comprehensive answer. For example: "Jia Baoyu's family members include his cousin Lin Daiyu and his grandmother Jia Mu. They are the core characters in "Dream of Red Mansions" and live in Rongguo Mansion." It can be seen that the answers obtained based on the above embodiment are more comprehensive and diverse, and the search solution based on community summaries is also faster.

[0151] In summary, in the data processing method and system provided in this specification, after the computing device obtains the target question sent by the user, the correlation between the target question and each community in the knowledge graph is analyzed from multiple relevant dimensions to determine at least one target community related to the target question from the knowledge graph, and then, based on the community summary corresponding to the at least one target community, the answer information corresponding to the target question is generated. Among them, the above-mentioned multiple relevant dimensions include: at least two of the text-related dimensions, the semantic-related dimensions, and the context-related dimensions. In the above scheme, the computing device searches for the target community from multiple different dimensions, so that the target community found is a community that is truly related to the target question, and then generates answer information based on the community summary corresponding to the target community, so that the generated answer information is more comprehensive and accurate. In addition, in the process of retrieving the knowledge graph, the retrieval is performed at the community granularity. Compared with the retrieval method at the node granularity, the retrieval speed of the knowledge graph can be improved, thereby helping to improve the response speed of the question. Furthermore, since the community summary corresponding to each target community is a general description of the nodes and their relationships in the target community, the answer information generated by the computing device based on the community summary corresponding to the target community can enhance the depth of the answer information, making the expression of the answer information more natural and having a certain summary depth.

[0152] Furthermore, the above embodiments in this specification effectively overcome the limitations of the RAG solution in semantic understanding and information integration. It uses community detection algorithms to modularize text data and generate highly relevant and comprehensive summaries for specific queries, which not only improves the quality and efficiency of summaries, but also enhances the responsiveness to user queries.

[0153] On the other hand, this specification provides a computer-readable non-transitory storage medium, storing at least one instruction set for data processing. When the at least one instruction set is executed by the processor, the at least one instruction set instructs the processor to implement the steps of the data processing method P400 described in this specification. In some possible implementations, various aspects of this specification can also be implemented in the form of a program product, which includes a program code. When the program product is run on a computing device 200, the program code is used to enable the computing device 200 to perform the steps of the data processing method P400 described in this specification. The program product for implementing the above method can use a portable compact disk read-only memory (CD-ROM) to include program code and can be run on the computing device 200. However, the program product of this specification is not limited to this. In this specification, the readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system or used in combination with it. The program product can use any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media include: portable disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. The computer-readable storage medium may include a data signal propagated in baseband as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above. The program code for performing the operations of this specification may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may execute entirely on computing device 200, partly on computing device 200, as a stand-alone software package, partly on computing device 200 and partly on a remote computing device, or entirely on a remote computing device.

[0154] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require a specific order or a continuous order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0155] In summary, after reading this detailed disclosure, those skilled in the art will appreciate that the foregoing detailed disclosure may be presented only by way of example and may not be limiting. Although not explicitly stated herein, those skilled in the art will appreciate that this specification requires various reasonable changes, improvements and modifications to the embodiments. These changes, improvements and modifications are intended to be proposed by this specification and are within the spirit and scope of the exemplary embodiments of this specification.

[0156] In addition, certain terms in this specification have been used to describe embodiments of this specification. For example, "one embodiment", "an embodiment" and / or "some embodiments" mean that a particular feature, structure or characteristic described in conjunction with the embodiment may be included in at least one embodiment of this specification. Therefore, it can be emphasized and should be understood that two or more references to "an embodiment" or "an embodiment" or "an alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics may be appropriately combined in one or more embodiments of this specification.

[0157] It should be understood that in the foregoing description of the embodiments of this specification, in order to help understand a feature and for the purpose of simplifying this specification, this specification combines various features in a single embodiment, figure or its description. However, this does not mean that the combination of these features is necessary. When reading this specification, it is entirely possible for a person skilled in the art to mark out some of the devices as separate embodiments. In other words, the embodiments in this specification can also be understood as the integration of multiple secondary embodiments. And the content of each secondary embodiment is also valid when it is less than all the features of a single aforementioned disclosed embodiment.

[0158] Each patent, patent application, publication of patent applications, and other materials, such as articles, books, specifications, publications, documents, articles, etc., cited herein, except to the extent that it is inconsistent or conflicting with this document or that has a limiting effect on the broadest scope of the claims, may be incorporated herein by reference and used for all purposes now or hereafter associated with this document. In addition, in the event of any inconsistency or conflict between the description, definition, and / or use of a term in any material and the description, definition, and / or use of a term in this document, the term in this document shall prevail.

[0159] Finally, it should be understood that the embodiments of the application disclosed herein are explanations of the principles of the embodiments of this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are only used as examples and not as limitations. Those skilled in the art can adopt alternative configurations according to the embodiments in this specification to implement the applications in this specification. Therefore, the embodiments of this specification are not limited to the embodiments accurately described in the application.

Claims

1. A data processing method, comprising: Get the target questions sent by users; Obtaining a knowledge graph, wherein the knowledge graph includes a plurality of nodes and edges between the plurality of nodes, wherein the knowledge graph is generated based on documents in a document library, wherein the nodes correspond to entities in the documents, and the edges correspond to relationships between different entities in the documents, wherein the knowledge graph is divided into a plurality of communities, and the closeness between nodes within the same community is greater than the closeness between nodes in different communities; Analyzing the relevance between the target question and each community in the knowledge graph from multiple relevant dimensions to determine at least one target community related to the target question from the knowledge graph, wherein the multiple relevant dimensions include: at least two of a text-related dimension, a semantic-related dimension, and a context-related dimension; and A community summary corresponding to each of the at least one target community is obtained, and answer information corresponding to the target question is generated based on the community summary corresponding to the at least one target community, wherein the community summary corresponding to each target community is a general description of nodes and their relationships in the target community.

2. The method according to claim 1, wherein: The analyzing the relevance between the target problem and each community in the knowledge graph from multiple relevant dimensions to determine at least one target community related to the target problem from the knowledge graph includes: From the text relevance dimension, based on the keywords in the target question, determine the text relevance between each community in the knowledge graph and the target question; From the semantic relevance dimension, based on the semantic information of the target question, determining the semantic relevance between each community in the knowledge graph and the target question; From the context-related dimension, based on the N most recent historical questions before the target question, determine the context-related relevance between each community in the knowledge graph and the target question, where N is an integer greater than 1; and Based on the text relevance, the semantic relevance, and the context relevance respectively corresponding to each community in the knowledge graph, at least one target community is determined in the knowledge graph.

3. The method according to claim 2, wherein: Each element in the community corresponds to an element summary, each element corresponds to a node or an edge, and determining the text relevance between each community in the knowledge graph and the target question based on the keywords in the target question includes: For each community in the knowledge graph, a target element is searched in the community, and based on the number of the searched target elements, a text relevance between the community and the target question is determined, wherein the element summary corresponding to the target element contains the keyword.

4. The method according to claim 2, wherein: The determining, based on the semantic information of the target question, the semantic relevance between each community in the knowledge graph and the target question, includes: For each community in the knowledge graph, a community summary corresponding to the community is obtained, and based on the semantic similarity between the semantic information of the target question and the community summary, the semantic relevance between the community and the target question is determined.

5. The method according to claim 2, wherein: Determining the context relevance between each community in the knowledge graph and the target question based on the N most recent historical questions before the target question from the context relevance dimension includes: For each community in the knowledge graph, K target historical questions are selected from the N historical questions, wherein the community is selected as the target community in the process of answering the target historical questions, and K is an integer greater than or equal to 1, Determine the similarity between the K target historical questions and the target question respectively, and The similarities between the K target historical questions and the target question are weighted and summed to obtain the contextual relevance between the community and the target question, wherein the weights corresponding to the K target historical questions increase gradually with the order of the K target historical questions.

6. The method according to claim 2, wherein: The determining of the at least one target community in the knowledge graph based on the text relevance, the semantic relevance, and the context relevance respectively corresponding to each community in the knowledge graph includes: For each community in the knowledge graph, based on the weights corresponding to the multiple related dimensions, weighted sum of the text relevance, the semantic relevance and the context relevance is performed to determine the comprehensive relevance between the target question and the community; Based on the comprehensive relevances respectively corresponding to the communities in the knowledge graph, k communities with the highest comprehensive relevances are selected as the at least one target community, where k is an integer greater than or equal to 1.

7. The method according to claim 6, wherein: The method further comprises: The weights corresponding to the multiple related dimensions are determined according to the type of the target question or at least one of the preference information of the user for the multiple related dimensions.

8. The method according to claim 1, wherein: The generating answer information corresponding to the target question based on the community summary corresponding to the at least one target community includes: Providing the community summary corresponding to each target community and the target question to the large model, and guiding the large model to generate candidate answers to the target question, thereby obtaining at least one candidate answer; and The at least one candidate answer is aggregated to obtain answer information corresponding to the target question.

9. The method according to claim 8, wherein: The step of aggregating the at least one candidate answer to obtain answer information corresponding to the target question includes: The target question, the at least one candidate answer, the historical questions before the target question, and the historical answers to the historical questions are provided to the big model, and the big model is guided to summarize the at least one candidate answer with reference to the historical questions and the historical answers to generate answer information corresponding to the target question.

10. The method according to claim 1, wherein: The method further includes: determining the type of the target problem; The obtaining of the community summary corresponding to each of the at least one target community, and generating answer information corresponding to the target question based on the community summary corresponding to the at least one target community, includes: When the type representation of the target question inquires about global information, a community summary corresponding to each of the at least one target community is obtained, and answer information corresponding to the target question is generated based on the community summary corresponding to the at least one target community.

11. The method according to claim 10, wherein: The method further comprises: When the type representation of the target question inquires about local information, obtaining an element summary corresponding to each element in the at least one target community, wherein each element corresponds to a node or an edge, and the element summary corresponding to each element is a general description of the element; and Based on the element summaries corresponding to the elements in the at least one target community, answer information corresponding to the target question is generated.

12. The method according to claim 1, wherein: The method further comprises: Extracting a plurality of triples from the documents in the document library, each triple including two entities and a relationship between the two entities; Generating the knowledge graph based on multiple triples; and The knowledge graph is divided into the multiple communities, and community summaries corresponding to the multiple communities are generated respectively.

13. The method according to claim 12, wherein: The extracting a plurality of triples from the documents in the document library includes: Providing the document to the large model, and guiding the large model to extract at least one triple from the document, and adding the at least one triple to a triple set; Repeat the following extraction process until no new triples are identified, or until the upper limit of the number of extractions is reached: provide at least part of the triples in the triple set to the large model, and guide the large model to extract new triples from the document using the at least part of the triples as examples, and add the new triples to the triple set.

14. The method according to claim 12, wherein: The step of dividing the knowledge graph into the plurality of communities and respectively generating community summaries corresponding to the plurality of communities includes: Based on the entities in the knowledge graph and the relationships between different entities, generate an element summary corresponding to each element in the knowledge graph, wherein each element corresponds to a node or an edge, and the element summary corresponding to each element is a general description of the element; Based on the relevance between different elements in the knowledge graph, dividing the knowledge graph into the plurality of communities; and For each community in the plurality of communities, a community summary corresponding to the community is generated based on element summaries corresponding to each element in the community.

15. A data processing system comprising: at least one storage medium storing at least one instruction set for performing data processing; as well as At least one processor is communicatively connected to the at least one storage medium, wherein when the data processing system is running, the at least one processor reads the at least one instruction set and executes the data processing method as described in any one of claims 1-14 according to the instructions of the at least one instruction set.

Images