Multi-agent text information intelligent extraction system, method and equipment based on large language model and medium

Abstract

The invention provides a multi-agent text information intelligent extraction system, method, equipment and medium based on a large language model in order to solve the problem that comprehensive analysis and flexible information extraction of complex texts are difficult to realize while high precision is ensured in the prior art. A multi-agent text information intelligent extraction system based on a large language model comprises an input layer, a task definition module, a multi-agent layer, a central coordinator, a dynamic knowledge base and an output layer. Wherein the multi-agent layer is composed of a plurality of agents, namely a dialogue extraction agent, an entity recognition agent, a relation extraction agent, an emotion analysis agent, an event extraction agent and a context understanding agent. Each agent is responsible for a specific analysis task. Through cooperation of a plurality of specialized agents of the multi-agent layer, the system can identify and extract complex text information more accurately, and the accuracy and comprehensiveness of information extraction are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of complex text information processing and knowledge extraction using a multi-agent system constructed using a large language model. Background Art

[0002] With the rapid development of information technology, the analysis and information extraction of large-scale text data has become an important research direction in the field of natural language processing (NLP). In application scenarios such as literary work analysis, large-scale content research, and knowledge graph construction, the demand for accurately extracting complex information from massive unstructured texts is growing. At present, there are mainly the following methods for text information extraction: rule-based methods, machine learning-based methods, and knowledge graph construction methods based on deep learning.

[0003] A common problem with existing technologies is that it is difficult to achieve comprehensive analysis of complex texts and flexible information extraction while ensuring high accuracy. Summary of the invention

[0004] In order to solve the problem in the prior art that it is difficult to achieve comprehensive analysis of complex texts and flexible information extraction while ensuring high accuracy, the present invention provides a multi-agent text information intelligent extraction system, method, device and medium based on a large language model.

[0005] The technical solution adopted by the present invention is:

[0006] A multi-agent text information intelligent extraction system based on a large language model, comprising:

[0007] The input layer is used to receive and pre-process various types of text data and output the pre-processed text data.

[0008] A task definition module allows users to customize extraction tasks and rules.

[0009] A multi-agent layer, the multi-agent layer is composed of multiple agents; the input end of the multi-agent layer is connected to the output end of the input layer, and the multi-agent layer receives the pre-processed text data output by the input layer; the input end of the multi-agent layer is also connected to the output end of the dynamic knowledge base, and the multi-agent layer receives the reference information output by the dynamic knowledge base; the input end of the multi-agent layer is also connected to the output end of the central coordinator, and the multi-agent layer receives the task control instructions output by the central coordinator; the input end of the central coordinator is connected to the output end of the task definition module, for receiving the data of the task definition module; the input end of the central coordinator is also connected to the output end of the multi-agent layer, for receiving the data of the multi-agent layer; the dynamic knowledge base is used to store and update information extracted from the text; the input end of the dynamic knowledge base is connected to the output end of the output layer, for receiving the data of the output layer; the output layer is used to generate structured extraction results; the input end of the output layer is connected to the output end of the central coordinator, for receiving the integrated data sent by the central coordinator.

[0010] Furthermore, the multiple agents in the multi-agent layer are:

[0011] A dialogue extraction agent is used to identify and extract dialogue content in text.

[0012] An entity recognition agent is used to recognize and classify named entities in text.

[0013] A relationship extraction agent is used to analyze and extract relationships between entities.

[0014] Sentiment analysis type text data is imported into the input layer, the input layer uses an agent, and the sentiment analysis agent is used to analyze the sentiment tendency of the text.

[0015] An event extraction agent is used to identify and extract events described in text.

[0016] A context understanding agent, wherein the context understanding agent is used to analyze context information of a text.

[0017] A multi-agent text information intelligent extraction method based on a large language model, the aforementioned multi-agent text information intelligent extraction system based on a large language model is implemented, comprising the steps of:

[0018] Step S1: importing various types of text data into the input layer, the input layer is used to receive and pre-process various types of text data, and send the pre-processed text data to the multi-agent layer.

[0019] Step S2: The user sends data to the task definition module to define extraction rules and extraction tasks.

[0020] Step S3: The task definition module parses the extraction rules and extraction tasks defined by the user in step S2, determines the agents in the multi-agent layer that need to be enabled, and sends data to the central coordinator.

[0021] Step S4: After receiving the data sent by the task definition module, the central coordinator sends the task control instruction to the corresponding agent in the multi-agent layer, and the task control instruction includes the extraction rules and extraction tasks described in step S3.

[0022] Step S5: First, the corresponding agents in each multi-agent layer simultaneously perform the extraction task on the pre-processed text data sent by the input layer in step S1 according to the extraction rules. Then the corresponding agents in each multi-agent layer send the parsed data to the central coordinator.

[0023] Step S6: The central coordinator collects and integrates the data sent by the corresponding agents in each multi-agent layer, and sends the integrated data to the output layer.

[0024] Step S7: The output layer generates a structured extraction result from the integrated data sent by the central coordinator in step S6, outputs it, and sends it to the dynamic knowledge base. The dynamic knowledge base is used to store and update the information extracted from the text, and the information extracted from the text is the integrated data sent by the central coordinator.

[0025] Furthermore, the method of preprocessing various types of text data in step S1 is: cleaning, segmenting and preliminarily analyzing the input text at the input layer.

[0026] Furthermore, in the task definition module described in step S2, the data sent by the user to the task definition module is also used for task optimization, that is, automatically optimizing the tasks defined by the user, and for expanding the detailed requirements defined by the user.

[0027] Furthermore, after receiving the data sent by the task definition module, the central coordinator sends the task control instruction to the corresponding agent in the multi-agent layer, and specifically performs the following steps:

[0028] Step S1: After receiving the data sent by the task definition module, the central coordinator decomposes the complex extraction task into multiple subtasks.

[0029] Step S2: First, the central coordinator performs dynamic scheduling based on the expertise of each agent, system load and task priority, dynamically allocates the subtasks decomposed in step S1, and then sends the task control instructions to the corresponding agents in the multi-agent layer.

[0030] Furthermore, the dynamic knowledge base in step S7 is also used for:

[0031] Cross-document association, the dynamic knowledge base supports establishing information associations between multiple documents.

[0032] Knowledge reasoning: the dynamic knowledge base performs simple logical reasoning based on existing information.

[0033] Version control: The dynamic knowledge base maintains historical versions of knowledge and supports backtracking and comparison.

[0034] Based on the same inventive concept, the present invention also provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the aforementioned multi-agent text information intelligent extraction method based on a large language model by executing the computer instructions.

[0035] Based on the same inventive concept, the present invention also provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to enable a computer to execute the aforementioned multi-agent text information intelligent extraction method based on a large language model.

[0036] The beneficial effects of the present invention are:

[0037] Through the collaboration of multiple specialized agents in the multi-agent layer, the system can more accurately identify and extract complex text information such as dialogues, entities, relations, emotions, and events, improving the accuracy and comprehensiveness of information extraction.

[0038] Through the task definition module, users can customize extraction tasks according to their needs. The plug-in design of the intelligent agent enables the system to easily adapt to new text types and extraction tasks, enhancing the flexibility and scalability of the system.

[0039] By adopting a parallel processing architecture, multiple agents work simultaneously, significantly improving the efficiency of processing complex texts. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 A block diagram of the system architecture.

[0042] Figure 2 The results extracted for the conversation.

[0043] Figure 3 The result of entity recognition.

[0044] Figure 4 The result of relation extraction.

[0045] Figure 5 The result of sentiment analysis.

[0046] Figure 6 The results extracted for the event.

[0047] Figure 7 As a result of contextual understanding. DETAILED DESCRIPTION

[0048] The present invention solves the problem in the prior art that it is difficult to achieve comprehensive analysis of complex texts and flexible information extraction while ensuring high accuracy, and provides a multi-agent text information intelligent extraction system, method, device and medium based on a large language model. Through the collaboration of multiple specialized agents in the multi-agent layer, the system can more accurately identify and extract complex text information, such as dialogues, entities, relationships, emotions and events, thereby improving the accuracy and comprehensiveness of information extraction; through the task definition module, users can customize extraction tasks according to their needs, and the plug-in design of the agent enables the system to easily adapt to new text types and extraction tasks, thereby enhancing the flexibility and scalability of the system; and by adopting a parallel processing architecture, multiple agents work simultaneously, significantly improving the efficiency of processing complex texts.

[0049] The embodiments provided in the following disclosure are merely examples and are not intended to limit the present invention.

[0050] The embodiments of the invention are described in detail below with reference to the accompanying drawings. Figure 1 A block diagram of the system architecture.

[0051] A multi-agent text information intelligent extraction system based on a large language model, comprising:

[0052] The input layer is used to receive and pre-process various types of text data and output the pre-processed text data.

[0053] A multi-agent layer, the multi-agent layer is composed of multiple agents; the input end of the multi-agent layer is connected to the output end of the input layer, and the multi-agent layer receives the pre-processed text data output by the input layer; the input end of the multi-agent layer is also connected to the output end of the dynamic knowledge base, and the multi-agent layer receives the reference information output by the dynamic knowledge base; the input end of the multi-agent layer is also connected to the output end of the central coordinator, and the multi-agent layer receives the task control instructions output by the central coordinator; the input end of the central coordinator is connected to the output end of the task definition module, for receiving the data of the task definition module; the input end of the central coordinator is also connected to the output end of the multi-agent layer, for receiving the data of the multi-agent layer; the dynamic knowledge base is used to store and update information extracted from the text; the input end of the dynamic knowledge base is connected to the output end of the output layer, for receiving the data of the output layer; the output layer is used to generate structured extraction results; the input end of the output layer is connected to the output end of the central coordinator, for receiving the integrated data sent by the central coordinator.

[0054] The following is an explanation of the structured format: structured means that there is a standard output format, which is conducive to program parsing or management and the output format used in the next stage. Here it mainly refers to a json format. For example:

[0055] {"name":"Mazu Niangniang","type":"CULTURAL_TERM","description":["A sea god worshipped in coastal areas of China, one of the gods enshrined in the restaurant."],"original_text": ["The eight-faced god and Mazu Niangniang are enshrined on the wall, and the incense is constantly burning."]}. The fields inside, such as name, type, description, and original_text are the content that needs to be output in advance. Structuring means that for each target output, the format is composed in this way, and the output content is also displayed under the defined fields, which is convenient for use.

[0056] Furthermore, the multiple agents in the multi-agent layer are:

[0057] A dialogue extraction agent is used to identify and extract dialogue content in text.

[0058] An entity recognition agent is used to recognize and classify named entities in text.

[0059] A relationship extraction agent is used to analyze and extract relationships between entities.

[0060] The sentiment analysis agent is used to analyze the sentiment tendency of the text.

[0061] An event extraction agent is used to identify and extract events described in text.

[0062] A context understanding agent, wherein the context understanding agent is used to analyze context information of a text.

[0063] A multi-agent text information intelligent extraction method based on a large language model, the aforementioned multi-agent text information intelligent extraction system based on a large language model, comprising the steps of:

[0064] Step S1: importing various types of text data into the input layer, the input layer is used to receive and pre-process various types of text data, and send the pre-processed text data to the multi-agent layer.

[0065] Step S2: The user sends data to the task definition module to define extraction rules and extraction tasks.

[0066] The extraction rules are defined, that is, users can define what specific content needs to be extracted. For example, if users need to extract all the psychological activities of the characters in the text, this psychological activity is equal to the defined rules. If the definition requires the extraction of all metaphors, then the category of metaphor is the defined extraction rule. It can be understood that users can customize their requirements.

[0067] Step S3: The task definition module parses the extraction rules and extraction tasks defined by the user in step S2, determines the agents in the multi-agent layer that need to be enabled, and sends data to the central coordinator.

[0068] Step S4: After receiving the data sent by the task definition module, the central coordinator sends the task control instruction to the corresponding agent in the multi-agent layer, and the task control instruction includes the extraction rules and extraction tasks described in step S3.

[0069] Step S5: First, the corresponding agents in each multi-agent layer simultaneously perform the extraction task on the pre-processed text data sent by the input layer in step S1 according to the extraction rules. Then the corresponding agents in each multi-agent layer send the parsed data to the central coordinator.

[0070] Step S6: The central coordinator collects and integrates the data sent by the corresponding agents in each multi-agent layer, and sends the integrated data to the output layer.

[0071] Step S7: The output layer generates a structured extraction result from the integrated data sent by the central coordinator in step S6, outputs it, and sends it to the dynamic knowledge base. The dynamic knowledge base is used to store and update the information extracted from the text, and the information extracted from the text is the integrated data sent by the central coordinator.

[0072] Furthermore, the method of preprocessing various types of text data in step S1 is: cleaning, segmenting and preliminarily analyzing the input text at the input layer.

[0073] Furthermore, in the task definition module described in step S2, the data sent by the user to the task definition module is also used for task optimization, that is, automatically optimizing the tasks defined by the user, and for expanding the detailed requirements defined by the user.

[0074] Here is an example to explain what is the task optimization:

[0075] The requirements raised by users are often very simple, such as: "I want the text of all the narrations."

[0076] But this cannot be used directly by the system. The system intelligently optimizes this sentence based on different input texts and internal training samples, and expands the detailed description of this requirement.

[0077] After optimization:

[0078] I want all the narration content. Narration is a narrative technique. It refers to the method of using a voice or text that does not appear in the picture to explain the plot, describe the scene, reveal the character's inner activities or provide background information in film and television works, literary works or other media.

[0079] Types include:

[0080] 1. First-person narration

[0081] 2. Third-person narration

[0082] 3. Character Inner Monologue

[0083] 4. Voice-over narration

[0084] Features are:

[0085] 1. Provide background information and context

[0086] 2. Advance the plot

[0087] 3. Reveal the inner world of the character

[0088] 4. Create atmosphere and emotional tone

[0089] 5. Connect different scenes or timelines

[0090] 6. Explain complex concepts or events

[0091] Note of potential confusion:

[0092] 1. The difference between narration and dialogue: narration is usually not a direct conversation between characters

[0093] 2. The difference between narration and visual / text description: Narration is an additional narrative layer, not directly visible / readable content.

[0094] 3. Balance between subjectivity and objectivity: Narration can be objective or subjective, depending on specific needs.

[0095] 4. Explicit vs. Implicit Choice: Narration can be an explicit voice or a more subtle narrative.

[0096] Furthermore, after receiving the data sent by the task definition module, the central coordinator sends the task control instruction to the corresponding agent in the multi-agent layer, and specifically performs the following steps:

[0097] Step S1: After receiving the data sent by the task definition module, the central coordinator decomposes the complex extraction task into multiple subtasks.

[0098] Step S2: First, the central coordinator performs dynamic scheduling based on the expertise of each agent, system load and task priority, dynamically allocates the subtasks decomposed in step S1, and then sends the task control instructions to the corresponding agents in the multi-agent layer.

[0099] Furthermore, the dynamic knowledge base in step S7 is also used for:

[0100] Cross-document association, the dynamic knowledge base supports establishing information associations between multiple documents.

[0101] Knowledge reasoning: the dynamic knowledge base performs simple logical reasoning based on existing information.

[0102] Version control: The dynamic knowledge base maintains historical versions of knowledge and supports backtracking and comparison.

[0103] The following is a detailed explanation of the knowledge reasoning: The dynamic knowledge base uses the stored knowledge associated with semantic similarity reasoning as the reference information of the intelligent agent. Semantic matching means, for example, "Have you eaten / Have you eaten breakfast?", "Breakfast" is the reference information, and the latter is more likely to make people answer. "Have you eaten" queries "breakfast", the two are not word-related, but have a semantic association. Similarly, the semantics associated with "Have you eaten" may include "breakfast", "lunch", "dinner", "food", "greetings", etc...., but this is an existing algorithm for the dynamic knowledge base, and it depends on what is found.

[0104] Based on the same inventive concept, this embodiment also provides a computer device, including: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the aforementioned multi-agent text information intelligent extraction method based on a large language model by executing the computer instructions.

[0105] Based on the same inventive concept, this embodiment also provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to enable a computer to execute the aforementioned multi-agent text information intelligent extraction method based on a large language model.

[0106] This example demonstrates the performance of a multi-agent text information intelligent extraction system based on a large language model when processing a science fiction text.

[0107] Enter text: Emily walked into the laboratory and found Jack debugging a complex instrument. "You're just in time," Jack looked up and said, "We need to calibrate the quantum tunneling device again." Emily nodded, thinking that this might be another long night. She walked to the console and began to enter the parameters. Suddenly, the alarm sounded and the red light flashed. "What's going on?" Emily exclaimed. Jack quickly checked the readings and his face turned pale. "The quantum loss is beyond expectations," he whispered, "We may have triggered a space-time rift." Emily felt dizzy, and she knew they were facing an unprecedented crisis. "We have to stabilize the system immediately," she said firmly, "otherwise the consequences will be disastrous." Jack nodded in agreement, and the two immediately plunged into the intense work.

[0108] Output results: The system conducts a comprehensive analysis of the input text and generates the following six aspects of structured information: dialogue extraction, entity recognition, relationship extraction, sentiment analysis, event extraction, and context understanding.

[0109] The results of dialogue extraction are shown in the attached Figure 2 As shown;

[0110] The results of entity recognition are shown in the attached Figure 3 As shown;

[0111] The results of relationship extraction are shown in the attached Figure 4 As shown;

[0112] The results of sentiment analysis are shown in the attached Figure 5 As shown;

[0113] The results of the event are as follows Figure 6 As shown;

[0114] The results of context understanding are shown in the attached Figure 7 shown.

[0115] Through this example, we can see that the system of the present invention can extract rich structured information from a short science fiction text. The system's multi-faceted analysis capabilities demonstrate its significant application value in the fields of literary analysis, large-scale content research, and knowledge graph construction. Specifically:

[0116] 1) In terms of literary analysis, the system not only accurately identifies the content of dialogue and the speaker, but also deeply captures the emotional changes of the characters, providing a solid foundation for character creation and plot development analysis.

[0117] 2) For large-scale content research, the system can quickly extract key entities, event sequences, and contextual information, greatly improving the efficiency and accuracy of text processing.

[0118] 3) In terms of knowledge graph construction, the system provides rich structured data for building complex, multi-dimensional knowledge networks by analyzing the relationships between entities and scene contexts.

[0119] These features together form an efficient, accurate, and flexible text information intelligent extraction solution. The innovative design of this system can not only meet current research needs, but also promote the further development of text analysis and information extraction technology. It provides new tools and methods for researchers and application developers in related fields, which may inspire more innovative applications and research directions, and bring new development opportunities to the entire field.

Claims

1. A multi-agent text information intelligent extraction system based on a large language model, characterized in that: include: An input layer, the input layer is used to receive and pre-process various types of text data, and output the pre-processed text data; A multi-agent layer, wherein the multi-agent layer is composed of a plurality of agents; The input end of the multi-agent layer is connected to the output end of the input layer, and the multi-agent layer receives the pre-processed text data output by the input layer; The input end of the multi-agent layer is also connected to the output end of the dynamic knowledge base, and the multi-agent layer receives the reference information output by the dynamic knowledge base; The input end of the multi-agent layer is also connected to the output end of the central coordinator, and the multi-agent layer receives the task control instructions output by the central coordinator; the input end of the central coordinator is connected to the output end of the task definition module, and is used to receive the data of the task definition module; the input end of the central coordinator is also connected to the output end of the multi-agent layer, and is used to receive the data of the multi-agent layer; The dynamic knowledge base is used to store and update information extracted from the text; the input end of the dynamic knowledge base is connected to the output end of the output layer, and is used to receive data from the output layer; The output layer is used to generate structured extraction results; the input end of the output layer is connected to the output end of the central coordinator, and is used to receive the integrated data sent by the central coordinator.

2. According to claim 1, a multi-agent text information intelligent extraction system based on a large language model is characterized in that: The multiple agents in the multi-agent layer are: A dialogue extraction agent, wherein the dialogue extraction agent is used to identify and extract dialogue content in the text; An entity recognition agent, the entity recognition agent is used to recognize and classify named entities in text; A relationship extraction agent, wherein the relationship extraction agent is used to analyze and extract the relationship between entities; A sentiment analysis agent, wherein the sentiment analysis agent is used to analyze the sentiment tendency of the text; An event extraction agent, wherein the event extraction agent is used to identify and extract events described in the text; A context understanding agent, wherein the context understanding agent is used to analyze context information of a text.

3. A multi-agent text information intelligent extraction method based on a large language model, which is implemented based on the multi-agent text information intelligent extraction system based on a large language model described in claim 2, characterized in that: Includes steps: Step S1: importing various types of text data into the input layer, the input layer is used to receive and pre-process various types of text data, and send the pre-processed text data to the multi-agent layer; Step S2: The user sends data to the task definition module to define extraction rules and extraction tasks; Step S3: the task definition module parses the extraction rules and extraction tasks defined by the user in step S2, determines the agents in the multi-agent layer that need to be enabled, and sends data to the central coordinator; Step S4: After receiving the data sent by the task definition module, the central coordinator sends the task control instruction to the corresponding agent in the multi-agent layer, and the task control instruction includes the extraction rule and the extraction task in step S3; Step S5: First, the corresponding agents in each multi-agent layer simultaneously perform the extraction task on the pre-processed text data sent by the input layer in step S1 according to the extraction rules; then the corresponding agents in each multi-agent layer send the parsed data to the central coordinator; Step S6: The central coordinator collects and integrates the data sent by the corresponding agents in each multi-agent layer, and sends the integrated data to the output layer. Step S7: the output layer generates a structured extraction result from the integrated data sent by the central coordinator in step S6, outputs it, and sends it to the dynamic knowledge base at the same time; The dynamic knowledge base is used to store and update information extracted from text, and the information extracted from text is integrated data sent by the central coordinator.

4. According to claim 3, a multi-agent text information intelligent extraction method based on a large language model is characterized in that: The method of preprocessing various types of text data in step S1 is: cleaning, segmenting and preliminarily analyzing the input text at the input layer.

5. According to claim 3, a multi-agent text information intelligent extraction method based on a large language model is characterized in that: In the task definition module of step S2, the data sent by the user to the task definition module is also used for task optimization, that is, automatically optimizing the tasks defined by the user, and for expanding the detailed requirements defined by the user.

6. The multi-agent text information intelligent extraction method based on a large language model according to claim 3 is characterized in that: After receiving the data sent by the task definition module, the central coordinator sends the task control instruction to the corresponding agent in the multi-agent layer, and specifically performs the following steps: Step S41: After receiving the data sent by the task definition module, the central coordinator decomposes the complex extraction task into multiple subtasks; Step S42: First, the central coordinator performs dynamic scheduling based on the expertise of each agent, system load and task priority, dynamically allocates the subtasks decomposed in step S41, and then sends the task control instructions to the corresponding agents in the multi-agent layer.

7. The multi-agent text information intelligent extraction method based on a large language model according to claim 3 is characterized in that: The dynamic knowledge base described in step S7 is also used for: The dynamic knowledge base supports establishing information associations between multiple documents, performing simple logical reasoning based on existing information, and maintaining historical versions of knowledge, supporting backtracking and comparison.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the multi-agent text information intelligent extraction method based on a large language model as described in any one of claims 3 to 7 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute a multi-agent text information intelligent extraction method based on a large language model as described in any one of claims 3 to 7.

Images