Long-text-oriented passage-level knowledge extraction method, system, device and medium

Abstract

This application relates to a method, system, device, and medium for text-level knowledge extraction from long texts, belonging to the field of knowledge extraction technology. The method includes: inputting diverse heterogeneous data, parsing it into text according to a format, and then performing language identification and translation; connecting to a knowledge platform to obtain and structure entity information, and generating large language model prompt words based on prompt word templates; designing example data format conversion sample data, inputting it into the model to generate extraction results, parsing it into RDF triples, and performing entity alignment and disambiguation. This application can accurately extract knowledge from diverse heterogeneous long texts, solve the problem of text fragmentation, and also achieve entity alignment and disambiguation, thereby improving the consistency and quality of knowledge graphs.

Description

Technical Field

[0001] This application relates to the field of knowledge extraction technology, and in particular to a method, system, device and medium for extracting chapter-level knowledge from long texts. Background Technology

[0002] Knowledge graphs, as structured semantic knowledge bases, are widely used in fields such as intelligent search, recommendation systems, big data analysis, and decision support. Their core construction lies in knowledge extraction, specifically extracting entities, relationships, and attributes from unstructured text to form triples of knowledge. Currently, the internet and various industry databases contain massive amounts of long-form text data, such as academic literature and clinical medical records. These texts are lengthy, structurally complex, and contain scattered information with numerous semantic connections across sentences and paragraphs.

[0003] To address the problem of knowledge extraction from text, several conventional methods exist. These include pipelined discrete extraction methods, which sequentially execute entity recognition, relation extraction, and attribute extraction—first identifying entities, then determining relations, and finally extracting attributes; end-to-end methods based on joint extraction, which synchronously extract entities and relations within the same model by sharing parameters and specific labeling strategies, mitigating error propagation; general methods based on large language models, leveraging their powerful natural language understanding capabilities and employing cueing engineering to extract triples; and joint knowledge extraction methods based on pre-trained language models and ontology constraints, using a shared pre-trained language model as an encoder and introducing rules or soft constraints to utilize ontology information during decoding or post-processing.

[0004] However, existing technologies still have significant drawbacks. Pipeline methods suffer from error propagation, neglect task relevance, and have weak document-level processing capabilities; joint extraction methods are inefficient and ineffective for long texts, have limited semantic depth, and struggle to incorporate prior knowledge; general methods based on large language models lack controllability and stability, are costly, and lack domain adaptability; and joint methods combining pre-trained language models and ontology constraints are unsuitable for long texts, make shallow use of ontology information, ignore document structure, present a trade-off between efficiency and accuracy, and cannot achieve true document-level knowledge extraction. Summary of the Invention

[0005] The purpose of this application is to provide a chapter-level knowledge extraction method for long texts, which can accurately extract knowledge from diverse and heterogeneous long texts, solve the problem of text fragmentation, and also achieve entity alignment and disambiguation, thereby improving the consistency and quality of knowledge graphs.

[0006] Firstly, this application provides a chapter-level knowledge extraction method for long texts, which adopts the following technical solution: A text-level knowledge extraction method for long texts includes: Input diverse and heterogeneous data, and output text data based on the file format of the data and the corresponding parsing rules; The text data is then language-identified and translated into preprocessed text that can be interpreted by a large language model. By connecting to a pre-defined knowledge platform, the knowledge ontology data containing entity information of the identifier of the specified project is obtained and parsed, and a structured representation of the knowledge ontology data is constructed. Based on the structured knowledge ontology data and the preset prompt word template, a large language model prompt word based on the knowledge extraction framework format is generated; Based on the knowledge extraction framework, an example data format is designed to convert the structured labeled sample data of the knowledge platform into example data used by the knowledge extraction framework. Receive the prompt words, example data and preprocessed text, and generate extraction result fields based on the knowledge extraction framework and large language model; Parse the extracted result fields and convert them into structured RDF triples; Entity alignment and disambiguation are performed on the RDF triples.

[0007] By adopting the above technical solution, firstly, diverse and heterogeneous data is input and corresponding parsing rules are used to output text data, solving the problem of converting multi-source data into unified text, thereby expanding the system's applicability. Based on this, language identification and translation are performed on the text data, providing a standard-based and unified context. Next, knowledge ontology data is acquired and parsed to construct a structured representation, providing a basis for subsequent prompt word generation and RDF generation. Based on knowledge ontology data and prompt word templates, large language model prompt words are generated, achieving customization and high-precision extraction for specific domains and specific ontology. Simultaneously, example data formats are designed and the number of labeled samples is converted. According to the data, the preparation cost of few-sample learning is reduced, thereby improving the system startup efficiency. Subsequently, based on the knowledge extraction framework and large language model, the extraction result fields are generated, which can realize chapter-level knowledge understanding and extraction. Finally, the extraction result fields are parsed and converted into structured RDF triples to ensure that the extraction results meet the injection requirements of the knowledge graph. Entity alignment and disambiguation of the RDF triples are performed to improve the data quality and consistency of the final knowledge graph. Ultimately, the automated processing from multivariate heterogeneous data to structured RDF triples is realized, solving the problems of fragmentation, structural loss, and the contradiction between efficiency and accuracy in long text chapter-level knowledge extraction.

[0008] In a preferred embodiment, this application can be further configured as follows: the step of connecting to a preset knowledge platform, obtaining and parsing knowledge ontology data containing entity information of the identifier based on the identifier of a specified project, and constructing a structured representation of the knowledge ontology data includes: Obtain the knowledge ontology data of the identifier. The knowledge ontology data is returned in JSON format and defines entity information, including class information of the category to which the entity belongs, attribute information of the characteristics or properties of the entity, and semantic relationship information between entities. The knowledge ontology data is parsed, and a structured representation of the knowledge ontology data is constructed in memory.

[0009] By adopting the above technical solution, it is possible to acquire and parse knowledge ontology data containing entity categories, attributes and relationship information of a specified project identifier, return the knowledge ontology data in JSON format, and construct its structured representation in memory, providing a basis for subsequent prompt word generation and RDF generation. This helps to achieve deep integration of knowledge graph ontology and text semantics, improve the accuracy and standardization of extraction results, and avoid illogical and absurd output.

[0010] In a preferred embodiment, this application can be further configured as follows: the knowledge extraction framework is the LangExtract framework, and the step of generating large language model prompts based on the structured knowledge ontology data and preset prompt templates includes: Define the prompt word template, which includes the output format requirements for the result fields extracted based on the LangExtract framework; The entity information from the knowledge ontology data is filled into the prompt word template to generate a structured task description; Based on the output format requirements, task description, ontology constraints of the LangExtract framework, and preprocessed text, the prompt words are generated.

[0011] By adopting the above technical solution, a prompt word template containing output format requirements is defined. Entity information from the knowledge ontology data is filled into the template to generate a structured task description. Then, combined with the output format requirements, task description, ontology constraints of the LangExtract framework, and preprocessed text, prompt words are generated. This allows the large language model to follow preset domain norms and logical constraints when performing knowledge extraction tasks, fundamentally improving the accuracy and standardization of extraction results, avoiding illogical and absurd outputs, and by strictly defining the output format, the subsequent parsing and RDF conversion processes can be completed automatically and accurately without complex post-processing.

[0012] In a preferred example, this application can be further configured as follows: the step of receiving the prompt words, example data, and preprocessed text, and generating extraction result fields based on the knowledge extraction framework and the large language model, includes: Based on the LangExtract framework, the large language model is invoked to perform discourse-level reasoning on the preprocessed text according to the instructions of the prompt words and example data, and the extracted result fields that meet the output format requirements of the prompt words are output.

[0013] By adopting the above technical solution and using the LangExtract framework to call a large language model to perform discourse-level reasoning on preprocessed text, the problem of text fragmentation can be effectively solved, and discourse-level knowledge understanding and extraction can be achieved. This enables the accurate extraction of entities, relationships, and attributes scattered in different locations in the text, ensuring the completeness and coherence of the extracted knowledge. Furthermore, by strictly adhering to the output format requirements for the extracted result fields, subsequent automated parsing is simplified, the reliability of automatically extracting structured information from free text is improved, and the contradiction between efficiency and accuracy is alleviated.

[0014] In a preferred embodiment, this application can be further configured such that the step of parsing the extracted result fields and converting them into structured RDF triples includes: Based on the output format requirements, the extracted result fields are parsed line by line to identify the text entity information in each line; The text entity information is encapsulated into a structured JSON object; Traverse the JSON objects and convert the JSON entries in each JSON object into RDF triples based on the class information and attribute information URIs defined in the knowledge ontology data.

[0015] By adopting the above technical solution, the extracted result fields can be parsed line by line according to the output format requirements to accurately identify text entity information, encapsulate the information into JSON objects to achieve structured organization, and convert JSON entries into RDF triples according to the URIs of class and attribute information defined in the knowledge ontology data. The extracted results are then injected into the knowledge graph to ensure the accuracy and reliability of the data results.

[0016] In a preferred embodiment, this application can be further configured such that the step of performing entity alignment and disambiguation on the RDF triples includes: Extract entity sets from the RDF triples, filter entities of the same type based on the class information defined by the knowledge ontology data, and combine the entities of the same type into pairs of entities to be merged. Input each pair of entities to be merged and its corresponding set of attributes, and calculate the attribute similarity of each pair of entities to be merged. Based on preset weight values, the attribute similarity is weighted and summed to calculate the similarity score of each entity pair to be merged.

[0017] By adopting the above technical solution, entities are extracted from RDF triples and pairs of entities of the same type are screened, and then entities of the same type are processed in a targeted manner. The similarity of attributes is calculated to measure the degree of entity similarity, and combined with the weighted summation of the importance of knowledge ontology attributes, making the similarity measurement more domain-specific and interpretable. This effectively solves the ambiguity problem of entities with the same name but different meanings and entities with the same meaning but different names, and improves the quality and consistency of knowledge graph data.

[0018] In a preferred embodiment, this application can be further configured such that the step of performing entity alignment and disambiguation on the RDF triples includes: Extract entity sets from the RDF triples, filter entities of the same type based on the class information defined by the knowledge ontology data, and combine the entities of the same type into pairs of entities to be merged. Input the entity pair to be merged and its corresponding relationship network, analyze and obtain the relationship set of each entity in the entity pair to be merged, and determine whether each entity in the entity pair to be merged has a common relationship with the same third entity; The number of common relationships is counted, where the number of common relationships is positively correlated with the entity similarity score.

[0019] By adopting the above technical solutions, entities of the same type are filtered based on knowledge ontology information, the similarity is measured by analyzing the number of common relationships between entities and third entities, and entity alignment and disambiguation are performed on RDF triples to avoid the problem of entities having the same name but different meanings, or having the same meaning but different names, affecting the construction of knowledge graphs and improving data quality and consistency.

[0020] Secondly, this application provides a chapter-level knowledge extraction system for long texts, employing the following technical solution: A text-level knowledge extraction system for long texts includes: Data preprocessing module: used to input diverse and heterogeneous data, and output text data based on the file format of the data and the corresponding parsing rules; Language identification and translation module: used to identify the language of the text data and translate it into preprocessed text that can be interpreted by a large language model; Knowledge ontology acquisition and parsing module: used to connect to the preset knowledge platform, acquire and parse the knowledge ontology data containing entity information of the identifier of the specified project, and construct the structured representation of the knowledge ontology data; Prompt word generation module: used to generate large language model prompt words based on the structured knowledge ontology data and preset prompt word templates; Example data generation module: used to design example data format based on the knowledge extraction framework, and convert the structured labeled sample data of the knowledge platform into example data used by the knowledge extraction framework; Knowledge extraction engine: used to receive the prompt words, example data and preprocessed text, and generate extraction result fields based on the knowledge extraction framework and large language model; RDF triple generation module: used to parse the extracted result fields and convert them into structured RDF triples; Knowledge fusion module: used to perform entity alignment and disambiguation on the RDF triples.

[0021] Thirdly, this application provides an electronic device that adopts the following technical solution: An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method for extracting chapter-level knowledge from long texts.

[0022] Fourthly, this application provides a computer storage medium, as follows: A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method for extracting chapter-level knowledge from long texts.

[0023] In summary, this application has the following beneficial technical effects: 1. This application uses a large language model and the LangExtract framework to perform text-level semantic analysis on diverse and heterogeneous data of different file formats after parsing and processing, extracting entities, relations and attributes across sentences and paragraphs, avoiding fragmentation of extraction results and ensuring knowledge integrity and coherence. 2. This application generates prompt words based on structured knowledge ontology data and prompt word templates, enabling the large language model to follow domain norms and logical constraints when processing text, deeply integrating knowledge graph ontology and text semantics, and improving the accuracy and standardization of extraction results; 3. This application utilizes a locally deployed large language model, combined with a pre-defined, strictly defined output format, to automate parsing and RDF conversion. This ensures high precision while accurately and quickly processing long texts, thus alleviating the conflict between efficiency and accuracy. Attached Figure Description

[0024] Figure 1 This is a flowchart of a chapter-level knowledge extraction method for long texts in one embodiment of this application.

[0025] Figure 2This is a flowchart of a sub-step of step S3 in one embodiment of this application.

[0026] Figure 3 This is a flowchart of a sub-step of step S4 in one embodiment of this application.

[0027] Figure 4 This is a flowchart of a sub-step of step S6 in one embodiment of this application.

[0028] Figure 5 This is a flowchart of a sub-step of step S7 in one embodiment of this application.

[0029] Figure 6 This is a sub-step of step S8 in one embodiment of this application. Figure 1 .

[0030] Figure 7 This is a sub-step of step S8 in one embodiment of this application. Figure 2 .

[0031] Figure 8 This is a schematic block diagram of an electronic device in one embodiment of this application. Detailed Implementation

[0032] The following is in conjunction with the appendix Figure 1-8 This application will be described in further detail.

[0033] It should be noted that, in the embodiments of this invention, the data related to object information, when applied to specific products or technologies, requires the permission or consent of the object. Furthermore, the collection, use, and processing of this data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. In other words, if the embodiments of this invention involve data related to an object, it must be obtained with the object's authorization and consent, the authorization and consent of relevant departments, and in accordance with the relevant laws, regulations, and standards of the country and region. If the embodiments involve personal information, the acquisition of all personal information requires the individual's consent. If sensitive information is involved, the separate consent of the information subject is required. The embodiments also need to be implemented with the object's authorization and consent.

[0034] refer to Figure 1 A text-level knowledge extraction method for long texts, specifically including: S1. Input multi-variable heterogeneous data, and output text data based on the file format of the data and the corresponding parsing rules.

[0035] Specifically, to address the issue of inconsistent formats in heterogeneous input data, a format support unit is provided. This unit can directly parse and support structured or semi-structured text formats, including but not limited to Word (.docx), PDF (text), TXT, and HTML, and extracts text content and its basic structural information (such as paragraphs and headings) by calling the corresponding parsing libraries (e.g., python-docx for Word, PyPDF2 or pdfplumber for PDF).

[0036] For data from which text cannot be directly extracted, an optical character recognition (OCR) unit is included. When the source of diverse and heterogeneous data is scanned PDFs or images (.jpg, .png, etc.), the Tesseract-OCR engine is integrated to perform optical character recognition, converting image information into machine-readable text. The process includes image preprocessing (such as binarization and noise reduction), layout analysis, character recognition, and post-correction.

[0037] For binary files, a binary file parsing unit is provided. This unit analyzes the binary file header or specific format specifications, deserializes the binary file into an in-memory data structure, and extracts valid, understandable Chinese text data from it. For example, for a specific log file or a binary stream with a custom format, log messages or text fields are extracted using predefined parsing rules.

[0038] Furthermore, the predefined parsing rules are mainly targeted at specific log files, such as the format of the log file, such as the prefix Info / Warning / Error / Debug, etc. The rules are broken down and extracted for this format, so that the log file is written as structured text, such as log level, log time, log content and log occurrence point.

[0039] S2. Perform language identification on the text data and translate it into preprocessed text that can be interpreted by a large language model.

[0040] Specifically, in this embodiment, the text data needs to be processed to be within the same Simplified Chinese context. In the processing flow of the original text data, firstly, plain text data is received, and then a language detection library (such as langdetect) is called to automatically identify the language of the text data. If the identified language is not Simplified Chinese, a locally deployed large language model is used for text translation. A prompt containing the original text and the instruction "Please translate the following content into Simplified Chinese" is constructed, passed to the LLM for processing, and the returned Simplified Chinese translation is obtained. The final unified Simplified Chinese text is then output to the downstream module.

[0041] Therefore, through its inherent logical process, the four heterogeneous processing steps of format parsing, OCR recognition, binary deserialization, and language unification are seamlessly integrated into a coherent technical whole, realizing an end-to-end automated data preprocessing pipeline. This enables the unified processing pipeline of multi-source heterogeneous data to automatically process various data sources, from electronic documents to scanned copies and binary files, thereby greatly expanding the applicability of the system.

[0042] S3. Connect to the preset knowledge platform, obtain and parse the knowledge ontology data containing entity information of the identifier based on the identifier of the specified project, and construct a structured representation of the knowledge ontology data.

[0043] Specifically, the knowledge graph ontology serves as a blueprint for knowledge extraction and resides within the knowledge platform. In this embodiment, the system interfaces with the knowledge platform via an application programming interface (API) to obtain complete knowledge ontology data based on the identifier of a specified project, providing a basis for subsequent prompt word generation and RDF generation.

[0044] S4. Based on structured knowledge ontology data and preset prompt word templates, generate prompt words for a large language model based on a knowledge extraction framework format.

[0045] Specifically, a locally deployed large language model is used, with strict format constraints to ensure subsequent automated parsing, and this model is included in the prompt words. Therefore, generating large language model prompt words based on knowledge ontology data and prompt word templates can achieve customization and high-precision extraction for specific domains and specific ontologies.

[0046] S5. Based on the knowledge extraction framework, design an example data format to convert the structured labeled sample data of the knowledge platform into example data used by the knowledge extraction framework.

[0047] Specifically, in this embodiment, the LangExtract framework is used as the knowledge extraction framework. During extraction, example data is required for few-shot learning. To adapt to the LangExtract framework, an example data format compatible with it is designed, typically consisting of an input text corresponding to an extracted result text that meets the output format requirements. The system accesses pre-generated labeled sample data through a visual annotation tool within the knowledge platform and stores it in a structured format. The stored structured annotations are transformed and encapsulated according to the designed example data format to generate example pairs that can be directly used by the LangExtract framework. This eliminates compatibility issues between data formats across different systems, allowing existing annotation results to be directly reused, reducing the preparation cost for few-shot learning, and thus improving system startup efficiency.

[0048] S6. Receive prompt words, sample data, and preprocessed text, and generate extraction result fields based on the knowledge extraction framework and large language model.

[0049] Specifically, based on the knowledge extraction framework, it calls the local large language model for reasoning. LLM can perform discourse-level understanding of long texts according to strict instructions in examples and prompts, and output the extracted result field text in a predetermined format.

[0050] S7. Parse the extracted result fields and convert them into structured RDF triples.

[0051] Specifically, following preset format rules, the extracted result field text is parsed line by line, and the parsed information is structured into JSON. Each JSON entry in the JSON object is then converted into a standard RDF triple to ensure that the extracted results meet the injection requirements of the knowledge graph.

[0052] S8. Perform entity alignment and disambiguation on RDF triples.

[0053] Specifically, entity alignment and disambiguation of RDF triples improve the data quality and consistency of the final knowledge graph, effectively solving the ambiguity problem of entities with the same name but different meanings, and entities with the same meaning but different names. Ultimately, it realizes the automated processing of multi-dimensional heterogeneous data into structured RDF triples, solving the problems of fragmentation, structural loss, and the contradiction between efficiency and accuracy in long text chapter-level knowledge extraction.

[0054] refer to Figure 2 Furthermore, in one embodiment, step S3 is refined into the following sub-steps: S30. Obtain the knowledge ontology data of the identifier. The knowledge ontology data is returned in JSON format and defines entity information, including class information of the category to which the entity belongs, attribute information of the characteristics or properties of the entity, and relationship information of semantic relationships between entities.

[0055] Specifically, class information is used to standardize the classification system of entities, attribute information is used to describe the inherent characteristics or state of entities, and relationship information is used to characterize the semantic associations between entities. By defining the domain and value range of relationships, it is clear which entities can establish what kind of relationship, thereby constructing a meaningful semantic network between entities. The above entity information is returned from the knowledge platform system in JSON format.

[0056] S31. Parse the knowledge ontology data and construct a structured representation of the knowledge ontology data in memory.

[0057] Specifically, parsing JSON-formatted knowledge ontology data and constructing a structured representation of the ontology in memory provides a basis for subsequent prompt word generation and RDF generation, which helps to achieve deep integration of knowledge graph ontology and text semantics, improve the accuracy and standardization of extraction results, and avoid illogical and absurd outputs.

[0058] In addition, refer to Figure 3 Furthermore, in one embodiment, the knowledge extraction framework is the LangExtract framework, and step S4 is refined into the following sub-steps: S40. Define a prompt word template, which includes the output format requirements for the result fields extracted based on the LangExtract framework.

[0059] Specifically, the LangExtract framework can standardize the processing of long text knowledge extraction, accurately extract entities and relationships, and support custom rules to adapt to multiple scenarios. Its built-in preprocessing module simplifies the process, while also being compatible with knowledge platforms and RDF generation, providing high-quality data for knowledge graph construction. Since a large language model is deployed locally, and the native ontology constraint functionality of the LangExtract framework is unavailable in this mode, a prompt word template based on knowledge ontology needs to be designed to dynamically generate instructions.

[0060] The prompt word template explicitly defines the format that the `extract_text` field of the LangExtract framework should follow. For example, it requires LLM to output line by line in a fixed format: "Entity Type: Entity Name -> Attribute Type: Attribute Value" or "Header Entity -> Relationship Type -> Tail Entity". This strict format constraint is crucial for subsequent automated parsing.

[0061] S41. Fill the entity information in the knowledge ontology data into the prompt word template to generate a structured task description.

[0062] Specifically, in this embodiment, the prompt word template is a predefined text structure that includes variable placeholders and fills the template with the acquired knowledge ontology information, including the definitions of class information, attribute information, and relationship information, to form a detailed and structured task description.

[0063] S42. Based on the output format requirements, task description, ontology constraints of the LangExtract framework, and preprocessed text, generate prompt words.

[0064] Specifically, the generated complete prompt will include: task description, ontology constraints, output format requirements, and the text content to be processed. An example of a prompt is shown below: prompt = f""" Extract the following information from the text: - Entity: {schema_info["entityStr"]}.

[0065] - Attribute: {schema_info["attrStr"]}.

[0066] - Relationship: {schema_info["relationshipStr"]}.

[0067] Please use the exact text snippets from the original document.

[0068] Please do not draw repeatedly. Please output the results in the following format. #Output Format - Entity: [Entity Information]@Entity Type: Entity Name, such as [Entity Information]@Person: Zhang San; - Attribute: [Attribute Information]@Entity Name||Attribute Name: Attribute Value, such as [Attribute Information]@Zhang San||Name: Zhang - Relationship: [Relationship Information]@Subject-->Relationship<--Object, such as [Relationship Information]@Zhang San-->Friend<--Li Si; """ The `schema_info` data is encapsulated based on JSON-formatted knowledge ontology data, where `entityStr` is the entity string, `attrStr` is the attribute string, and `relationshipStr` is the relationship string. Example data is as follows: { "entityStr": "TV variety shows; movies; songs; countries; awards; people; literary works; entertainment figures;", "attrStr": "Entity type: TV variety show attributes include: id, value type string; name, value type string; Entity type: movie attributes include: id, value type string; release date, value type string; name, value type string; Entity type: song attributes include: id, value type string; name, value type string; Entity type: country attributes include: name, value type string; Entity type: award attributes include: name, value type string; number of sessions, value type string; winning work, value type double; release date, value type string; Entity type: person attributes include: name, value type string; film work, value type string; (test) age, value type integer; birthday, value type datetime; gender, value type string; Entity type: literary work attributes include: name, value type string; id, value type string; Entity type: entertainment person attributes include: id, value type string; name, value type string;", "relationshipStr": "Subject: The relationship between characters is: teacher and student; Subject: The relationship between characters is: friends; Subject: The relationship between characters is: superior and subordinate; Subject: The relationship between characters is: husband and wife; Subject: Film and Object: The relationship between characters is: acting; Subject: The relationship between characters is: wife; Subject: The relationship between characters in literary works is: author; Subject: The relationship between characters in entertainment is: voice acting; Subject: The relationship between characters in entertainment and objects is: award; Subject: The relationship between characters in songs is: lyricist; Subject: The relationship between characters in television variety shows is: guest; Subject: The relationship between characters in countries is: nationality; } By strictly defining the prompt word template, the subsequent parsing and RDF conversion processes can be completed automatically and accurately, without the need for complex post-processing.

[0069] In addition, refer to Figure 4 Furthermore, in one embodiment, step S6 is refined into the following sub-steps: S60. Based on the LangExtract framework, the large language model is called to perform discourse-level reasoning on the preprocessed text according to the instructions of the prompt words and example data, and the extracted result fields that meet the output format requirements of the prompt words are output.

[0070] Specifically, after receiving prompts, example data, and preprocessed text, the system uses the LangExtract framework to call a local large language model for inference. LLM performs discourse-level understanding of long texts based on strict instructions in the examples and prompts, outputting the extracted_text field in a predefined format. This effectively solves the problem of text fragmentation, achieving discourse-level knowledge understanding and extraction. It can then accurately extract entities, relationships, and attributes scattered across different parts of the text, ensuring the completeness and coherence of the extracted knowledge. By strictly adhering to the output format requirements in the prompts, the system simplifies subsequent automated parsing, improving the reliability of automatically extracting structured information from free text and alleviating the trade-off between efficiency and accuracy.

[0071] In addition, refer to Figure 5 Furthermore, in one embodiment, step S7 is refined into the following sub-steps: S70. Based on the output format requirements, the extracted result fields are parsed line by line to identify the text entity information in each line.

[0072] Specifically, after receiving the extracted result fields, the text is parsed line by line according to preset formatting rules. Using string processing and regular expression techniques, entity information such as entity type, entity name, attribute type, attribute value, and relationship type in each line is identified.

[0073] S71. Encapsulate text entity information into a structured JSON object.

[0074] Specifically, the parsed information is encapsulated into a structured JSON object, which can clearly organize the extracted knowledge units.

[0075] S72. Traverse the JSON objects and convert the JSON entries in each JSON object into RDF triples based on the class and attribute information URIs defined in the knowledge ontology data.

[0076] Specifically, by iterating through the generated JSON objects and converting each JSON entry into a standard RDF triple based on the class and attribute URIs defined in the knowledge ontology file, the process works as follows: For example, converting "Person: Zhang San -> Age: 30" into a triple: <http: / / example.org / entity / Zhang San><http: / / example.org / ontology / type><http: / / example.org / class / Person>. And <http: / / example.org / entity / Zhang San><http: / / example.org / ontology / age>"30"^^<http: / / www.w3.org / 2001 / XMLSchema#integer>.

[0077] In addition, refer to Figure 6 , further, in one embodiment, step S8 is refined into the following sub-steps: S80: Extract the entity set from the RDF triples, filter the entities of the same type based on the class information defined in the knowledge ontology data, and combine the entities of the same type in pairs into the entity pairs to be fused.

[0078] Specifically, for knowledge fusion based on entity attributes, it is necessary to input the entity pairs to be fused and their attribute sets, and obtain the relevance between the two through attribute information.

[0079] S81: Input each entity pair to be fused and the corresponding attribute set, and calculate the attribute similarity of each entity pair to be fused.

[0080] Specifically, for each pair of attributes, calculate the similarity of their attribute values and handle the entities of the same type specifically. The calculation methods include: Semantic similarity: For text attribute values, use word vector models (such as Word2Vec, Sentence - BERT) to calculate the vector cosine similarity.

[0081] Numeric similarity: For numeric attribute values, calculate the normalized difference or ratio.

[0082] S82: Based on the preset weight values, perform weighted summation on the attribute similarities, and calculate the similarity score of the entities in each entity pair to be fused.

[0083] Specifically, calculate the attribute similarity to quantify the entity similarity degree, and perform weighted summation in combination with the importance of the knowledge ontology attributes. The weight values are predefined in the knowledge ontology design stage, which reflects the importance of different attributes in differentiating entities. Finally, an entity similarity score between 0 and 1 is obtained. Through entity similarity calculation, it can effectively solve the ambiguity problems of entities with the same name but different meanings and entities with the same meaning but different names, and improve the quality and consistency of the knowledge graph data.

[0084] Furthermore, in this embodiment, a unit for decision-making and alerting is provided, which includes a configurable threshold. If the entity similarity exceeds this threshold, the system generates an alert message along with relevant evidence, prompting the user to perform manual or semi-automatic knowledge fusion operations (such as merging entities).

[0085] In addition, refer to Figure 7 Furthermore, in one embodiment, step S8 is refined into the following sub-steps: S83. Extract entity sets from RDF triples, filter entities of the same type based on class information defined by knowledge ontology data, and combine entities of the same type into pairs to be merged.

[0086] Specifically, relationship-based entity fusion requires inputting the entity pairs to be fused and their relationship network, and obtaining the correlation between the two through the relationship information.

[0087] S84. Input the entity pair to be merged and its corresponding relationship network, analyze and obtain the relationship set of each entity in the entity pair to be merged, and determine whether each entity in the entity pair to be merged has a common relationship with the same third entity.

[0088] Specifically, by analyzing the relationship sets of two entities, we can determine whether they have the same relationship with the same third entity, in order to measure the similarity of the entities.

[0089] S85. Count the number of common relationships, where the number of common relationships is positively correlated with the similarity score of the entities.

[0090] Specifically, the number of common relations is counted. The more common relations there are, the more similar the roles and contexts of the two entities in the knowledge network are, and the higher the calculated entity similarity is. This enables entity alignment and disambiguation of RDF triples, avoiding the problems of entities with the same name but different meanings, or entities with the same meaning but different names, from affecting knowledge graph construction and improving data quality and consistency.

[0091] Furthermore, the two similarity calculation implementations, knowledge fusion based on entity attributes and entity fusion based on relationships, can be used independently or in combination, and the final comprehensive similarity can be obtained through weighting or voting mechanisms.

[0092] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0093] This application also provides a chapter-level knowledge extraction system for long texts, which corresponds one-to-one with the chapter-level knowledge extraction methods for long texts in the embodiments.

[0094] A text-level knowledge extraction system for long texts includes: a data preprocessing module, a language identification and translation module, a knowledge ontology acquisition and parsing module, a prompt word generation module, a sample data generation module, a knowledge extraction engine, an RDF triple generation module, and a knowledge fusion module. Detailed descriptions of each functional module are as follows: Data preprocessing module: Used to input diverse and heterogeneous data, and output text data based on the file format of the data and the corresponding parsing rules.

[0095] Language identification and translation module: Used to identify the language of text data and translate it into preprocessed text that can be interpreted by large language models.

[0096] Knowledge ontology acquisition and parsing module: Used to connect to the preset knowledge platform, acquire and parse the knowledge ontology data containing entity information of the identifier based on the identifier of the specified project, and construct a structured representation of the knowledge ontology data.

[0097] Prompt word generation module: Used to generate prompt words for a large language model based on a knowledge extraction framework format, using structured knowledge ontology data and preset prompt word templates.

[0098] Example data generation module: Used to design example data formats based on the knowledge extraction framework, converting the structured labeled sample data of the knowledge platform into example data used by the knowledge extraction framework.

[0099] Knowledge extraction engine: It receives prompt words, sample data and preprocessed text, and generates extraction result fields based on the knowledge extraction framework and large language model.

[0100] RDF triple generation module: used to parse the extracted result fields and convert them into structured RDF triples.

[0101] Knowledge Fusion Module: Used for entity alignment and disambiguation of RDF triples.

[0102] The system comprises several key components: a data preprocessing module that automatically handles diverse and heterogeneous data, converting data from different file formats into text data, significantly expanding the system's applicability; a language identification and translation module that identifies the language of text data and translates it into preprocessed text suitable for interpretation by large language models, ensuring subsequent processing is based on a unified linguistic context; a knowledge ontology acquisition and parsing module that connects to a pre-defined knowledge platform to acquire and parse knowledge ontology data containing entity information, constructing its structured representation to provide a basis for subsequent prompt word generation and RDF generation; and a prompt word generation module that generates prompt words for large language models based on structured knowledge ontology data and pre-defined prompt word templates, enabling the model to adhere to pre-defined domain specifications and logical constraints when understanding text. (Example) The data generation module can design example data formats based on the knowledge extraction framework, converting the structured labeled sample data of the knowledge platform into example data used by the knowledge extraction framework, reducing the dependence on large-scale labeled data and achieving rapid domain adaptation; the knowledge extraction engine can receive prompt words, example data and preprocessed text, and generate extraction result fields based on the knowledge extraction framework and large language model, realizing chapter-level understanding and extraction of long texts; the RDF triple generation module can parse the extraction result fields and convert them into structured RDF triples, ensuring that chapter-level knowledge extraction results can be accurately injected into the knowledge graph; the knowledge fusion module can perform entity alignment and disambiguation on RDF triples, effectively solving entity ambiguity problems and improving the internal consistency and quality of the final constructed knowledge graph.

[0103] Specific limitations regarding the document-level knowledge extraction system for long texts can be found in the context of limitations on document-level knowledge extraction methods for long texts, and will not be repeated here. Each module in the aforementioned document-level knowledge extraction system for long texts can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in an electronic device, or stored in the memory of the electronic device in software form, so that the processor can call and execute the corresponding operations of each module. In one embodiment, an electronic device is provided, which is a user terminal. (Reference) Figure 8 The electronic device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores detection data tables. The network interface allows communication with external terminals via a network connection. When executed by the processor, the computer program implements a text-level knowledge extraction method for long texts.

[0104] In one embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: S1. Input multi-variable heterogeneous data, and output text data based on the file format of the data and the corresponding parsing rules.

[0105] S2. Perform language identification on the text data and translate it into preprocessed text that can be interpreted by a large language model.

[0106] S3. Connect to the preset knowledge platform, obtain and parse the knowledge ontology data containing entity information of the identifier based on the identifier of the specified project, and construct a structured representation of the knowledge ontology data.

[0107] S4. Based on structured knowledge ontology data and preset prompt word templates, generate prompt words for a large language model based on a knowledge extraction framework format.

[0108] S5. Based on the knowledge extraction framework, design an example data format to convert the structured labeled sample data of the knowledge platform into example data used by the knowledge extraction framework.

[0109] S6. Receive prompt words, sample data, and preprocessed text, and generate extraction result fields based on the knowledge extraction framework and large language model.

[0110] S7. Parse the extracted result fields and convert them into structured RDF triples.

[0111] S8. Perform entity alignment and disambiguation on RDF triples.

[0112] In one embodiment, the sub-steps of step S3 refinement include: S30. Obtain the knowledge ontology data of the identifier. The knowledge ontology data is returned in JSON format and defines entity information, including class information of the category to which the entity belongs, attribute information of the characteristics or properties of the entity, and relationship information of semantic relationships between entities.

[0113] S31. Parse the knowledge ontology data and construct a structured representation of the knowledge ontology data in memory.

[0114] In one embodiment, the sub-steps of step S4 refinement include: S40. Define a prompt word template, which includes the output format requirements for the result fields extracted based on the LangExtract framework.

[0115] S41. Fill the entity information in the knowledge ontology data into the prompt word template to generate a structured task description.

[0116] S42. Based on the output format requirements, task description, ontology constraints of the LangExtract framework, and preprocessed text, generate prompt words.

[0117] In one embodiment, the sub-steps of step S6 are further refined as follows: S60. Based on the LangExtract framework, the large language model is called to perform discourse-level reasoning on the preprocessed text according to the instructions of the prompt words and example data, and the extracted result fields that meet the output format requirements of the prompt words are output.

[0118] In one embodiment, the sub-steps of step S7 are further refined as follows: S70. Based on the output format requirements, the extracted result fields are parsed line by line to identify the text entity information in each line.

[0119] S71. Encapsulate text entity information into a structured JSON object.

[0120] S72. Traverse the JSON objects and convert the JSON entries in each JSON object into RDF triples based on the class and attribute information URIs defined in the knowledge ontology data.

[0121] In one embodiment, the sub-steps of step S8 refinement include: S80. Extract entity sets from RDF triples, filter entities of the same type based on class information defined by knowledge ontology data, and combine entities of the same type into pairs to be merged.

[0122] S81. Input each entity pair to be merged and its corresponding attribute set, and calculate the attribute similarity of each entity pair to be merged.

[0123] S82. Based on the preset weight values, the attribute similarity is weighted and summed to calculate the similarity score of each entity pair to be merged.

[0124] In one embodiment, the sub-steps of step S8 refinement include: S83. Extract entity sets from RDF triples, filter entities of the same type based on class information defined by knowledge ontology data, and combine entities of the same type into pairs to be merged.

[0125] S84. Input the entity pair to be merged and its corresponding relationship network, analyze and obtain the relationship set of each entity in the entity pair to be merged, and determine whether each entity in the entity pair to be merged has a common relationship with the same third entity.

[0126] S85. Count the number of common relationships, where the number of common relationships is positively correlated with the similarity score of the entities.

[0127] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0128] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

Claims

1. A method for extracting chapter-level knowledge from long texts, characterized in that, include: Input diverse and heterogeneous data, and output text data based on the file format of the data and the corresponding parsing rules; The text data is then language-identified and translated into preprocessed text that can be interpreted by a large language model. By connecting to a pre-defined knowledge platform, the knowledge ontology data containing entity information of the identifier of the specified project is obtained and parsed, and a structured representation of the knowledge ontology data is constructed. Based on the structured knowledge ontology data and the preset prompt word template, a large language model prompt word based on the knowledge extraction framework format is generated; Based on the knowledge extraction framework, an example data format is designed to convert the structured labeled sample data of the knowledge platform into example data used by the knowledge extraction framework. Receive the prompt words, example data and preprocessed text, and generate extraction result fields based on the knowledge extraction framework and large language model; Parse the extracted result fields and convert them into structured RDF triples; Entity alignment and disambiguation are performed on the RDF triples.

2. The method according to claim 1, characterized in that, The steps of connecting to the preset knowledge platform, obtaining and parsing knowledge ontology data containing entity information of the identifier based on the identifier of the specified project, and constructing a structured representation of the knowledge ontology data include: Obtain the knowledge ontology data of the identifier. The knowledge ontology data is returned in JSON format and defines entity information, including class information of the category to which the entity belongs, attribute information of the characteristics or properties of the entity, and semantic relationship information between entities. The knowledge ontology data is parsed, and a structured representation of the knowledge ontology data is constructed in memory.

3. The method according to claim 2, characterized in that, The knowledge extraction framework is the LangExtract framework. The step of generating large language model prompts based on the structured knowledge ontology data and preset prompt templates includes: Define the prompt word template, which includes the output format requirements for the result fields extracted based on the LangExtract framework; The entity information from the knowledge ontology data is filled into the prompt word template to generate a structured task description; Based on the output format requirements, task description, ontology constraints of the LangExtract framework, and preprocessed text, the prompt words are generated.

4. The method according to claim 3, characterized in that, The step of receiving the prompt words, example data, and preprocessed text, and generating extraction result fields based on the knowledge extraction framework and large language model, includes: Based on the LangExtract framework, the large language model is invoked to perform discourse-level reasoning on the preprocessed text according to the instructions of the prompt words and example data, and the extracted result fields that meet the output format requirements of the prompt words are output.

5. The method according to claim 4, characterized in that, The step of parsing the extracted result fields and converting them into structured RDF triples includes: Based on the output format requirements, the extracted result fields are parsed line by line to identify the text entity information in each line; The text entity information is encapsulated into a structured JSON object; Traverse the JSON objects and convert the JSON entries in each JSON object into RDF triples based on the class information and attribute information URIs defined in the knowledge ontology data.

6. The method according to claim 5, characterized in that, The steps of performing entity alignment and disambiguation on the RDF triples include: Extract entity sets from the RDF triples, filter entities of the same type based on the class information defined by the knowledge ontology data, and combine the entities of the same type into pairs of entities to be merged. Input each pair of entities to be merged and its corresponding set of attributes, and calculate the attribute similarity of each pair of entities to be merged. Based on preset weight values, the attribute similarity is weighted and summed to calculate the similarity score of each entity pair to be merged.

7. The method according to claim 5, characterized in that, The steps of performing entity alignment and disambiguation on the RDF triples include: Extract entity sets from the RDF triples, filter entities of the same type based on the class information defined by the knowledge ontology data, and combine the entities of the same type into pairs of entities to be merged. Input the entity pair to be merged and its corresponding relationship network, analyze and obtain the relationship set of each entity in the entity pair to be merged, and determine whether each entity in the entity pair to be merged has a common relationship with the same third entity; The number of common relationships is counted, where the number of common relationships is positively correlated with the entity similarity score.

8. A document-level knowledge extraction system for long texts, characterized in that, include: Data preprocessing module: used to input diverse and heterogeneous data, and output text data based on the file format of the data and the corresponding parsing rules; Language identification and translation module: used to identify the language of the text data and translate it into preprocessed text that can be interpreted by a large language model; Knowledge ontology acquisition and parsing module: used to connect to the preset knowledge platform, acquire and parse the knowledge ontology data containing entity information of the identifier of the specified project, and construct the structured representation of the knowledge ontology data; Prompt word generation module: used to generate large language model prompt words based on the structured knowledge ontology data and preset prompt word templates; Example data generation module: used to design example data format based on the knowledge extraction framework, and convert the structured labeled sample data of the knowledge platform into example data used by the knowledge extraction framework; Knowledge extraction engine: used to receive the prompt words, example data and preprocessed text, and generate extraction result fields based on the knowledge extraction framework and large language model; RDF triple generation module: used to parse the extracted result fields and convert them into structured RDF triples; Knowledge fusion module: used to perform entity alignment and disambiguation on the RDF triples.

9. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed as any one of the document-level knowledge extraction methods for long texts as claimed in claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer program is stored that can be loaded by a processor and executed as any one of the document-level knowledge extraction methods for long texts as claimed in claims 1 to 7.

Images