A Method and System for Unstructured Text Parsing and Question Answering Based on a Large Language Model

Abstract

This application provides a method and system for unstructured text parsing and question answering based on a large language model. The method includes: acquiring raw unstructured text data from a data source at preset intervals; performing deep semantic analysis on the raw unstructured text data using a large language model to extract key entities and data structures; dynamically generating or updating an existing JSON structure template based on the key entities and data structures; converting the raw unstructured text data into standard structured data according to the JSON structure template; storing the structured data in a time-series database and establishing a multi-dimensional index; obtaining time information and query intent from natural language query information input by the user; and generating question-answering results or trend analysis results using a large language model based on the query intent and the structured data obtained from the time-series database according to the time information.

Description

Technical Field

[0001] This application relates to the fields of natural language processing and intelligent data analysis, and in particular to a method and system for unstructured text parsing and question answering based on a large language model. Background Technology

[0002] In various management scenarios, large amounts of periodic unstructured text data are often generated, such as daily construction reports, experimental records, work summaries, and equipment maintenance logs. This data usually exists in free text form, lacking a unified structural representation, which brings many challenges to data management, retrieval, statistical analysis, and trend prediction.

[0003] In existing technologies, periodically generated unstructured text data lacks a unified structural representation, making efficient retrieval and trend analysis difficult. With the rapid development of large language model technology, it has demonstrated powerful capabilities in text understanding, information extraction, and natural language generation. However, how to effectively apply large language models to the parsing and intelligent question answering of periodically unstructured text, achieving automated conversion from unstructured text to structured data, and supporting intelligent queries and trend analysis based on natural language remains a pressing technical problem to be solved. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the purpose of this application is to provide a method and system for unstructured text parsing and question answering based on a large language model, thereby enabling automatic parsing of unstructured text, time-series data management, and intelligent question answering.

[0005] To achieve the above objectives, this application provides a method for unstructured text parsing and question answering based on a large language model, including:

[0006] Retrieve raw unstructured text data from the data source according to a preset cycle;

[0007] The original unstructured text data is subjected to deep semantic analysis using a large language model, thereby extracting key entities and data structures;

[0008] Based on the key entities and the data structure, dynamically generate or update the existing JSON structure template;

[0009] According to the JSON structure template, the original unstructured text data is converted into standard structured data;

[0010] The structured data is stored in a time-series database, and a multi-dimensional index is created.

[0011] Obtain time information and query intent from the natural language query information entered by the user;

[0012] Based on the query intent and the structured data obtained from the time series database according to the time information, question-answering results or trend analysis results are generated through a large language model.

[0013] Furthermore, the step of performing deep semantic analysis on the original unstructured text data using a large language model to extract key entities and data structures includes:

[0014] The original unstructured text data is preprocessed using a large language model to perform deep semantic analysis, resulting in text semantic analysis.

[0015] Key entities are extracted from the semantic analysis results of the text using named entity recognition.

[0016] Dependency syntax is used to analyze the semantic relationships and logical structure between key entities;

[0017] Based on the semantic relationships and logical structure, the field types and value ranges of the key entities are inferred;

[0018] The key entities include time, location, personnel, equipment, quantity, and event.

[0019] Furthermore, the step of dynamically generating or updating the existing JSON structure template based on the key entity and the data structure further includes: building and maintaining a structure template library based on preset basic structure templates for various common text types; automatically updating the structure template according to the field type of the newly identified key entity; and performing version control and backward compatibility processing on the updated structure template.

[0020] Furthermore, the step of storing the structured data in a time-series database and establishing a multi-dimensional index includes: converting the time information in the original unstructured text data into a standard timestamp format, establishing a hierarchical directory structure by year, month, and day, and establishing a time dimension index, a key field index, and a full-text search index.

[0021] Furthermore, the step of obtaining time information and query intent from the natural language query information input by the user includes: identifying absolute time, relative time, and time range expressions in the query information; classifying user questions according to query intent, including single-point queries, range queries, trend analysis, and statistical summaries; and converting various expressions of time into standard query parameters to achieve parameter normalization.

[0022] Furthermore, before the step of generating question-answering results or trend analysis results through a large language model based on the query intent and the structured data obtained from the time series database according to the time information, the method further includes: determining the data retrieval range according to the time information and obtaining structured data from the time series database;

[0023] The step of determining the data retrieval range based on time information and obtaining structured data from the time series database includes: for a single point-in-time query, locating and reading the JSON file corresponding to the date;

[0024] For multi-time point or trend queries, relevant data are extracted by time series and assembled into a time series array;

[0025] A field filtering mechanism is adopted to extract only the data of relevant fields based on the query intent in order to improve processing efficiency.

[0026] Furthermore, the step of generating question-and-answer results or trend analysis results based on the query intent and the structured data obtained from the time-series database according to the time information through a large language model includes: combining the obtained structured data with the query information into complete context information; performing logical reasoning and correlation analysis on the complete context information through a large language model, and generating answer text, data tables, or visualization charts according to the type of query intent.

[0027] Furthermore, the large language model employs a pre-trained neural network model based on an attention mechanism.

[0028] To achieve the above objectives, this application also provides an unstructured text parsing and question-answering system based on a large language model, employing the unstructured text parsing and question-answering method based on a large language model as described above, including:

[0029] The text data acquisition module is used to read raw unstructured text data from unstructured text files;

[0030] The semantic parsing module uses a large language model to perform semantic understanding and structure recognition on the raw unstructured text data, and obtains key entities and data structures;

[0031] The structure template management module is used to dynamically generate and maintain JSON structure templates based on the key entities and the data structure;

[0032] The data conversion module is used to convert the original unstructured text data into structured data according to the JSON structure template;

[0033] The time-series database module is used to store and manage structured data along the time dimension;

[0034] The question-and-answer processing module is used to process the natural language query information input by the user and obtain time information and query intent;

[0035] The data retrieval module is used to retrieve and aggregate the corresponding structured data from the time-series database module based on the time information and the query intent, and then pass it to the intelligent question answering module.

[0036] The intelligent question-answering module utilizes a large language model to generate and output question-answering results and trend analysis based on the query intent and the corresponding structured data.

[0037] Furthermore, the semantic parsing module includes:

[0038] The entity recognition submodule is used to extract key entities from text data;

[0039] The relation extraction submodule is used to identify the semantic relationships between the key entities;

[0040] The structure inference submodule is used to infer the logical structure of the text data and the field types of the key entities.

[0041] Furthermore, the time-series database module includes: a storage engine that adopts a distributed storage architecture to support large-scale data storage; an index manager for multi-dimensional indexing, including time-dimensional indexes, key field indexes, and full-text search indexes; and a caching module for memory caching of frequently accessed data to improve query performance.

[0042] Furthermore, the question-answering processing module includes: a natural language understanding submodule for analyzing the semantics and intent of user questions; a time parsing submodule for extracting standardized time information from the input natural language query information; and a query planning submodule for formulating the optimal data retrieval strategy.

[0043] Furthermore, it also includes a visualization output module for generating and outputting formatted results from the question-and-answer results and trend analysis; the formatted results include natural language text, visualization charts, and structured tables;

[0044] The visualization output module supports multiple chart types, including line charts, bar charts, scatter plots, heatmaps, tables, and lists.

[0045] The unstructured text parsing and question answering method based on a large language model provided in this application performs unstructured text processing, time-series data management and intelligent question answering based on a large language model, and realizes automatic parsing of unstructured text and generation of questions and answers.

[0046] Other features and advantages of this application will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing this application. Attached Figure Description

[0047] The accompanying drawings are provided to further illustrate the present application and form part of the specification. Together with the embodiments of the present application, they serve to explain the present application but do not constitute a limitation thereof. In the drawings:

[0048] Figure 1 The flowchart below shows the unstructured text parsing and question answering method based on a large language model according to Embodiment 1 of this application.

[0049] Figure 2 Here is a flowchart of the JSON Schema adaptive generation algorithm according to Embodiment 1 of this application;

[0050] Figure 3 This is a flowchart illustrating the text parsing and structured data conversion process according to Embodiment 1 of this application;

[0051] Figure 4 This is a schematic diagram of the time-series database storage method and indexing mechanism according to Embodiment 1 of this application;

[0052] Figure 5 This is a flowchart of the intelligent question-answering process according to Embodiment 1 of this application;

[0053] Figure 6 This is a flowchart of the trend analysis process according to Embodiment 1 of this application;

[0054] Figure 7 The flowchart below shows the unstructured text parsing and question answering method based on a large language model, according to Example 2.

[0055] Figure 8 This is a schematic diagram of the intelligent schema learning and multimodal output process according to Embodiment 3 of this application;

[0056] Figure 9 This is a schematic diagram illustrating the maintenance of different domain schema libraries according to Embodiment 4 of this application;

[0057] Figure 10 This is an unstructured text parsing and question answering system based on a large language model, according to Embodiment 5 of this application. Detailed Implementation

[0058] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application.

[0059] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0060] The term "comprising" and its variations as used herein are open-ended inclusion, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; and the term "some embodiments" means "at least some embodiments".

[0061] It should be noted that the terms "first" and "second" may be used in this application only to distinguish different devices, components or parts, and are not used to define the order of functions performed by these devices, components or parts or their interdependence.

[0062] It should be noted that the terms "one" and "more" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless explicitly stated otherwise in the context, they should be understood as "one or more". "More" should be understood as two or more.

[0063] The following terms may be used in the embodiments of this application:

[0064] Large Language Model: A large-scale pre-trained neural network model based on the Transformer architecture, possessing powerful natural language understanding and generation capabilities;

[0065] Transformer: A deep learning model that incorporates an attention mechanism;

[0066] JSON Schema: A metadata schema used to describe the structure of JSON data, defining information such as data fields, types, and constraints;

[0067] Time series database: A database system used to store and manage time series data, supporting efficient querying and analysis along the time dimension;

[0068] Semantic understanding: Deeply analyzes the meaning, structure, and logical relationships of text using natural language processing techniques;

[0069] Entity recognition: Identifying and extracting specific types of entity information from text, such as names of people, places, times, quantities, etc.

[0070] Relation extraction: Identifying semantic relationships and logical connections between entities in text;

[0071] Schema Adaptive: The ability to automatically adjust and optimize the data structure schema based on data characteristics;

[0072] Few-shot learning: A machine learning method that uses a small number of examples to train a model to perform a specific task;

[0073] Inverted index: An index structure that maps words to documents based on document content, supporting full-text search;

[0074] B+ tree index: A balanced multi-way search tree data structure that supports efficient range queries and sorting;

[0075] Context window: The text length limit that a large language model can process simultaneously, in units of tokens;

[0076] Token: The smallest semantic unit in text processing, which can be a character, word, or subword;

[0077] Backward compatibility: The ability of a newer system version to correctly handle older data and formats;

[0078] Distributed storage: A storage architecture that distributes data across multiple physical nodes;

[0079] Consistent hashing: A hashing algorithm used for data sharding and load balancing in distributed systems;

[0080] Knowledge distillation: a model compression technique that transfers knowledge from a large model to a smaller model.

[0081] Definitions for other terms will be provided in the following description.

[0082] Example 1

[0083] Figure 1 The flowchart of the unstructured text parsing and question answering method based on a large language model according to Embodiment 1 of this application will be referenced below. Figure 1 This application provides a detailed description of the unstructured text parsing and question answering method based on a large language model.

[0084] First, in step S101: acquire raw unstructured text data according to a preset period.

[0085] In the embodiments of this application, raw unstructured text data is read from a specified data source according to a preset period (such as daily or weekly), and multiple file formats including TXT, DOC, PDF, etc. are supported.

[0086] In the embodiments of this application, a file monitoring mechanism is established to detect in real time whether a new text file has been added to the data source; if so, the data of the newly added text file is read and preprocessed, such as character encoding conversion, format standardization and noise filtering.

[0087] In step S102: Deep semantic analysis is performed on the original unstructured text data using a large language model to extract key entities and data structures.

[0088] In the embodiments of this application, a large language model is used to perform deep semantic analysis on the original unstructured text data. Few-shot learning technology is used to provide a small number of examples to guide the large language model to understand the text data structure of a specific domain. Based on the text semantic analysis results obtained by the large language model, named entity recognition (NER) technology is used to extract key entities, such as time, location, people, equipment, quantity, and event entities. Dependency syntax is used to analyze the semantic relationships and logical structures between key entities. Based on the semantic relationships and logical structures, the field types, value ranges, and constraints of key entities are automatically inferred.

[0089] In the embodiments of this application, the large language model adopts a pre-trained neural network model based on the Transformer (a deep learning model that introduces an attention mechanism) architecture, with a model parameter scale of no less than 7 billion and supporting a context length of no less than 8192 tokens (a semantic unit).

[0090] In step S103: Dynamically generate or update the existing JSON structure template based on key entities and data structures.

[0091] In the embodiments of this application, this step includes: building and maintaining a schema template library by pre-setting basic schema (structure) templates for common text types; dynamically generating or updating JSON schema templates based on semantic analysis results, including field definitions, data types, required fields, and validation rules; supporting schema version management, performing version control on updated structure templates to ensure backward compatibility; and employing an incremental learning mechanism to continuously optimize the schema based on data from newly added text files. Dynamic field expansion is supported, i.e., automatically expanding the schema template based on the field types of newly identified key entities. Figure 2 As shown, Figure 2 The flowchart of the JSON Schema adaptive generation algorithm according to Embodiment 1 of this application illustrates the detailed process of dynamic schema generation.

[0092] In the embodiments of this application, the JSON Schema includes the following field types: string, numeric, boolean, date, array, and object, and supports nested structures and optional field definitions.

[0093] In step S104: The original unstructured text data is converted into standard structured data according to the JSON structure template.

[0094] In the embodiments of this application, the step includes: converting the original unstructured text data into structured data in standardized JSON format based on the generated JSON structure template; interacting with the large language model using a multi-turn dialogue mechanism to ensure the accuracy and completeness of the conversion results; performing data quality checks on the conversion results, including format verification, value range checks, and logical consistency verification; and marking and manually intervening in unparseable text data through an established exception handling mechanism.

[0095] Figure 3 The flowchart of text parsing and structured data conversion according to Embodiment 1 of this application illustrates in detail the processing of unstructured text data to JSON structured data in steps S103 and S104 above.

[0096] In step S105: The structured data is stored in the time series database according to the time dimension, and a multidimensional index is established.

[0097] Figure 4 The diagram below illustrates the time-series database storage method and indexing mechanism according to Embodiment 1 of this application. (Refer to...) Figure 4 In the embodiments of this application, this step includes: converting the time information in the original unstructured text data into a standard timestamp format; establishing a hierarchical directory structure by year, month, and day; and establishing a time dimension index, a key field index, and a full-text search index. For example, the time information is standardized to ISO 8601 format timestamps; a hierarchical storage structure is adopted, with a directory hierarchy established by year / month / day for easy and rapid location; a multi-dimensional indexing mechanism is established, including a time dimension index, a key field index, and a full-text search index; and historical data is compressed and stored to save space, achieving data compression and archiving.

[0098] In step S106: Receive and process the natural language query information input by the user, and extract time information and query intent from it.

[0099] In the embodiments of this application, this step includes: receiving natural language query information input by the user; using a large language model to understand the query information and extract the query intent and key information; using time entity recognition technology to identify time expressions in the query information such as absolute time (e.g., "March 15, 2024"), relative time (e.g., "yesterday", "last week"), and time range (e.g., "last month"); classifying the query intent into types such as single-point query, range query, trend analysis, and statistical summary; and standardizing the time expressions to convert them into standardized query parameters for the database, thereby achieving parameter normalization.

[0100] In step S107: Based on the query range determined by the time information, obtain the corresponding structured data from the time series database.

[0101] In the embodiments of this application, the data retrieval range is determined based on time information; for single-point-of-time queries, the JSON file corresponding to the date can be directly located and read; for multi-point-of-time or trend queries, multiple point-of-time data are extracted according to the time series and formed into a time series array; a field filtering mechanism is adopted to extract only the data of relevant fields according to the query intent, thereby improving processing efficiency; common query patterns are pre-calculated and cached to support data pre-aggregation.

[0102] In step S108: Combining structured data and the user's query intent, a large language model is used to generate question-and-answer results or trend analysis results.

[0103] Figure 5 The flowchart of the intelligent question-answering process according to Embodiment 1 of this application illustrates the complete process of processing user queries and generating question-answering results. Figure 5 As shown in the embodiments of this application, this step includes: combining the retrieved structured data with the user's query information to construct complete contextual information, and then performing logical reasoning and correlation analysis through a large language model to generate an answer that meets the user's needs; selecting the output format according to the type of query intent, including natural language description, data tables, statistical charts, etc.; for trend analysis questions, it also includes the step of calculating relevant statistical indicators such as mean, variance, growth rate, etc., and generating a visual chart.

[0104] In the embodiments of this application, step S108 further includes: analyzing the periodicity, trend and anomaly characteristics in multiple time point data; calculating statistical indicators such as mean, variance, and growth rate; and converting the trend analysis results into visualization forms such as line charts and bar charts.

[0105] Figure 6 The flowchart of trend analysis processing according to Embodiment 1 of this application illustrates the detailed process of time series data analysis and visualization output.

[0106] The unstructured text parsing and question answering method based on a large language model provided in this application has the following advantages:

[0107] (1) Significantly improved parsing accuracy: Through the deep semantic understanding capability of the large language model, the accuracy of structuring unstructured text data is significantly improved, and it can handle complex and diverse text formats and expressions.

[0108] (2) Optimized query efficiency: The hierarchical storage and multi-dimensional indexing mechanism are adopted, and the single query response time is controlled within 500 milliseconds. The response time is shortened by more than 60% compared with the traditional full-text search method, and it supports real-time query of tens of millions of data.

[0109] (3) Strong adaptability: It adopts the JSON Schema adaptive generation mechanism, which can dynamically adjust the data structure according to the changes in text content, greatly improving adaptability and handling new text formats and field types without manual intervention.

[0110] (4) Improved interactive experience: It supports natural language query, so users do not need to learn complicated query syntax. The query success rate reaches more than 88%, and user satisfaction is improved compared with traditional keyword search.

[0111] (5) Possesses trend analysis capabilities: Based on time series data modeling, it can identify periodic patterns, abnormal changes and development trends, with a trend prediction accuracy of over 85%, providing strong support for decision-making.

[0112] (6) Strong system scalability: The modular architecture design supports flexible configuration and functional expansion, and the adaptation time for adding new text types is shortened from several weeks in traditional solutions to several hours.

[0113] Example 2

[0114] This embodiment uses a construction daily report text as an example to describe in detail another implementation of the unstructured text parsing and question-answering method based on a large language model of this application.

[0115] Figure 7 As shown in the flowchart of the unstructured text parsing and question answering method based on a large language model according to Embodiment 2 of this application, the unstructured text parsing and question answering method based on a large language model of this application includes:

[0116] Step 1), perform initial configuration. This step includes:

[0117] Configure a large language model: adopt a large pre-trained model based on the Transformer architecture with 13 billion parameters and support a context length of 16,384 tokens;

[0118] Set the text monitoring directory to: / data / construction_reports / , and the monitoring formats include .txt, .doc, and .docx;

[0119] Configure the time series database: use distributed storage, with the root directory being / database / timeseries / , and establish a three-level directory structure by year / month / day;

[0120] Initialize the schema template library: pre-set basic schema templates for construction, equipment, personnel and other fields.

[0121] Step 2), obtain unstructured text and preprocess it.

[0122] In this embodiment, the example unstructured text input is: "March 15, 2024, sunny, temperature 8-15℃. Excavation work in Area A continues, with 120 cubic meters of earthwork excavated, 2 excavators and 8 workers deployed. Concrete pouring work in Area B, with a pouring volume of 45 cubic meters, passed quality inspection. A drainage problem was found in Area C, and repairs have been arranged."

[0123] The specific steps of preprocessing include: character encoding detection: recognizing it as UTF-8 encoding; format standardization: unifying line breaks and removing redundant spaces; noise filtering: removing non-text characters and retaining punctuation marks; text segmentation: splitting text into independent sentences by period.

[0124] Step 3) Perform semantic analysis and data structure recognition.

[0125] In this embodiment, the identification result of the key entity is as follows:

[0126] Time Entity: March 15, 2024;

[0127] Weather conditions: Sunny, 8-15℃;

[0128] Project entities: Excavation work in area A, concrete pouring in area B;

[0129] Quantity of items: 120 cubic meters, 2 units, 8 people, 45 cubic meters;

[0130] Status: Progress continues, quality inspection passed, drainage issue;

[0131] The relationships between the key entities extracted are as follows:

[0132] Area A → Excavation Project → 120 cubic meters of earthwork excavation completed;

[0133] Excavation work → Equipment used → 2 excavators;

[0134] Excavation work → Personnel involved → 8 workers;

[0135] Area B → Concrete pouring → Pouring volume 45 cubic meters;

[0136] Area B → Quality Status → Qualified.

[0137] Step 4) Adaptively generate a JSON Schema template. This step generates a JSON Schema based on the semantic analysis results of Step 3).

[0138] Step 5) performs structured data transformation. This step, based on the schema template generated in Step 4), converts the original unstructured text data into structured JSON data.

[0139] Step 6) Store the structured data in a time-series database.

[0140] In this embodiment, the storage file path is: / database / timeseries / 2024 / 03 / 20240315.json; the time index is a B+ tree index created based on the date field; the field index is a hash index built on key entity fields such as area_id and work_type; and the full-text index is an inverted index built on descriptive text fields.

[0141] Step 7), intelligent question-answering processing. For example, consider a user query: "What is the excavation progress in area A on March 15th?"

[0142] First, we understood the problem. The specific process is as follows: the time was extracted from March 15th to 2024-03-15; the extracted area was area A; the query intent was progress query; and the query type was single-point query. Based on this, we performed data retrieval: locating the file " / database / timeseries / 2024 / 03 / 20240315.json"; we filtered the fields and extracted the relevant data in work_areas (work area) where area_id (area number) was "area A", including: excavation work, 120 cubic meters of earthwork excavation completed, 2 excavators deployed, and 8 workers.

[0143] Step 8) Generate intelligent question-answering results. This step combines the data retrieved in Step 7) with the user's question and generates an answer using a large language model, such as: "The excavation project in Area A on March 15th is progressing well. 120 cubic meters of earthwork were excavated that day, with 2 excavators and 8 workers deployed. The project status is 'Continuing Progress'."

[0144] Example 3

[0145] This embodiment adds intelligent schema learning and multimodal output steps to the existing embodiment 1.

[0146] Figure 8 As a schematic diagram of the intelligent schema learning and multimodal output process according to Embodiment 3 of this application, the following will refer to... Figure 8The unstructured text parsing and question answering method based on a large language model in this embodiment is described in detail.

[0147] In this embodiment, an incremental schema learning mechanism is used for intelligent schema learning. This step includes: establishing a schema version control system to generate a new version number each time the schema is updated; performing backward compatibility checks to ensure that the new schema can handle historical data formats; using a weighted voting mechanism to adjust the weight of fields in the schema according to the frequency and importance of text data; and setting the schema stability threshold to 0.95. When the schema consistency of 100 consecutive parsed data exceeds this threshold, the current schema is set as a stable version.

[0148] In this embodiment, multilingual processing is enhanced through the following steps: expanding the large language model to support mixed Chinese and English text processing; establishing a bilingual terminology lookup table to achieve Chinese-English mapping of keywords; employing character-level language detection methods to accurately identify multilingual segments in text data; and standardizing the time expressions in different languages.

[0149] This embodiment also includes a step of optimizing the visualization output of question-and-answer results. For example, if a user queries: "Recent week's excavation progress trend in area A", the processing flow includes:

[0150] Analysis time range: the most recent week → March 9, 2024 to March 15, 2024;

[0151] Aggregated data: Extract mining data from region A within 7 days;

[0152] Calculate trends: Calculate daily completion amount, cumulative progress, and average efficiency;

[0153] Based on the extracted mining data and daily completion volume, cumulative progress, and average efficiency, visualization results are generated, including line charts, bar charts, scatter plots, heatmaps, and structured output formats such as tables and lists.

[0154] This embodiment also includes steps for anomaly detection and early warning. These steps include: calculating the normal range of various indicators based on 30 days of historical data and establishing a baseline model; setting anomaly thresholds, such as triggering an early warning when progress deviation exceeds ±20% or resource input deviation exceeds ±15%; automatically performing anomaly detection based on the baseline model and set anomaly thresholds after each new data entry, achieving real-time monitoring and anomaly detection; and determining whether to generate an early warning report based on the anomaly detection results, with the report including the anomaly type, degree of deviation, possible causes, and recommended measures.

[0155] Compared with Example 1, this embodiment achieves significant improvements in the following aspects:

[0156] Schema Adaptability: The template adaptability has been greatly improved by moving from a single fixed template to adaptive learning.

[0157] Query response speed: Through pre-aggregation and caching mechanisms, the average response time has been reduced from 500ms to 200ms;

[0158] Visual richness: Supports 6 chart types, enhancing visual expression capabilities;

[0159] Anomaly detection accuracy: The accuracy is improved compared to threshold alarm methods by adopting an anomaly detection and early warning mechanism.

[0160] Example 4

[0161] This embodiment illustrates the adaptability of the unstructured text parsing and question answering method based on a large language model to different fields through different application scenarios.

[0162] Application Scenario Example 1: Biological Experiment Records. Input Text Example: "March 15, 2024, Experiment No. BIO-2024-0315, Day 7 of mouse drug administration experiment. Group A (control group) 10 mice, average weight gain 2.3g; Group B (low-dose group) 10 mice, average weight gain 1.8g; Group C (high-dose group) 10 mice, average weight gain 0.9g. A significant decrease in activity and appetite was observed in Group C mice. Blood samples have been collected and are awaiting testing."

[0163] In this example, by identifying the features of the experimental text mentioned above, a new JSON Schema template is generated, the Schema is adaptively adjusted, and thus the data structure transformation is completed.

[0164] Experimental verification shows that the unstructured text parsing and question answering method based on a large language model proposed in this application has high domain adaptability, as demonstrated below: It automatically adapts to switching from the construction domain to the biological experiment domain in 15 minutes; it achieves a parsing accuracy of 90% on biological experiment text; it can automatically learn and recognize experiment-specific terms such as "drug administration," "control group," and "dosage"; and it supports domain-specific queries such as "data from day one of the experiment" and "weight change trends of each group." Example of a cross-domain query: A user queries "What is the weight change trend of group C mice during the experiment?" The query identifies the domain as biological experiment, the object as group C mice, the indicator as weight change, and the time range as the experimental process (automatically inferred to be from the start of the experiment to the present).

[0165] Figure 9 To illustrate the maintenance of different domain schema libraries according to Embodiment 4 of this application, such as... Figure 9As shown, in this embodiment, different domain schema libraries are maintained to adapt to different application scenarios, such as construction, biological experiments, equipment maintenance, and meeting minutes.

[0166] Application Scenario Example 2: For large-scale enterprise applications, a distributed deployment architecture is supported, including: data sharding strategy, load balancing, caching strategy, and fault tolerance mechanism. The data sharding strategy includes horizontal sharding by time dimension, with each shard covering 3 months of data; load balancing includes using a consistent hashing algorithm to distribute query requests; the caching strategy includes a three-level caching architecture (including application cache, distributed cache, and database cache); and the fault tolerance mechanism includes using multiple replicas to store data, with master-slave failover time controlled within 30 seconds.

[0167] Application Scenario Example 3: Lightweight Mobile Version. The streamlined version optimized for mobile devices includes: compressing large language models to 1GB using knowledge distillation technology; supporting offline operation, with only complex queries requiring an internet connection; employing an incremental synchronization strategy between the mobile client and server to reduce data transmission; and power optimization: an intelligent scheduling mechanism that automatically switches to power-saving mode when battery is low.

[0168] Application Scenario Example 4: Multimodal expansion to support multimodal data input such as images and voice. For example, integrating text recognition to automatically extract text information from images; supporting voice input and converting it to text for processing in real time; automatically recognizing and parsing table structures and converting them into structured data; and supporting the digitization of handwritten documents.

[0169] The following experimental data and performance comparison demonstrate the advantages of the unstructured text parsing and question answering method based on a large language model proposed in this application.

[0170] Experimental conditions: The hardware environment consisted of an Intel Xeon Gold 6248R CPU, 64GB RAM, and an NVIDIA A100 GPU; the software environment consisted of Ubuntu 20.04, Python 3.9, and PyTorch 1.12; a total of 50,000 documents from 5 different fields were collected as the dataset, including daily construction reports, experimental records, equipment maintenance, meeting minutes, and project reports.

[0171] The performance test results are shown in Table 1:

[0172] Table 1

[0173]

[0174] It is evident that the method of this application has significant advantages in various aspects.

[0175] In the embodiments of this application, in order to better utilize the unstructured text parsing and question answering method based on a large language model provided in this application, some common fault types and improvement measures are provided below:

[0176] Fault Type 1: Schema mismatch fault, manifested as the inability to correctly parse new text formats. This is caused by text formats exceeding the coverage of the existing schema, requiring enhancement of the schema adaptive learning algorithm to improve generalization ability. This can be addressed by establishing preventative mechanisms, such as a schema early warning system, to identify format change trends in advance.

[0177] Fault Type 2: Inference timeout in large language models, manifested as complex text processing time exceeding a set threshold. This type of fault is caused by text length exceeding the model's processing capacity or insufficient computing resources. Consideration should be given to text segmentation and result merging, or the use of model parallelism and streaming processing methods.

[0178] Fault Type 3: Data consistency issue, manifested as inconsistent parsing results for the same document at different times. This is a backward compatibility problem caused by schema version updates. The solution is to strengthen the version control mechanism to ensure parsing consistency. To further guarantee quality, data consistency checks and repair functions can be added.

[0179] Example 5

[0180] In this embodiment, an unstructured text parsing and question-answering system based on a large language model is also provided. The system uses the unstructured text parsing and question-answering method based on a large language model as described above to achieve automatic parsing of unstructured text, time-series data management, and intelligent question answering.

[0181] Figure 10 To illustrate the unstructured text parsing and question answering system based on a large language model according to Embodiment 5 of this application, reference will be made below. Figure 10 The unstructured text parsing and question answering system based on a large language model, according to embodiments of this application, is described in detail.

[0182] The unstructured text parsing and question-answering system based on a large language model, as described in this application, includes:

[0183] The text data acquisition module 501 is responsible for automatically reading and preprocessing unstructured text data. In this embodiment, the text data acquisition module 501 automatically reads the raw unstructured text data from the unstructured text file from the data source at a preset period and passes it to the semantic parsing module 502.

[0184] The semantic parsing module 502 integrates a large language model, which is used to perform semantic understanding and structure recognition on the original unstructured text data, obtain key entities and data structures, and output them to the structure template management module 503.

[0185] The structure template management module 503 is used to dynamically generate and maintain JSON structure templates based on the key entities and data structures provided by the semantic parsing module 502.

[0186] The data conversion module 504 is used to convert the original unstructured text data into structured data and output it according to the JSON structure template in the structure template management module 503.

[0187] The time-series database module 505 stores and manages the structured data output by the data transformation module 504 according to the time dimension.

[0188] The question-and-answer processing module 506 is used to receive and process natural language query information input by the user to obtain time information and query intent.

[0189] The data retrieval module 507 is used to retrieve and aggregate the corresponding structured data from the time series database module 505 based on the time information and query intent processed by the question-and-answer processing module 506, and then pass it to the intelligent question-and-answer module 508.

[0190] The intelligent question-answering module 508 integrates a large language model, which is used to generate question-answering results and trend analysis based on the user's query intent and the structured data retrieved by the data retrieval module 507, and output them to the visualization output module 509.

[0191] The visualization output module 509 is used to generate formatted results from the question-and-answer results and trend analysis and then visualize them. The formatted results include natural language text, visual charts, and structured tables.

[0192] In the embodiments of this application, the visualization output module 509 supports the output of various chart types, including line charts, bar charts, scatter plots, heatmaps, tables, and other structured output formats such as lists.

[0193] In the embodiments of this application, the semantic parsing module 502 includes: an entity recognition submodule for extracting key entities from text data; a relation extraction submodule for identifying semantic relationships between key entities; and a structure inference submodule for inferring the logical structure of the text data and the field types of key entities.

[0194] In the embodiments of this application, the time-series database module 505 includes: a storage engine that adopts a distributed storage architecture and supports large-scale data storage; an index manager for creating and managing multidimensional indexes, including time dimension indexes, key field indexes, and full-text search indexes; and a caching module for caching frequently accessed data in memory to improve query performance.

[0195] In the embodiments of this application, the question-answering processing module 506 includes: a natural language understanding submodule for analyzing the semantics and intent of user questions; a time parsing submodule for extracting standardized time information from the input natural language query information; and a query planning submodule for formulating the optimal data retrieval strategy.

[0196] It will be understood by those skilled in the art that the above descriptions are merely preferred embodiments of this application and are not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for unstructured text parsing and question answering based on a large language model, comprising: Retrieve raw unstructured text data from the data source according to a preset cycle; Deep semantic analysis is performed on the original unstructured text data using a large language model. Key entities are extracted from the semantic analysis results through named entity recognition. These key entities include time, location, personnel, equipment, quantity, and event. Dependency syntax is used to analyze the semantic relationships and logical structure between key entities. Based on the semantic relationships and logical structure, the field types and value ranges of the key entities are inferred, thereby obtaining data structure information. Based on the key entities and the data structure information, dynamically generate or update existing JSON structure templates, including: building and maintaining a structure template library based on preset basic structure templates for various common text types; automatically updating the structure templates according to the field types of newly identified key entities; and performing version control and backward compatibility processing on the updated structure templates. According to the JSON structure template, the original unstructured text data is converted into standard structured data; The structured data is stored in a time-series database and a multi-dimensional index is established, including: converting the time information in the original unstructured text data into a standard timestamp format, establishing a hierarchical directory structure by year, month, and day, and establishing a time dimension index, a key field index, and a full-text search index. Obtain time information and query intent from the natural language query information entered by the user; Based on the query intent and the structured data obtained from the time series database according to the time information, question-answering results or trend analysis results are generated through a large language model.

2. The method according to claim 1, wherein, The steps of obtaining time information and query intent from the natural language query information input by the user include: identifying absolute time, relative time, and time range expressions in the query information; classifying user questions according to query intent, including single-point queries, range queries, trend analysis, and statistical summaries; and converting various expressions of time into standard query parameters to achieve parameter normalization.

3. The method for unstructured text parsing and question answering based on a large language model according to claim 1, characterized in that, Before the step of generating question-answering results or trend analysis results through a large language model based on the query intent and the structured data obtained from the time series database according to the time information, the method further includes: determining the data retrieval range according to the time information and obtaining structured data from the time series database; The step of determining the data retrieval range based on time information and obtaining structured data from the time series database includes: for a single point-in-time query, locating and reading the JSON file corresponding to the date; For multi-time point or trend queries, relevant data are extracted by time series and assembled into a time series array; A field filtering mechanism is adopted to extract only the data of relevant fields based on the query intent in order to improve processing efficiency.

4. The method for unstructured text parsing and question answering based on a large language model according to claim 1, characterized in that, The step of generating question-and-answer results or trend analysis results based on the query intent and the structured data obtained from the time-series database according to the time information through a large language model includes: combining the obtained structured data with the query information into complete context information; performing logical reasoning and correlation analysis on the complete context information through a large language model, and generating answer text, data tables or visualization charts according to the type of query intent.

5. The method for unstructured text parsing and question answering based on a large language model according to claim 1, characterized in that, The large language model employs a pre-trained neural network model based on an attention mechanism.

6. A system for unstructured text parsing and question answering based on a large language model, employing the method for unstructured text parsing and question answering based on a large language model as described in any one of claims 1-5, characterized in that, include: The text data acquisition module is used to read raw unstructured text data from unstructured text files; The semantic parsing module uses a large language model to perform semantic understanding and structure recognition on the raw unstructured text data, and obtains key entities and data structures; The structure template management module is used to dynamically generate and maintain JSON structure templates based on the key entities and the data structure; The data conversion module is used to convert the original unstructured text data into structured data according to the JSON structure template; The time-series database module is used to store and manage structured data along the time dimension; The question-and-answer processing module is used to process the natural language query information input by the user and obtain time information and query intent; The data retrieval module is used to retrieve and aggregate the corresponding structured data from the time-series database module based on the time information and the query intent, and then pass it to the intelligent question answering module. The intelligent question-answering module utilizes a large language model to generate and output question-answering results and trend analysis based on the query intent and the corresponding structured data.

7. The unstructured text parsing and question-answering system based on a large language model according to claim 6, characterized in that, The semantic parsing module includes: The entity recognition submodule is used to extract key entities from text data; The relation extraction submodule is used to identify the semantic relationships between the key entities; The structure inference submodule is used to infer the logical structure of the text data and the field types of the key entities.

8. The unstructured text parsing and question-answering system based on a large language model according to claim 6, characterized in that, The time-series database module includes: a storage engine that adopts a distributed storage architecture and supports large-scale data storage; an index manager for multi-dimensional indexing, including time-dimensional indexes, key field indexes, and full-text search indexes; and a caching module for memory caching of frequently accessed data to improve query performance.

9. The unstructured text parsing and question-answering system based on a large language model according to claim 6, characterized in that, The question-answering processing module includes: a natural language understanding submodule, used to analyze the semantics and intent of user questions; a time parsing submodule, used to extract standardized time information from the input natural language query information; and a query planning submodule, used to formulate the optimal data retrieval strategy.

10. The unstructured text parsing and question-answering system based on a large language model according to claim 6, characterized in that, It also includes a visualization output module, which is used to generate and output formatted results from the question-and-answer results and trend analysis; The formatted results include natural language text, visual charts, and structured tables; The visualization output module supports multiple chart types, including line charts, bar charts, scatter plots, heatmaps, tables, and lists.

Images