Storage and query system and method based on intelligent body length text tool output

Abstract

The invention belongs to the technical field of artificial intelligence, and discloses a storage and query system and method based on intelligent body length text tool output, and the system comprises a storage module, an original response receiving module, a standardization processing module, a fragmentation module and an indexing module, cutting the standardized text into searchable fragments containing text data according to a preset strategy, and generating corresponding index entries and metadata for each searchable fragment; the NL2SQL module is used for generating an SQL query according to user input; the method comprises the following steps: executing an aggregation and abstract module, executing SQL query, matching index entries or metadata to obtain text data corresponding to a searchable fragment, performing aggregation processing on the text data, and calling a large model to generate a natural language answer according to the aggregated text data, so that the problem of context overload caused by a full-amount long text input model is avoided; meanwhile, the limitation of the model input upper limit is avoided, and the operation efficiency and adaptability of the system are improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a storage and retrieval system and method based on the output of a long text tool for intelligent entities. Background Technology

[0002] With the rapid iteration of large language model technology, intelligent agents driven by it have been widely applied in various fields such as office automation, enterprise data analysis, intelligent customer service, and scientific literature review. One of the core advantages of these intelligent agents is their ability to flexibly utilize external tools (such as professional database query tools, vertical domain search engines, industry data APIs, and literature retrieval tools) to compensate for the shortcomings of their own inherent knowledge in terms of timeliness, professionalism, and data coverage, thereby outputting more accurate and valuable results.

[0003] However, in practical applications, the results returned by these external tools often exhibit characteristics of "long texts and high-density information," posing a significant challenge to the efficient processing by intelligent agents. For example, the output of industry in-depth analysis tools typically covers multiple dimensions such as market size, competitor dynamics, and user preferences, often reaching tens of thousands of words; the results returned by database batch query tools contain hundreds of data records and related dimension descriptions, sometimes exceeding one hundred thousand words. These long text tool outputs are key information sources for intelligent agents to complete complex tasks, but existing processing methods have the following obvious shortcomings: Direct input to the model: Provide the model with all the returned content of the tool as context, but this method can lead to excessively long contexts, high computational overhead, and may even exceed the model's input limit.

[0004] Abstract compression: This provides a simple summary of the returned results, but it is easy to lose detailed information, which affects the accuracy and reliability of the agent.

[0005] Therefore, how to effectively process long text outputs within the intelligent agent framework, ensuring both the preservation of complete information and the efficient extraction of necessary parts based on the problem, has become an urgent technical problem to be solved. Summary of the Invention

[0006] This application aims to address the shortcomings of existing technologies by proposing a storage and query system and method based on the output of intelligent long text tools, in order to solve the technical problem that existing technologies cannot simultaneously meet the needs of static and dynamic urban governance and have low governance efficiency.

[0007] To achieve the objectives of this application, in a first aspect, this application provides a storage and retrieval system based on the output of an intelligent long text tool, the system comprising: The database module is used to receive the raw response from the intelligent agent long text tool and perform standardization processing on the raw response to obtain standardized text. The sharding and indexing module is used to cut standardized text into searchable fragments containing text data according to a preset strategy, and to generate corresponding index items and metadata for each searchable fragment; The data storage module is used to store the searchable fragments, index items, and metadata; The NL2SQL module is used to generate SQL queries based on user input; The aggregation and summarization module is used to perform queries based on SQL and to obtain text data of corresponding searchable fragments by matching index items or metadata. After aggregating the text data, the large model is called to generate natural language answers based on the aggregated text data.

[0008] Furthermore, the system also includes: The SQL validation and sandbox module is used to validate SQL statements and send the validated SQL statements to the execution aggregation and summary module.

[0009] By adding an SQL validation and sandbox module, a validation step is added after the SQL query is generated by the NL2SQL module and before the query is executed by the aggregation and summary modules. This solves the data security risks (such as malicious operation and unauthorized access) that may be caused by the direct execution of unvalidated SQL and the problem of invalid queries consuming system resources. It can filter illegal and insecure SQL, ensure the security of data in the data storage module, and avoid invalid queries occupying system resources, thereby improving the reliability and resource utilization efficiency of the system's query operations.

[0010] Furthermore, the verification method for SQL verification includes at least one of static verification, cost estimation, or permission verification.

[0011] Static validation can quickly identify SQL syntax errors and dangerous operation statements, preventing illegal query execution; cost estimation can predict the resource consumption of queries in advance, preventing high-cost queries from blocking the system; permission validation ensures that SQL only accesses data within the authorized scope. This setting solves the problem of insufficient coverage of a single validation method, ensuring the security, efficiency, and compliance of SQL execution from multiple dimensions such as syntax compliance, resource controllability, and permission compliance.

[0012] Furthermore, the standardization processing of the original response includes at least one of the following methods: character normalization, noise reduction, text fingerprinting for deduplication, or metadata extraction.

[0013] This solution addresses the issues of inconsistent formats, redundant content, duplicate data, and missing metadata in the raw responses of intelligent agents. It transforms the raw responses into standardized, concise, and high-quality text, providing consistent and high-quality input data for subsequent sharding and indexing modules. This improves the efficiency of sharding and indexing, while reducing redundant data usage in the storage stage and ensuring data consistency and availability within the system.

[0014] Furthermore, the preset segmentation strategy of the segmentation and indexing module includes at least one of the following strategies: semantic paragraph, length threshold constraint, topic boundary detection, or special retention rule segmentation strategy.

[0015] It solves the problem of easily cutting off semantic logic and losing special content structures such as tables / code blocks when segmenting long texts. It can ensure the semantic integrity of each searchable segment, while retaining the structural information of special content. This enables subsequent segment-based searches to accurately match semantically related content, improves the accuracy of search results, and enhances the information integrity and usability of searchable segments.

[0016] Furthermore, the index terms include vector embedding, full-text index terms, and inverted index terms.

[0017] By setting the index terms as a combination of vector embeddings, full-text index terms, and inverted index terms, the problem that a single type of index cannot simultaneously meet the needs of semantic retrieval and precise retrieval is solved: vector embeddings support semantic similarity-based retrieval, matching the semantic needs of users' natural language queries; full-text index terms and inverted index terms support keyword-based queries such as precise retrieval and Boolean retrieval. This multi-dimensional index configuration achieves comprehensive coverage of retrieval capabilities, improves the flexibility and efficiency of retrieval, better adapts to user query needs in different scenarios, and improves the accuracy of retrieval results.

[0018] Furthermore, the data storage module uses a Milvus vector database to store index items and metadata, and a PostgreSQL database to store searchable fragments; the data storage module supports vector similarity search, exact search, Boolean search, and fuzzy search.

[0019] This setup addresses the challenge of a single database simultaneously and efficiently supporting both vector data storage and semantic retrieval, as well as text data storage and precise retrieval. The Milvus vector database, adapted to the storage characteristics of index items and metadata, efficiently manages vector data, supports fast vector similarity retrieval, and accurately matches the semantic needs of users' natural language queries. The PostgreSQL database, adapted to the storage of searchable text fragments, leverages its relational database advantages to stably support precise retrieval, Boolean retrieval, fuzzy retrieval, and other text-based queries, meeting the need for precise keyword matching. The combination of these two databases ensures both the efficiency and rationality of different data storage types, while also covering multi-dimensional retrieval needs such as semantics and precision, improving the comprehensiveness and response efficiency of retrieval. It better adapts to diverse user query scenarios, enhancing the accuracy and speed of search results.

[0020] Secondly, this application provides a method for storing and querying long text outputs from an intelligent agent tool, the method comprising: Receive the raw response from the intelligent agent long text tool, and perform standardization processing on the raw response to obtain standardized text; Standardized text is segmented into searchable segments containing text data according to a preset strategy, and corresponding index items and metadata are generated for each searchable segment; Store the retrieveable fragments, index items, and metadata; Generate an SQL query based on user input; The system performs a query based on SQL and obtains the corresponding searchable text data by matching index items or metadata. After aggregating the text data, it calls a large model to generate a natural language answer based on the aggregated text data.

[0021] Thirdly, this application provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described storage and retrieval method based on the output of an intelligent agent long text tool.

[0022] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of the above-described storage and retrieval method based on the output of an intelligent long text tool.

[0023] The beneficial effects of this application are: This application, on the one hand, segments long texts into semantically complete searchable fragments through a segmentation and indexing module. The subsequent execution of the aggregation and summarization module only matches and retrieves the corresponding required fragments based on SQL queries. This avoids the context overload problem caused by the full long text input model, reduces computational overhead, and avoids the limitation of the upper limit of model input, thereby improving the system's operating efficiency and adaptability.

[0024] On the other hand, the database module standardizes the original response, the fragmentation and indexing module preserves the semantic integrity of the fragments, and the data storage module stores the complete searchable fragments and indexes / metadata, thus achieving complete retention of long text information. The subsequent execution aggregation and summarization module aggregates the matched complete fragment content to generate answers, avoiding the loss of detailed information and ensuring the accuracy and reliability of the agent's output answers. Attached Figure Description

[0025] Figure 1 A block diagram of a storage and retrieval system based on the output of an intelligent long text tool according to the present invention; Figure 2 The flowchart of this invention is a method for storing and querying long text outputs from an intelligent body tool. Detailed Implementation

[0026] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0027] In the description of this invention, unless otherwise specified and limited, it should be noted that the terms "installation", "connection" and "linking" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two components. They can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0028] The present invention provides a storage and query method based on the output of an intelligent body long text tool. The execution subject includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in the embodiments of this application: a server, a terminal, or a computer. In other words, a storage and query method based on the output of an intelligent body long text tool can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0029] Reference Figure 1 In a first aspect, the present invention discloses a storage and retrieval system based on the output of an intelligent long text tool, the system comprising: The database module is used to receive the raw response from the intelligent agent long text tool and perform standardization processing on the raw response to obtain standardized text. The standardization process for the original response includes at least one of the following methods: character normalization, noise reduction, text fingerprinting for deduplication, or metadata extraction. For example, character normalization includes using UTF-8 encoding, standardizing whitespace, and removing garbled control characters. Noise reduction includes removing templated prompts: removing common templated prompts (such as "The following is the answer:"), duplicate headers and footers, and cleaning up redundant HTML / Markdown content. Text fingerprinting for deduplication includes generating a raw_hash (SHA256) and checking the original document library for deduplication; if a complete duplicate is found, the reference relationship is recorded and the process ends (or the metadata is merged). Metadata extraction includes automatically identifying the language, possible document type (log / answer / report), and whether it contains tables or code snippets (recorded as booleans).

[0030] In some embodiments, the original response includes information such as the original text, tool name, input parameters, and timestamp. The original text is saved as a "full-text record," recording a unique full_id (UUID recommended), the receiving time, the source tool, call_id, etc.

[0031] Standardized text is pushed to a "shard queue," which supports batch processing and retry mechanisms. If sharding of very large text (greater than 5MB) fails, it is first written to "streaming shards" temporary storage and an alarm is generated.

[0032] This setup enables complete backup and traceability of the original data. If discrepancies occur in the sharding / retrieval results, the original text can be retrieved and verified using the full_id. At the same time, unique identifiers and metadata (source tools, call IDs, etc.) make data management more standardized and facilitate subsequent auditing and traceability.

[0033] The batch processing mechanism reduces the frequency of calls to the sharding module, improves the efficiency of batch processing of long texts, and reduces system resource consumption; the retry mechanism avoids the loss of standardized text due to temporary network / module failures, thus improving the reliability of data flow.

[0034] Avoiding large text files (greater than 5MB) from blocking the regular sharding queue ensures the queue's processing efficiency; temporary storage prevents the loss of large text data; and the alarm mechanism allows operations and maintenance personnel to intervene and handle anomalies in a timely manner, improving the system's fault tolerance and stability.

[0035] In some embodiments, the data for monitoring the data entry module includes receiving rate, deduplication hit rate, data entry delay, and anomaly rate.

[0036] This setup allows for real-time monitoring of the load pressure on the inbound module, timely detection of data influx peaks, prevention of module overload and crashes, and ensures the continuity of data reception and system stability. Evaluating the effectiveness of the deduplication strategy—a high hit rate indicates that duplicate data is effectively filtered, reducing redundant storage usage; a low hit rate can trigger deduplication rule optimization, improving storage resource utilization.

[0037] Identify efficiency bottlenecks in the data ingestion process, such as time consumption in standardization processing and queue congestion. Reduce latency by optimizing steps (e.g., batch processing tuning) to ensure data can quickly enter the sharding and indexing process, improving the overall system response speed. Promptly capture errors during the data ingestion process, such as invalid data formats or storage failures, quickly locate and intervene at the fault point, reduce the risk of data loss, and improve system reliability and data integrity.

[0038] The sharding and indexing module is used to segment standardized text into searchable fragments according to a preset strategy, and to generate corresponding index entries and metadata for each searchable fragment. For example, standardized text is segmented into searchable fragments based on semantic paragraphs and length thresholds, and metadata, vector embeddings, and full-text / inverted index entries are generated for each fragment.

[0039] In some embodiments, the preset segmentation strategy of the sharding and indexing module includes at least one of the following strategies: semantic paragraph, length threshold constraint, topic boundary detection, or special retention rule segmentation strategy.

[0040] Semantic paragraphing strategies prioritize paragraphs, headings, or natural segmentation points, including identifying line breaks, heading markers, numbering, or lists.

[0041] In the length threshold constraint strategy, when the paragraph length exceeds N tokens (N is usually 500-2000, depending on the retrieval or generation model), the paragraph is divided into sentences or clauses and an overlapping window of 10-20% or 50-200 tokens is retained to ensure contextual coherence.

[0042] A topic boundary detection strategy can use a lightweight semantic model to detect topic boundaries, avoiding cutting at the points where topics cross boundaries.

[0043] Special preservation rule segmentation strategies preserve tables, code blocks, and citations as individual pieces or mark fragment types separately to maintain structural information.

[0044] The index terms include vector embeddings, full-text index terms, and inverted index terms. Embedding refers to the process of converting unstructured data such as text and images into low-dimensional numerical vectors that can be recognized by computers. These vectors retain the semantic information of the original data; the closer the distance between vectors, the higher the semantic similarity of the corresponding content. The vector embeddings are generated using the qwen3-embedding model and processed by L2 normalization. The full-text index terms include title, keywords, named_entities, and content fields. The metadata includes fragment source, location, and type information. The full-text index terms include writing to a full-text search engine (ES / Postgres full-text) to generate index fields (title, keywords, named_entities, content) that can perform precise / Boolean / fuzzy searches. The inverted index terms indicate the position of the full-text paragraph from which each record originates, facilitating tracing back to the original text.

[0045] In some embodiments, batch processing of embeddings is performed during batch processing to save on invocation costs; streaming data import ensures low-latency retrieval availability.

[0046] If the retrieved fragments are highly fragmented, adjacent fragments can be dynamically merged in the backend for secondary reordering or summarization.

[0047] The data storage module is used to store the index items, metadata, and text data of the searchable fragments; it provides high-quality, low-latency semantic similarity retrieval, while retaining a text library or search engine for precise / Boolean / fuzzy retrieval, and jointly providing the most relevant results with keyword retrieval.

[0048] In some embodiments, the data storage module uses a Milvus vector database to store index items and metadata and a PostgreSQL database to store text data.

[0049] Milvus is an open-source vector database specifically designed for storing, indexing, and managing large-scale vector data (such as the embedding vectors of searchable fragments in this solution). Its core function is to support efficient vector similarity retrieval. Through optimized indexing algorithms (such as IVF, HNSW, etc.), it can quickly match semantically similar results from massive vectors, making it a commonly used tool for implementing "semantic similarity retrieval" scenarios.

[0050] For example, the vector database Milvus stores the following fields: fragment_id (reference), vector, metadata_pointer (pointer to the original text / fragment table), timestamp.

[0051] PostgreSQL is an open-source relational database management system (RDBMS) that supports standard SQL syntax and has rich extensibility capabilities (such as full-text search, transaction processing, and complex join queries). Its core function is to store structured data and provide accurate query, sorting, and filtering capabilities. It is one of the commonly used text and structured data storage tools in enterprise scenarios.

[0052] For example, a PostgreSQL database is used to store fragmented text and structured fields, supporting Boolean queries and sorting.

[0053] The data storage module supports vector similarity search, exact search, Boolean search, and fuzzy search. For vector similarity search, the input or user input / question is vectorized and then queried in the Milvus vector database. Alternatively, a PostgreSQL database can be used for full-text search, using a full-text index combined with whitelists, time, source, and other conditions to filter the recall set.

[0054] The NL2SQL module is used to generate SQL queries based on user input. In some embodiments, the NL2SQL module translates user intent into secure SQL queries for use by the data layer's structured queries; it outputs candidate SQL, confidence scores, and interpretation paths.

[0055] NL2SQL is a technology for "Natural Language to SQL". In this solution, the core function of the NL2SQL module is to translate user input (such as user questions and session context) into secure and compliant SQL queries, while outputting candidate SQL, confidence level, and interpretation path, serving as a bridge between "user natural language requirements" and "structured queries at the data layer".

[0056] For example, the input could be: user_question, conversation_context (recent messages); schema_summary (a whitelist description of tables / fields with limited length—keep it concise and structured); access_control(which tables / columns are accessible, column sensitivity level); examples (few-shot templates, if using few-shot); generation_constraints (allow only SELECT, maximum number of joins, disable subqueries or joins, etc.).

[0057] The generation strategy can be: Generates only SELECT statements and provides structured output outside the SQL statement: {"sql":"...","confidence":0.xx,"explain":"..."}. Example question-answer pair: xxx. Generates 5 candidate SQL statements each time. Each candidate has a confidence score (the probability output by the model or the frequency based on multiple samplings). Output example: candidates:[ {sql_text:"...",confidence:0.85,explain:"xxx"}.

[0058] SQL is an abbreviation for "Structured Query Language," a standardized language for manipulating and querying database data. In this solution, SQL refers to query commands that the data storage module (such as PostgreSQL) can recognize, used to accurately retrieve matching, searchable fragments of data from stored fragmented text and structured fields.

[0059] The SQL validation and sandboxing module is used to validate SQL statements. Once validated, the SQL statements are sent to the execution aggregation and summary module. Any SQL statements generated by the model are rigorously reviewed before execution to ensure security and low resource consumption; execution is performed in a sandbox or read-only copy when necessary.

[0060] The SQL verification includes at least one of the following verification methods: static verification, cost estimation, or permission verification.

[0061] The static validation process includes parsing the SQL into an AST (Abstract Syntax Tree) to facilitate subsequent checks (avoiding the use of only regex).

[0062] Cost estimation uses database EXPLAIN or statistical information-based estimation to determine full table scan, expected number of rows returned, and expected IO / CPU. If the cost exceeds a set threshold, the process is rejected or downgraded (e.g., by adding a WHERE condition or sampling execution).

[0063] Permission verification includes whitelist verification, checking whether the table or column exists at the access_control whitelist level, and requiring special approval or masking / de-identification for access to sensitive data (PII).

[0064] In some embodiments, SQL validation also includes security policy checks: Only SELECT is allowed; DDL / DML operations such as INSERT / UPDATE / DELETE / CREATE / DROP are prohibited.

[0065] Check for dangerous statements such as UNION, EXECUTE, PROCEDURE CALL, and external functions.

[0066] Complexity estimation involves parsing the number of table joins in FROM, checking for Cartesian products (without ON conditions), and examining aggregation / subquery depths and DISTINCT usage.

[0067] Injection / placeholder checks; use parameterized placeholders for all external inputs; prohibit string concatenation.

[0068] Execution permissions and audit flags are set, and SQL queries are bound to user_id / session_id and recorded.

[0069] The SQL validation and sandbox module uses a read-only database replica or restricted user environment with resource quotas, maximum runtime, and maximum number of rows returned (default 1000). It employs a tiered execution strategy: lightweight queries are executed directly on the primary database (short time, low cost); high-cost or sensitive queries are pre-executed in the sandbox / replica. Execution is monitored, recording the execution plan, actual execution time, number of rows scanned, and I / O usage.

[0070] The aggregation and summarization module performs queries based on SQL and obtains text data of corresponding searchable fragments by matching index items or metadata. After aggregating the text data, the large model is called to generate natural language answers including source and confidence level based on the aggregated text data.

[0071] The large models in this embodiment include closed-source models, such as GPT-4, which is adapted to complex semantic synthesis and long text summarization tasks, can accurately understand the correlation logic of multi-segment information, and is suitable for scenarios with high requirements for answer quality (such as natural language answer generation for enterprise data analysis and scientific literature review). Wenxin Yiyan 4.0 has stronger adaptability to Chinese semantics and is suitable for tool output processing scenarios centered on long Chinese texts (such as aggregated summaries of Chinese industry reports and domestic policy documents). Tongyi Qianwen 3.0: The "qwen3-embedding" (an embedding model from the Tongyi Qianwen series) used in the ecosystem and solutions has higher adaptability and can improve the collaborative efficiency of vector retrieval and semantic synthesis.

[0072] This also includes open-source large-scale models, such as: Llama 3, which supports long context windows, adapts to the "long text semantic synthesis after secondary fragmentation / merging" requirement in this solution, and can be deployed privately to ensure data security; ChatGLM4, which has outstanding Chinese processing capabilities and supports long text summarization and multi-source information fusion, suitable for generating aggregated answers from Chinese tools; and Qwen14B / 72B, which, like "qwen3-embedding" used in the solution, belongs to the Tongyi Qianwen ecosystem, and whose vector features are consistent with the semantic understanding logic of large-scale models, reducing cross-model information bias.

[0073] Its execution steps include: 1. Pagination / Batch Reading: Avoid fetching a large number of rows at once (using pagination and cursor).

[0074] 2. Desensitize the results (e.g., sensitive information such as name and ID number).

[0075] 3. Perform secondary segmentation / merging on long texts (merging adjacent segments semantically to generate a summary).

[0076] 4. Multi-source fusion: If the answer depends on multiple fragments / tables, aggregate at the rule level (group by, top-N) and then send to LLM (Large Language Model) for semantic synthesis.

[0077] The steps involved in generating a natural language answer include: 1. Summary generation prompt: "Please clearly and concisely summarize your input; do not generate any guesses. Output format: Answer text, SQL statement used, confidence level, original query results." 2. If LLM generation times out or fails: Return structured results (table / JSON) with the annotation "No natural language summary generated".

[0078] This system employs integrated processing and retrieval of multimodal data, innovatively combining vectorized semantic retrieval, full-text / Boolean retrieval, and structured data querying. This is similar to a "full-stack retrieval" approach, handling different types of queries through a unified entry point, significantly improving the comprehensiveness and accuracy of information retrieval.

[0079] In the final stage, the system uses large models (such as qwen3-max) to perform semantic aggregation and summary generation on the retrieved multi-source and multi-modal results, rather than simply listing them. Ultimately, it provides users with direct, natural answers that include source and confidence information, resulting in a better user experience.

[0080] Meanwhile, the secure and controllable NL2SQL (Natural Language to SQL) process is undoubtedly the core innovation of the system. It does not simply throw user questions at a large model, but builds a complete security chain that includes strict SQL validation, cost estimation, and sandbox execution to ensure the legality, security, and performance of database queries, effectively preventing SQL injection and resource abuse.

[0081] Secondly, this invention discloses a storage and retrieval method based on the output of an intelligent agent long text tool, applied to the aforementioned system, comprising the following steps: Receive the raw response from the intelligent agent long text tool, and perform standardization processing on the raw response to obtain standardized text; Standardized text is segmented into searchable segments containing text data according to a preset strategy, and corresponding index items and metadata are generated for each searchable segment; Store the retrieveable fragments, index items, and metadata; Generate an SQL query based on user input; The system performs a query based on SQL and obtains the corresponding searchable text data by matching index items or metadata. After aggregating the text data, it calls a large model to generate a natural language answer based on the aggregated text data.

[0082] Thirdly, the present invention discloses a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described storage and retrieval method based on the output of an intelligent agent long text tool.

[0083] Fourthly, the present invention discloses a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the steps of the above-described storage and retrieval method based on the output of a long text tool for intelligent agents.

[0084] In the description of this specification, the references to terms such as "an embodiment," "some embodiments," "example," "specific example," "a implementation," "a preferred implementation," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0085] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A storage and retrieval system based on the output of an intelligent long text tool, characterized in that, The system includes: The database module is used to receive the raw response from the intelligent agent long text tool and perform standardization processing on the raw response to obtain standardized text. The sharding and indexing module is used to cut standardized text into searchable fragments containing text data according to a preset strategy, and to generate corresponding index items and metadata for each searchable fragment; The data storage module is used to store the searchable fragments, index items, and metadata; The NL2SQL module is used to generate SQL queries based on user input; The aggregation and summarization module is used to execute SQL queries and match index items or metadata to obtain text data of corresponding searchable fragments. After aggregating the text data, the large model is called to generate natural language answers based on the aggregated text data.

2. The storage and retrieval system based on the output of an intelligent agent long text tool according to claim 1, characterized in that, The system also includes: The SQL validation and sandbox module is used to validate SQL queries and send the validated SQL queries to the execution aggregation and summary module.

3. The storage and retrieval system based on the output of an intelligent long text tool according to claim 2, characterized in that, The verification methods for validating SQL queries include at least one of static verification, cost estimation, or permission verification.

4. The storage and retrieval system based on the output of an intelligent long text tool according to claim 1, characterized in that, The standardization processing of the original response includes at least one of the following methods: character normalization, noise reduction, text fingerprinting for deduplication, or metadata extraction.

5. A storage and retrieval system based on the output of an intelligent long text tool according to claim 1, characterized in that, The preset segmentation strategy of the segmentation and indexing module includes at least one of the following strategies: semantic paragraph, length threshold constraint, topic boundary detection, or special retention rule segmentation strategy.

6. The storage and retrieval system based on the output of an intelligent long text tool according to claim 1, characterized in that, The index terms include vector embedding, full-text index terms, and inverted index terms.

7. The storage and retrieval system based on the output of an intelligent agent long text tool according to claim 1, characterized in that, The data storage module uses a Milvus vector database to store index items and metadata, and a PostgreSQL database to store searchable fragments; the data storage module supports vector similarity search, exact search, Boolean search, and fuzzy search.

8. A method for storing and retrieving data based on the output of an intelligent body long text tool, applied to the storage and retrieval system based on the output of an intelligent body long text tool according to any one of claims 1-7, characterized in that, The method includes: Receive the raw response from the intelligent agent long text tool, and perform standardization processing on the raw response to obtain standardized text; Standardized text is segmented into searchable segments containing text data according to a preset strategy, and corresponding index items and metadata are generated for each searchable segment; Store the retrieveable fragments, index items, and metadata; Generate an SQL query based on user input; Execute SQL queries and match index items or metadata to obtain text data of corresponding searchable segments. After aggregating the text data, call the large model to generate natural language answers based on the aggregated text data.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the storage and retrieval method based on the output of the intelligent agent long text tool as described in claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the storage and retrieval method based on the output of the intelligent agent long text tool as described in claim 8.

Images