A method for ingesting and processing unstructured industry documents and dual-track retrieval

Abstract

This invention relates to the fields of artificial intelligence and knowledge management technology, and discloses a method for the ingestion, segmentation, and dual-track retrieval of unstructured industry documents. The method involves dividing the document to be ingested into content units, constructing structured input data for each content unit, and then feeding it into a large language model for parsing to obtain structured output. The method performs hierarchical processing based on the confidence score and routes the content units to either a structured processing channel or a semantic processing channel based on the content type tag, forming a dual-track storage structure. For content units routed to the structured channel, the field mapping results and the schema of the target data table are verified for compatibility before being written into the corresponding data table. For content units routed to the semantic processing channel, the method performs block processing according to semantic boundaries, generates semantic vectors for each semantic block and stores them in a vector database, and establishes soft associations between the semantic blocks and structured entities. Based on the dual-track storage structure and soft association relationships, the method performs structured queries, semantic retrieval, or hybrid queries on the knowledge base. This invention effectively controls costs while ensuring accuracy.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and knowledge management technology, specifically to a method for the storage, tiered processing, and dual-track retrieval of unstructured industry documents. Background Technology

[0002] With the deepening of enterprise digital transformation, the knowledge management and intelligent retrieval of unstructured documents (such as product specifications, industry research reports, technical standards, and operation manuals) have become urgent needs; existing enterprise knowledge base systems mainly adopt the following methods: 1. Keyword Search: Relies on a full-text search engine, matching keywords when a user queries. Disadvantages: Cannot understand semantics; poor performance with synonyms / colloquialisms / cross-language searches; high false negative rate. 2. Fixed-length block RAG system: The document is split into blocks of a fixed number of tokens (e.g., 512 tokens) and then vectorized. Disadvantage: The splitting points may truncate complete semantic units (e.g., a complete product description is split into two blocks), resulting in incomplete or meaningless content fragments being returned during retrieval; 3. Pure vector retrieval type: All content is vectorized before semantic search is performed. For numerical or enumerated data (such as product specifications, price ranges, and lists of places of origin), vector similarity cannot provide accurate filtering and sorting capabilities. 4. Manual annotation type: Relies on manual classification, tagging, and association of knowledge. Disadvantages: Extremely high maintenance costs, difficult to sustain as the amount of knowledge grows, and dependent on the professional level of the annotators. Summary of the Invention

[0003] The purpose of this invention is to solve the above-mentioned problems by designing a method for the entry, segmentation, and dual-track retrieval of unstructured industry documents.

[0004] This invention provides a method for the ingestion, tiered processing, and dual-track retrieval of unstructured industry documents. The method includes the following steps: The documents to be imported into the database are divided into content units. Structured input data is constructed for each content unit and fed into a large language model for parsing to obtain structured output including content type labels, field mapping results, schema compatibility scores and confidence scores. The content is graded according to the confidence score and routed to the structured processing channel or the semantic processing channel according to the content type tag, forming a dual-track storage structure. For content units routed to the structured channel, the field mapping results and the schema of the target data table are verified for compatibility before being written to the corresponding data table; For the content units routed to the semantic processing channel, they are divided into blocks according to semantic boundaries. Semantic vectors are generated for each semantic block and stored in the vector database. Soft associations are established between the semantic blocks and structured entities. Based on a dual-track storage structure and soft association relationships, the knowledge base can be used for structured queries, semantic retrieval, or hybrid queries.

[0005] Optionally, in the first implementation of the present invention, the document to be imported into the database is segmented into content units, structured input data is constructed for each content unit, and then fed into a large language model for parsing to obtain a structured output including content type tags, field mapping results, schema compatibility scores, and confidence scores, including: The document to be imported is parsed into multiple independent text paragraphs or text blocks according to the page, paragraph or chapter structure, and used as content units, and its position information in the original document is recorded. For each content unit, the original text and the corresponding target data table schema information are organized into structured input data including text and field definitions according to a preset template; The structured input data is fed into the large language model, and field values ​​are extracted from the original text according to the schema definition to determine the content type; It receives the structured output returned by the large language model and obtains the content type label, field mapping results, schema compatibility score, and confidence score calculated by multi-dimensional weighting for the content unit.

[0006] Optionally, in a second implementation of the present invention, the step of performing grading based on the confidence score includes: The confidence score is compared with a preset first threshold and a second threshold, wherein the first threshold is greater than the second threshold; When the confidence score is greater than or equal to the first threshold, it is determined to be of high confidence level. Then all fields of the content unit are completely matched with the target data table schema, and no tags are added to the content unit. When the confidence score is greater than or equal to the second threshold and less than the first threshold, it is judged as medium confidence level. If some fields of the content unit match the target data table schema, there are missing fields or incomplete type compatibility, and a supplementary mark is added to the content unit. When the confidence score is less than the second threshold, it is judged as a low confidence level. In this case, the content unit is incompatible with the target data table schema and will not enter the routing process.

[0007] Optionally, in a third implementation of the present invention, the step of performing compatibility verification on the field mapping results and the schema of the target data table for the content units routed to the structured channel and then writing them into the corresponding data table includes: Compare the actual data types of each field in the field mapping result with the data types defined in the corresponding fields of the target data table for type compatibility. For fields marked as enumeration type in the target data table schema, check whether the values ​​in the field mapping results exist in the predefined list of valid enumeration values ​​for that field. Summarize the inspection results of all fields. If there are fields with incompatible types or invalid enumeration values, return the result with modification suggestions. After all fields pass the inspection, construct a database write statement and execute the UPSERT operation to write the relational data table.

[0008] Optionally, in the fourth implementation of the present invention, the step of segmenting the content units routed to the semantic processing channel according to semantic boundaries, generating semantic vectors for each semantic segment and storing them in a vector database, and establishing soft associations between the semantic segments and structured entities includes: Obtain the content units routed to the semantic processing channel, segment them by title level, and perform topic conversion segmentation when the semantic similarity between adjacent paragraphs is lower than the third threshold; The text content of each semantic block is input into the vector embedding model, and the fixed-dimensional semantic vector output by the vector embedding model is used as the vector representation of the current semantic block. A unique identifier is generated for each semantic block, and the block identifier, the original text of the semantic block, the semantic vector corresponding to the semantic block, and the source information of the semantic block are associated and stored in a vector database using the HNSW index structure. Get the unique values ​​of the pre-configured associated fields, summarize them into a candidate entity set and load them into memory, and refresh incrementally as the structured data table is updated; For each entity in the candidate entity set, perform transliteration normalization, case normalization, and abbreviation expansion processing to map multiple spellings of the same entity to the same standard identifier; For each semantic block, extract text fragments and calculate normalized similarity based on edit distance with candidate entity names. Records of candidate entities that are higher than the fourth threshold are used as the matching items of the corresponding semantic block in the associated field. Cross-validation is performed when a semantic block matches candidate entities in multiple associated fields; conflict resolution is performed when a semantic block matches multiple candidate entities in the same associated field. The resolved soft association references are stored in semantic block records without modifying the schema of the structured table; When new entity data is entered into the structured data table, all existing semantic blocks that have not yet established new entity associations are scanned and matched to create soft association references for the semantic blocks.

[0009] Optionally, in the fifth implementation of the present invention, the step of obtaining the content unit routed to the semantic processing channel, segmenting it according to the title level, and performing topic conversion segmentation when the semantic similarity between adjacent paragraphs is lower than a third threshold includes: The text structure of content units is analyzed, the heading level markers in the text are identified, and the content under different headings is divided into different semantic blocks; For consecutive paragraphs at heading levels, the semantic similarity between adjacent paragraphs is calculated. When the semantic similarity is lower than a preset similarity threshold, it is determined that a topic shift has occurred between adjacent paragraphs, and the paragraphs are split at that position. For each semantic block obtained after segmentation, verify whether the semantic block includes a complete knowledge unit. When a semantic block does not include a complete knowledge unit, adjust the boundary of the semantic block and incorporate the missing part from the adjacent semantic block until each semantic block includes a complete knowledge unit.

[0010] Optionally, in the sixth implementation of the present invention, the step of performing cross-validation when a semantic block matches candidate entities on multiple associated fields includes: Obtain the matching results of semantic blocks on each associated field, determine whether the candidate entities matched by the same semantic block on different associated fields have associated records in the structured data table, and increase the confidence of the matching results when associated records exist.

[0011] Optionally, in the seventh implementation of the present invention, the structured query includes: Based on the intent category determined by the route, the conditional parameters involved in the query statement are extracted into structured query parameters; A parameterized SQL query statement is constructed based on the extracted parameters. The query statement is executed on the target data table to obtain structured data results. When the number of returned results is zero, the system automatically switches to semantic search mode.

[0012] Optionally, in the eighth implementation of the present invention, the semantic retrieval includes: The query input vector is embedded into a model and converted into a query vector. Cosine similarity is then calculated in a vector database. Obtain all semantic blocks in the vector database whose cosine similarity to the query vector is greater than or equal to a preset similarity threshold as preliminary search results; The first preset number of search results, sorted by similarity score from highest to lowest, are input into the semantic re-ranking model. The semantic re-ranking model calculates a relevance score for each input result, sorts the results by relevance score from highest to lowest, and then selects the second preset number of input results. The selected input results are then deduplicated.

[0013] Optionally, in the ninth implementation of the present invention, after performing structured queries, semantic retrieval, or hybrid queries on the knowledge base based on the dual-track storage structure and soft association relationships, the method further includes: Periodically retrieve query logs from the production environment, and filter out zero-result queries where both structured and semantic searches return empty results, as well as low-similarity queries where the highest similarity in semantic search is lower than a preset similarity threshold; The selected query input vectors are embedded into a model and converted into query vectors. A density-based clustering algorithm is then used to automatically cluster the query vectors using the cosine distance between them as a metric. The query closest to the cluster center in each cluster is selected as a representative example. The clusters are sorted from high to low according to the query frequency, and a knowledge gap report is generated, which includes the gap topic, representative query examples, query frequency, and suggested directions for supplementation.

[0014] Optionally, in the tenth implementation of the present invention, before performing structured queries, semantic retrieval, or hybrid queries on the knowledge base based on the dual-track storage structure and soft association relationships, a query preprocessing step is further included: Obtain the standard language space identifier of the knowledge base configuration, translate the query statement to the standard language space, and obtain the standard language version of the query statement; The query statements in the standard language version are processed for synonym normalization, term locking is performed, the domain-specific terminology list maintained by the knowledge base is obtained, and words belonging to the domain-specific terminology list are identified in the query statements.

[0015] Optionally, in the eleventh implementation of the present invention, the preprocessed query statement is input into a three-layer progressive routing system to determine the query intent and query mode, including: The first layer is regular expression keyword routing: it retrieves the pre-configured keyword rule table, and matches the query statement with the regular expressions in the keyword rule table in descending order of rule priority. When a match is successful, the corresponding query action is executed directly. If the first layer fails to match, proceed to the second layer of centroid vector matching: obtain the centroid vectors of each intent category maintained by the system, convert the query statement into a query vector, calculate the cosine similarity with each centroid vector, and take the intent category with the highest similarity as the matching result; When the second layer fails to find a match, the system proceeds to the third layer: intelligent routing using the large language model. This involves constructing prompt words from the query statement and a small sample set as input to the large language model, which then outputs the intent classification result.

[0016] Optionally, in the twelfth implementation of the present invention, the hybrid query step: When the routing result corresponds to a mixed query mode, the structured query process and the semantic retrieval process are started in parallel to obtain the list of structured query results and the list of semantic retrieval results, respectively. For each result item in the structured query result list, obtain the rank number of that item in the structured query result list; for each result item in the semantic search result list, obtain the rank number of that item in the semantic search result list. For each unique document appearing in both result lists, its structured ranking contribution value is determined based on whether the document appears in the structured query result list, and its semantic ranking contribution value is determined based on whether the document appears in the semantic retrieval result list. The structured ranking contribution value and the semantic ranking contribution value are added together to obtain the final fusion score of the document. The results are then sorted and output in descending order of the final fusion score.

[0017] A second aspect of the present invention provides a device for the input and dual-track retrieval of unstructured industry documents, the device comprising a memory and at least one processor, the memory storing instructions; the at least one processor invokes the instructions in the memory to cause the device to perform the steps of the input and dual-track retrieval method for unstructured industry documents as described in any of the preceding claims.

[0018] A third aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed by a processor, implement the steps of the unstructured industry document inbound and dual-track retrieval method as described in any of the preceding claims.

[0019] The technical solution provided by this invention divides the document to be entered into a database into content units. Structured input data is constructed for each content unit and fed into a large language model for parsing, yielding a structured output including content type tags, field mapping results, schema compatibility scores, and confidence scores. The confidence scores are used for hierarchical processing, and the content units are routed to either a structured processing channel or a semantic processing channel based on the content type tags, forming a dual-track storage structure. For content units routed to the structured channel, the field mapping results and the schema of the target data table are verified for compatibility before being written into the corresponding data table. For content units routed to the semantic processing channel, they are segmented according to semantic boundaries, and a semantic vector is generated for each semantic segment and stored in a vector database. A soft association is established between the semantic segments and structured entities. Based on the dual-track storage structure and soft association relationships, structured queries, semantic retrieval, or hybrid queries are performed on the knowledge base. This invention ensures that each segment is a complete knowledge unit, preventing the return of fragmented content during retrieval, improving semantic integrity, and enabling data cross-track communication between the two tracks, achieving true cross-track retrieval while effectively controlling costs while maintaining accuracy. Attached Figure Description

[0020] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention.

[0021] Figure 1 This is a schematic diagram of the first embodiment of the method for processing unstructured industry documents into a database and performing dual-track retrieval, provided by an embodiment of the present invention. Figure 2 A schematic diagram of the second embodiment of the method for processing unstructured industry documents into a database and performing dual-track retrieval, provided by an embodiment of the present invention; Figure 3 A schematic diagram of the third embodiment of the method for processing unstructured industry documents into a database and performing dual-track retrieval, provided by an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an unstructured industry document import and dual-track retrieval device provided in an embodiment of the present invention. Detailed Implementation

[0022] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” or “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0023] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 The first embodiment of the method for processing and retrieving unstructured industry documents in a database using a dual-track system is illustrated in this invention. The method specifically includes the following steps: Step 101: Divide the documents to be entered into the database into content units, construct structured input data for each content unit, and send it into the large language model for parsing to obtain structured output including content type labels, field mapping results, schema compatibility scores and confidence scores; In this embodiment, unstructured industry documents to be stored are obtained, and corresponding parsing methods are adopted according to the document format type. For documents in PDF, Word, HTML and other formats, they are parsed according to page, paragraph or chapter structure, and the document is parsed into multiple text paragraphs or text blocks. Each text paragraph or text block is treated as an independent content unit, and the position information of each content unit in the original document is recorded, including page number, chapter title and paragraph number. For each content unit, obtain the schema information of the target data table corresponding to the content unit. The schema information includes the table name of the target data table, the list of field names, the data type of each field, whether each field is a required field, and the range of enumerated values ​​for each field. The original text of the content unit and the obtained target data table schema information are organized according to a preset input template to construct structured input data. The structured input data format is a JSON or XML structure containing original text fields and schema description fields, so that the large language model can simultaneously know the content text to be processed and the target data structure.

[0024] In this embodiment, the constructed structured input data is sent to the large language model, and a parsing instruction is sent to the large language model. The parsing instruction requires the large language model to read and understand the original text, extract the corresponding field values ​​from the original text according to the field definition of the target data table schema, and determine whether the content unit belongs to structured data or narrative knowledge. Receive the structured output returned by the large language model, which includes the following fields: The content type tag is used to mark whether the content unit belongs to structured data or narrative knowledge, with values ​​of structured or semantic; The field mapping results are used to record the names of each field extracted by the large language model from the original text and their corresponding field values. Each field value is also accompanied by an extraction confidence indicator, which indicates the degree of confidence of the large language model in the extraction result of that field. Schema compatibility score, used to record the degree of matching between the field mapping results and the field types of the target data table schema; The confidence score is used to comprehensively evaluate the overall confidence of the content unit's inclusion in the database. The confidence score is calculated by weighting the following four dimensions: the first dimension is the coverage of required fields, which is the number of successfully extracted required fields divided by the total number of required fields defined in the target table schema; the second dimension is the field type matching degree, which is the number of type-compatible fields divided by the total number of extracted fields; the third dimension is the enumeration value validity score, which is the number of fields whose values ​​fall within the valid enumeration set divided by the total number of enumeration fields. When there are no enumeration fields, this dimension is 1; and the fourth dimension is the entity alias matching degree, which is the number of fields that successfully match existing entities through alias normalization divided by the total number of associatable fields. The confidence score is obtained by multiplying the values ​​of the above four dimensions by their corresponding preset weight coefficients and summing them. The sum of all weight coefficients is 1.

[0025] Step 102: Perform hierarchical processing based on the confidence score, and route the content units to the structured processing channel or the semantic processing channel according to the content type tag to form a dual-track storage structure; In this embodiment, a confidence score is obtained and compared with a preset first threshold and a second threshold, where the first threshold is greater than the second threshold. When the confidence score is greater than or equal to the first threshold, it is determined to be of high confidence level, meaning all fields of the content unit completely match the target data table schema. When the confidence score is greater than or equal to the second threshold but less than the first threshold, it is determined to be of medium confidence level, meaning some fields of the content unit match the target data table schema, but there are missing fields or incomplete type compatibility. When the confidence score is less than the second threshold, it is determined to be of low confidence level, meaning the content unit is incompatible with the target data table schema.

[0026] In this embodiment, content units determined to be of high confidence level are not marked and directly enter the subsequent routing process; content units determined to be of medium confidence level are marked with a "to be supplemented" tag and then enter the subsequent routing process. This tag is used to prompt human reviewers to verify and supplement the inbound records; content units determined to be of low confidence level are not entered into the subsequent routing process, but are placed in the human review queue, waiting for human reviewers to confirm or correct the content before resubmitting.

[0027] In this embodiment, content type tags are obtained. For content units that are allowed to be stored after hierarchical processing, the content units are routed to different processing channels according to the content type tags. When the content type tag is "structured", the content unit is routed to the structured processing channel, which is used to process structured data that can be extracted into field-value pairs. When the content type tag is "semantic", the content unit is routed to the semantic processing channel, which is used to process narrative knowledge that requires semantic understanding.

[0028] In this embodiment, through routing processing, structured data content and semantic content are separated into different processing flows during the database entry stage, forming two parallel storage tracks. The structured processing channel is responsible for writing data into relational database tables, while the semantic processing channel is responsible for vectorizing data and writing it into a vector database. The two tracks are independent of each other at the physical storage level.

[0029] Step 103: For the content units routed to the structured channel, perform compatibility verification between the field mapping results and the schema of the target data table, and then write them into the corresponding data table; In this embodiment, the field mapping result is obtained, which includes the extracted field names and their corresponding field values. The type compatibility of each field and its field value is compared with the definition of the corresponding field in the target data table. For each field, the data type definition of the field in the target data table is obtained, and it is determined whether the actual data type of the field value in the field mapping result is the same as or can be automatically converted to the data type defined for the field in the target data table. For a field marked as an enumeration type in the target data table schema, obtain the list of valid enumeration values ​​defined for that field in the target data table schema, and determine whether the value of that field exists in the list of valid enumeration values ​​in the field mapping result; if the value of that field exists in the list of valid enumeration values, it is determined to be valid; if the value of that field does not exist in the list of valid enumeration values, it is determined to be invalid.

[0030] Summarize the inspection results of all fields and calculate the compatibility score. When all fields pass the type compatibility check and the enumeration value validity check, the compatibility score is full. When a field fails the type compatibility check or the enumeration value validity check, record the name of each unqualified field, the reason for the failure, and the modification suggestion. Return the unqualified item along with the modification suggestion and wait for correction before resubmitting it into the database. When all checks pass, match the field values ​​in the field mapping result to the field names in the target data table, construct a database write statement, and execute the UPSERT operation to write it into the corresponding relational data table.

[0031] Step 104: For the content units routed to the semantic processing channel, divide them into blocks according to semantic boundaries, generate semantic vectors for each semantic block and store them in the vector database, and establish soft associations between the semantic blocks and structured entities. Step 105: Based on the dual-track storage structure and soft association relationships, perform structured queries, semantic retrieval, or hybrid queries on the knowledge base.

[0032] In this embodiment, a natural language query statement input by a user is received, and query rewriting preprocessing is performed on the natural language query statement; the standard language space identifier configured in the knowledge base is obtained, and the natural language query statement input by the user is translated into the standard language space to obtain a standard language version of the query statement; the translated query statement is subjected to synonym normalization processing, converting colloquial expressions into standard terms defined in the knowledge base. Specifically, a synonym mapping table maintained by the knowledge base is obtained, which records the correspondence between colloquial expressions and standard terms, and the colloquial expressions appearing in the query statement are replaced with the corresponding standard terms; a terminology locking operation is performed, and a domain-specific terminology table maintained by the knowledge base is obtained. The domain-specific terminology table records the domain proper nouns that need to be protected and their correct translations or expressions. Words belonging to the domain-specific terminology table are identified in the query statement, and erroneous translations that may be generated by the general translation model are locked to the correct expressions defined in the terminology table to prevent the general translation model from incorrectly converting domain terms; the rewritten query statement is stored in a cache, and the cache validity period is set to a preset number of days. When the same query is received subsequently, the rewritten result is directly returned from the cache.

[0033] In this embodiment, the preprocessed query statement is input into the three-layer progressive routing system to determine the query intent and query mode. Each layer executes in sequence, and the process ends when a match is found. The first layer is regular expression keyword routing: obtain the pre-configured keyword rule table, which contains multiple rule entries. Each rule contains a keyword regular expression and its corresponding query action. According to the rule priority from high to low, the query statement is matched with the regular expression in the keyword rule table in turn. When the match is successful, the corresponding query action is executed directly, and the routing process ends. When the first layer fails to match, the process proceeds to the second layer: centroid vector matching. This involves obtaining the centroid vectors of each intent category maintained by the system. Each centroid vector is the average of all historically labeled query vectors under that intent category. The query statement input vector is embedded into the model to convert it into a query vector. The cosine similarity between the query vector and the centroid vector of each intent category is calculated. The intent category with the highest cosine similarity is taken as the matching result. When the highest cosine similarity is lower than a preset threshold, the process is considered a miss and proceeds to the third layer. When the second layer fails to find a match, the system proceeds to the third layer: intelligent routing using the large language model. The query statement and a few sample example set are used to construct prompt words for input into the large language model. The few sample example set contains multiple example pairs labeled with query-intent categories. The large language model classifies the query statement according to the prompt words and outputs the intent classification result.

[0034] In this embodiment, each column in the database is accompanied by a three-part comment: Taking the "Origin Name" column in the "Origin Table" as an example, its three-part comment is: "Origin Name. Text format. Exact match query. Note: Case sensitive (e.g., Ethiopia, not ethiopia). Enumerated values: [Brazil,Colombia,Ethiopia,...,Vietnam]".

[0035] Each column comment contains three pieces of information: 1. Semantic Description: Field Meaning 2. Query Mode: Which of the following matching modes (exact, fuzzy, or range) does this field support? 3. Valid value range: an enumerated list of values ​​or a range of values. When Catlaxy-AIAgent retrieves table metadata through the ` / tables` interface, these annotations serve as context injection for the LLM, guiding it to generate valid and efficient queries.

[0036] Technical benefits: LLM no longer performs fuzzy searches on numeric fields and no longer uses non-existent values ​​for enumerated fields; the validity rate of generated queries is significantly improved; adding new fields only requires adding comments, without modifying the routing logic.

[0037] In this embodiment, when the routing result corresponds to the structured query mode, the structured query is executed, including: extracting the conditional parameters involved in the query statement into structured query parameters according to the intent category determined by the routing, including the field name to be queried, the matching method and the matching value. The matching method includes exact matching, fuzzy matching, range matching or multi-value matching. Based on the extracted parameters, a parameterized SQL query statement is constructed to prevent SQL injection attacks. The query statement is executed on the relational database table to obtain structured data results. When the number of returned results is zero, the system automatically switches to semantic retrieval mode. When the routing result returns zero results for the semantic retrieval mode or the structured query, triggering automatic degradation, semantic retrieval is performed. The query statement input vector is embedded into the model and converted into a query vector. Cosine similarity is calculated in the vector database. All semantic blocks in the vector database with a cosine similarity greater than or equal to a preset similarity threshold are obtained as preliminary retrieval results. The top N results are sorted from high to low according to similarity scores and input into the semantic re-ranking model. The semantic re-ranking model calculates a relevance score for each input result and sorts the top M results from high to low according to relevance scores. The obtained results are deduplicated and diversity is controlled so that the number of results from the same source does not exceed a preset number. When the routing result corresponds to a hybrid query mode, the hybrid query is executed, and the structured query process and semantic retrieval process are started in parallel. The structured query result list and the semantic retrieval result list are obtained respectively. For each result item in the structured query result list, its ranking number in the structured query result list is obtained. For each result item in the semantic retrieval result list, its ranking number in the semantic retrieval result list is obtained. For each unique document appearing in both result lists, its structured ranking number is determined based on whether the document appears in the structured query result list, and its semantic ranking number is determined based on whether the document appears in the semantic retrieval result list. The structured ranking contribution value and the semantic ranking contribution value are calculated respectively. The structured ranking contribution value and the semantic ranking contribution value are added together to obtain the final fusion score of the document. The results are sorted in descending order of the final fusion score and then output.

[0038] In this embodiment, the technical solution is as follows: This invention employs a pure configuration-driven multi-tenant architecture, requiring only four configuration files to be written for new customers / industries to go live: `project.config.json` Project metadata Project name, industry, language, data source path `schema.config.json` Table structure definition Field name, field type, embedding dimension, constraints `pipeline.config.json` Warehouse entry process Blocking strategy, deduplication rules, filtering logic `query.config.json` Query rules API endpoints, searchable fields, routing rules Multi-tenant isolation: Single-instance database, with independent schemas for each client (e.g., `coffee_demo`, `qwell_prod`, `client_x`); Single-instance worker service, with routing `project_id → schema_name` via distributed key-value store; Read-write separated authentication tokens.

[0039] Technical benefits: The launch time for new industry projects has been shortened from "weeks of custom development" to "hours of writing configurations"; the framework code requires zero modification, reducing maintenance and upgrade risks; verified: the coffee industry (coffee_demo) and the healthy drinking water industry (qwell) use the same framework.

[0040] In this embodiment, a message queue-driven asynchronous inbound pipeline is used to decouple the inbound task from user operations; the processing flow of the asynchronous inbound pipeline is as follows: Step 1: Schema Check Before Data Entry. After a user submits a document (single or batch), the system first performs a schema check before data entry, including: (1) checking the type compatibility between the fields to be entered and the target table fields; (2) checking whether the values ​​of the enumerated fields are within the allowed range; (3) outputting a compatibility score, and returning unqualified items with modification suggestions. Step 2: Dispatch according to complexity. Dispatch tasks according to complexity and batch size: (1) Simple inbound tasks (few entries) are processed synchronously and the processing results are returned immediately; (2) Medium batch inbound tasks are pushed into the message queue and processed asynchronously; (3) High uncertainty content is marked as pending manual review. Step 3: Asynchronous consumption of the message queue. The message queue consumer processes the inbound tasks in batches. The number of entries processed in each batch can be adjusted through the configuration file. Each entry can be retried a maximum of a preset number of times (configurable) if processing fails. Entries that still fail after retrying are transferred to the Dead Letter Queue. The contents of the Dead Letter Queue have the following characteristics: (1) They do not block the execution of the normal inbound process; (2) They retain complete error context information and the original request payload; (3) The system provides an interface for viewing error details of failed tasks and re-entering them into the database. Inbound task tracking: Each inbound operation generates a unique task ID and records the status (queuing / processing / completed / partially failed / failed); the front end can poll the task status to get real-time progress; after the task is completed, the following are recorded: number of successfully inbound records, number of skipped records, number of failed records, and error details.

[0041] Technical benefits: Batch data entry does not block user operations; users can continue other work after submission; a single failed entry does not affect the entry of other content in the same batch; dead-letter queue preserves failed data, and zero data loss task tracking provides complete audit trails for data entry.

[0042] Please see Figure 2 The second embodiment of the method for processing and retrieving unstructured industry documents in the database using a dual-track system, provided by this invention, is illustrated in the diagram. The method includes: Step 201: Obtain the content units routed to the semantic processing channel, segment them according to the title level, and perform topic conversion segmentation when the semantic similarity between adjacent paragraphs is lower than the third threshold; In this embodiment, content units routed to the semantic processing channel are obtained, and the text content of the content unit is segmented according to semantic boundaries. The text structure of the content unit is parsed, and the title level markers in the text are identified, including first-level titles and second-level titles. Segmentation is forcibly performed at the location of a first-level or second-level title, dividing the content under different titles into different semantic blocks. For consecutive paragraphs between title levels, the semantic similarity between adjacent paragraphs is calculated. Each paragraph is converted into a paragraph vector through a vector embedding model, and the cosine similarity between two adjacent paragraph vectors is calculated. When the cosine similarity is lower than a preset similarity threshold, it is determined that a topic change has occurred between the two adjacent paragraphs, and segmentation is performed at that location. For each block obtained after the above segmentation, it is verified whether the block contains a complete knowledge unit. The complete knowledge unit refers to a complete variety description, a complete method introduction, or a complete event narrative. When a block does not contain a complete knowledge unit, the boundary of the block is adjusted, and the missing part is incorporated from the adjacent blocks until each block contains a complete knowledge unit.

[0043] Step 202: Input the text content of each semantic block into the vector embedding model, and obtain the fixed-dimensional semantic vector output by the vector embedding model as the vector representation of the current semantic block; In this embodiment, for each semantic block obtained, the text content of the semantic block is input into a vector embedding model. The vector embedding model is a pre-trained bilingual semantic coding model that can encode the input text into a semantic vector of fixed dimensions. The vector output by the model is used as the semantic vector representation of the semantic block. The above operation is performed on each semantic block to obtain the semantic vector corresponding to each semantic block.

[0044] Step 203: Generate a unique identifier for each semantic block, and associate the block identifier, the original text of the semantic block, the semantic vector corresponding to the semantic block, and the source information of the semantic block into a vector database using the HNSW index structure. In this embodiment, a unique block identifier is generated for each semantic block. The block identifier, the original text of the semantic block, the semantic vector corresponding to the semantic block, and the source information of the semantic block are associated and stored in the vector database. The source information includes the original document identifier, chapter information, and location information. The vector database adopts a vector index structure based on the HNSW algorithm, which supports efficient nearest neighbor search of the stored semantic vectors.

[0045] Step 204: Obtain the unique values ​​of the pre-configured associated fields, summarize them into a candidate entity set and load them into memory, and refresh incrementally as the structured data table is updated; In this embodiment, a candidate entity set is extracted from the structured data table of the relational database. Specifically, the following steps are taken: A pre-configured list of associated fields is obtained. This list records the field mapping relationships that need to be established between the semantic blocks and the structured tables, including the mapping between the variety field and the variety name field of the variety table, and the mapping between the origin field and the origin name field of the origin table. For each configured associated field, all unique values ​​of that field in the corresponding structured data table are read, and all unique values ​​are aggregated into a candidate entity set. The candidate entity set is loaded into the system memory when the inbound process starts. When the structured data table is updated, an incremental refresh is performed on the candidate entity set, adding the newly added unique values ​​to the candidate entity set.

[0046] Step 205: Perform transliteration normalization, case normalization, and abbreviation expansion processing on each entity in the candidate entity set to map multiple spellings of the same entity to the same standard identifier. In this embodiment, alias normalization is performed on each entity name in the candidate entity set, mapping multiple surface forms of the same entity to the same standard identifier. Alias ​​normalization includes the following operations: transliteration variant normalization, mapping the same entity name with different transliterations to a unified variety identifier; case normalization, unifying all letters to lowercase before comparison; abbreviation expansion, expanding the abbreviation into its complete form before matching. The normalized mapping relationship is stored in an alias mapping table maintained by the system. This alias mapping table supports manual addition of new alias mapping entries and also supports automatic learning. When a new spelling variant is discovered through fuzzy matching, the variant is recorded in the confirmation queue and automatically added to the alias mapping table after operational review and confirmation.

[0047] Step 206: Extract text fragments for each semantic block, calculate normalized similarity based on edit distance with candidate entity names, and take candidate entity records that are higher than the fourth threshold as the matching items of the corresponding semantic block in the associated field; In this embodiment, for each semantic block obtained, the text content of the semantic block is acquired, and a text fragment to be matched is extracted from the semantic block text. The text fragment to be matched is a word or phrase in the semantic block that may be related to the entity name. The extracted text fragment to be matched is compared with each entity name in the candidate entity set to calculate character-level similarity. The character-level similarity is based on the edit distance algorithm. The edit distance is defined as the minimum number of single-character editing operations required to convert one string into another string. A single-character editing operation includes inserting a character, deleting a character, and replacing a character. The similarity value is equal to 1 minus the edit distance divided by the maximum value of the lengths of the two strings. The similarity value ranges from 0 to 1. The larger the value, the more similar the two strings are. The calculated similarity value is compared with a preset matching threshold. When the similarity value is greater than or equal to the matching threshold, the candidate entity is taken as the matching item of the semantic block in the associated field, and the matching similarity score of each matching item is recorded.

[0048] Step 207: When a semantic block matches candidate entities in multiple associated fields, perform cross-validation; when a semantic block matches multiple candidate entities in the same associated field, resolve conflicts. In this embodiment, when a semantic block matches candidate entities across multiple associated fields, cross-validation is performed. Specifically, the matching results of the semantic block across each associated field are obtained, and it is determined whether there are associated records for the candidate entities matched by the same semantic block across different associated fields in the structured data table. If associated records exist, the confidence of the matching result is increased. When a semantic block matches multiple candidate entities across the same associated field, conflict resolution is performed, and the following priority rules are used to determine which matching result to retain: The first priority is exact matching over fuzzy matching, i.e., the matching result is retained if the text fragment to be matched is exactly the same as the candidate entity name; the second priority is the highest similarity score, i.e., when no exact match exists, the matching result with the highest similarity score is retained; the third priority is support for co-occurrence across multiple fields, i.e., when the similarity scores are the same, the matching result supported by the multi-field cross-validation is retained. After conflict resolution, the entity ranked first is retained as the primary reference for the associated field, and the remaining matching results are stored as alternative references.

[0049] Step 208: Store the resolved soft association references in the semantic block record without modifying the schema of the structured table; In this embodiment, the established soft association references are stored in the records of semantic blocks without modifying the schema structure of the structured data table. Each soft association reference field contains the reference target and the matching confidence score.

[0050] Step 209: When new entity data is entered into the structured data table, scan all existing semantic blocks that have not yet established new entity associations, and match them to create soft association references for the semantic blocks.

[0051] In this embodiment, when new entity data is entered into the structured data table, the system automatically triggers an incremental backfill process; it obtains the entity name and all aliases of the newly entered entity, scans all existing semantic blocks that have not yet established an association with the entity, and performs fuzzy matching between the name and aliases of the newly entered entity and the text of the semantic block for each unassociated semantic block; when a match is successful, it automatically establishes a soft association reference with the new entity for the semantic block. The backfill operation is executed asynchronously and does not block the normal entry process.

[0052] Please see Figure 3 The third embodiment of the method for processing and retrieving unstructured industry documents in the database using a dual-track system, provided by this invention, is illustrated in the diagram. This method includes: Step 301: Periodically obtain production environment query logs and filter out zero-result queries where both structured retrieval and semantic retrieval return empty results, as well as low-similarity queries where the highest similarity in semantic retrieval is lower than a preset similarity threshold. In this embodiment, production environment query logs are periodically acquired, and each log entry is analyzed sequentially. For each query record, the system first checks whether the number of results returned in the structured retrieval channel is zero, and simultaneously checks whether the highest similarity score returned in the semantic retrieval channel is lower than a preset similarity threshold. When a query simultaneously meets both conditions—zero structured retrieval results and a highest semantic similarity score lower than the preset threshold—it is marked as content not effectively covered by the knowledge base and included in the candidate analysis set. When a query only meets the condition of a highest semantic similarity score lower than the preset threshold but returns results in the structured retrieval channel, it indicates that relevant structured data exists in the knowledge base but lacks corresponding narrative knowledge descriptions; this query is also included in the candidate analysis set. After the above filtering process, a set of queries requiring knowledge gap analysis is obtained. Each query in this set represents a content blind spot where the user has a need but the knowledge base fails to provide a satisfactory answer.

[0053] Step 302: Convert the selected query statement input vectors into query vectors by embedding the model, and use a density-based clustering algorithm to automatically cluster the query vectors using the cosine distance between them as a metric. In this embodiment, for all the queries to be analyzed obtained through screening, the query statement of each query is input into a vector embedding model for processing. The vector embedding model encodes the text-based query statement into a fixed-dimensional numerical vector, obtaining a query vector representation corresponding to each query. All query vectors constitute a vector set to be clustered. Then, a density-based clustering algorithm is used to automatically cluster this vector set. During the clustering process, the cosine distance between any two query vectors is used as a distance metric to measure the semantic similarity between the two queries. The smaller the cosine distance, the closer the semantics of the two queries are. The density-based clustering algorithm determines the clusters by identifying densely distributed regions of data points in the vector space. The algorithm automatically traverses all query vectors and, based on preset neighborhood radius parameters and minimum number of points parameters, groups data points that are density-connected into the same cluster. At the same time, isolated data points in regions with low density are identified as noise points and excluded. The number of clusters is automatically determined by the algorithm based on the distribution characteristics of the data itself and does not need to be preset. After clustering, each cluster represents a knowledge gap in a specific topic area. All queries within the same cluster are proposed around that topic area but fail to be effectively satisfied by the knowledge base.

[0054] Step 303: Select the query closest to the cluster center in each cluster as a representative example, sort the clusters by query frequency from high to low, and generate a knowledge gap report including gap topics, representative query examples, query frequency, and suggested supplementary directions.

[0055] In this embodiment, for each cluster obtained through clustering, the average vector of all query vectors in the cluster is calculated as the cluster center vector. Then, the cosine distance between the query vector of each query within the cluster and the cluster center vector is calculated, and the query closest to the cluster center is selected as the representative query example of the cluster. Simultaneously, for each cluster, the total number of query records contained in the cluster is counted as the query frequency of the knowledge gap topic. A higher query frequency indicates a greater user demand for that topic, and a higher urgency for the knowledge base to supplement content in that direction. Semantic summarization and topic induction are performed on the query text in each cluster, and the core topic words or expressions commonly involved in all queries of that cluster are extracted as the topic name of the gap. Finally, all clusters are sorted in descending order of query frequency, and the sorted cluster information is compiled into a knowledge gap report. Each record in the knowledge gap report contains four fields: gap topic, representative query example, query frequency, and suggested supplementation direction.

[0056] In this embodiment, a benchmark test set is maintained for each knowledge base project. The benchmark test set contains multiple benchmark test records. Each benchmark test record includes query text, expected query pattern, expected set of key results, and a relevance score baseline. After each new content entry operation is completed, the benchmark test set regression verification is automatically triggered. Each query text in the benchmark test set is submitted to the knowledge base for retrieval in sequence, and the retrieval results for each query before and after entry are recorded. The retrieval results after entry are compared with the retrieval results before entry. When the comparison results meet any of the following conditions, the item is marked as degraded: any item in the expected set of key results disappears from the return list after entry; the ranking of the expected key result in the result list after entry drops by more than a preset value; the highest relevance score of the returned result is lower than a preset proportion of the baseline score before entry. An alarm report is automatically generated for the items marked as degraded. The alarm report includes query text, comparison of results before and after entry, and impact rating, and an alarm notification is sent to the operation and maintenance personnel.

[0057] Figure 4 This is a schematic diagram of a device for processing and retrieving unstructured industry documents via a dual-track system, as provided in an embodiment of the present invention. This device 400 can vary significantly due to differences in configuration or performance. It may include one or more central processing units (CPUs) 410 (e.g., one or more processors) and a memory 420, and one or more storage media 430 (e.g., one or more mass storage devices) for storing application programs 433 or data 432. The memory 420 and storage media 430 can be temporary or persistent storage. The program stored in the storage media 430 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the device 400 for processing and retrieving unstructured industry documents via a dual-track system. Furthermore, the processor 410 can be configured to communicate with the storage medium 430 and execute a series of instruction operations in the storage medium 430 on the unstructured industry document inbound and dual-track retrieval device 400 to implement the method provided in the above embodiment.

[0058] The unstructured industry document import, sorting, and dual-track retrieval device 400 may also include one or more power supplies 440, one or more wired or wireless network interfaces 450, one or more input / output interfaces 460, and / or one or more operating systems 431, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 4The structure of the unstructured industry document entry and dual-track retrieval device shown does not constitute a limitation on the computer device provided by the present invention. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0059] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the various steps of the unstructured industry document entry and dual-track retrieval method provided in the above embodiments.

[0060] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described equipment or apparatus / unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0061] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0062] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for the input, tiered processing, and dual-track retrieval of unstructured industry documents, characterized in that: The method includes the following steps: The documents to be imported into the database are divided into content units. Structured input data is constructed for each content unit and fed into a large language model for parsing to obtain structured output including content type labels, field mapping results, schema compatibility scores and confidence scores. The content is graded according to the confidence score and routed to the structured processing channel or the semantic processing channel according to the content type tag, forming a dual-track storage structure. For content units routed to the structured channel, the field mapping results and the schema of the target data table are verified for compatibility before being written to the corresponding data table; For the content units routed to the semantic processing channel, they are divided into blocks according to semantic boundaries. Semantic vectors are generated for each semantic block and stored in the vector database. Soft associations are established between the semantic blocks and structured entities. Based on a dual-track storage structure and soft association relationships, the knowledge base can be used for structured queries, semantic retrieval, or hybrid queries.

2. The method for processing and dual-track retrieval of unstructured industry documents upon entry into a database as described in claim 1, characterized in that, The process involves segmenting the documents to be imported into the database into content units, constructing structured input data for each content unit, and then feeding it into a large language model for parsing. This yields a structured output including content type tags, field mapping results, schema compatibility scores, and confidence scores, comprising: The document to be imported is parsed into multiple independent text paragraphs or text blocks according to the page, paragraph or chapter structure, and used as content units, and its position information in the original document is recorded. For each content unit, the original text and the corresponding target data table schema information are organized into structured input data including text and field definitions according to a preset template; The structured input data is fed into the large language model, and field values ​​are extracted from the original text according to the schema definition to determine the content type; It receives the structured output returned by the large language model and obtains the content type label, field mapping results, schema compatibility score, and confidence score calculated by multi-dimensional weighting for the content unit.

3. The method for unstructured industry documents to be imported and processed in a separate database and retrieved using a dual-track system as described in claim 1, characterized in that... The grading process based on the confidence score includes: The confidence score is compared with a preset first threshold and a second threshold, wherein the first threshold is greater than the second threshold; When the confidence score is greater than or equal to the first threshold, it is determined to be of high confidence level. Then all fields of the content unit are completely matched with the target data table schema, and no tags are added to the content unit. When the confidence score is greater than or equal to the second threshold and less than the first threshold, it is judged as medium confidence level. If some fields of the content unit match the target data table schema, there are missing fields or incomplete type compatibility, and a supplementary mark is added to the content unit. When the confidence score is less than the second threshold, it is judged as a low confidence level. In this case, the content unit is incompatible with the target data table schema and will not enter the routing process.

4. The method for processing unstructured industry documents through database entry and dual-track retrieval as described in claim 1, characterized in that, The step of performing compatibility verification between the field mapping results and the schema of the target data table for the content units routed to the structured channel, and then writing them into the corresponding data table, includes: Compare the actual data types of each field in the field mapping result with the data types defined in the corresponding fields of the target data table for type compatibility. For fields marked as enumeration type in the target data table schema, check whether the values ​​in the field mapping results exist in the predefined list of valid enumeration values ​​for that field. Summarize the inspection results of all fields. If there are fields with incompatible types or invalid enumeration values, return the result with modification suggestions. After all fields pass the inspection, construct a database write statement and execute the UPSERT operation to write the relational data table.

5. The method for processing unstructured industry documents through database entry and dual-track retrieval as described in claim 1, characterized in that, The content units routed to the semantic processing channel are segmented according to semantic boundaries, a semantic vector is generated for each semantic segment and stored in a vector database, and a soft association is established between the semantic segments and structured entities, including: Obtain the content units routed to the semantic processing channel, segment them by title level, and perform topic conversion segmentation when the semantic similarity between adjacent paragraphs is lower than the third threshold; The text content of each semantic block is input into the vector embedding model, and the fixed-dimensional semantic vector output by the vector embedding model is used as the vector representation of the current semantic block. A unique identifier is generated for each semantic block, and the block identifier, the original text of the semantic block, the semantic vector corresponding to the semantic block, and the source information of the semantic block are associated and stored in a vector database using the HNSW index structure. Get the unique values ​​of the pre-configured associated fields, summarize them into a candidate entity set and load them into memory, and refresh incrementally as the structured data table is updated; For each entity in the candidate entity set, perform transliteration normalization, case normalization, and abbreviation expansion processing to map multiple spellings of the same entity to the same standard identifier; For each semantic block, extract text fragments and calculate normalized similarity based on edit distance with candidate entity names. Records of candidate entities that are higher than the fourth threshold are used as the matching items of the corresponding semantic block in the associated field. Cross-validation is performed when a semantic block matches candidate entities in multiple associated fields, and conflict resolution is performed when a semantic block matches multiple candidate entities in the same associated field. The resolved soft association references are stored in semantic block records without modifying the schema of the structured table; When new entity data is entered into the structured data table, all existing semantic blocks that have not yet established new entity associations are scanned and matched to create soft association references for the semantic blocks.

6. The method for unstructured industry documents to be imported and processed in a separate database and retrieved using a dual-track system as described in claim 5, characterized in that... The process of obtaining content units routed to the semantic processing channel, segmenting them by title level, and performing topic conversion segmentation when the semantic similarity between adjacent paragraphs is below a third threshold includes: The text structure of content units is analyzed, the heading level markers in the text are identified, and the content under different headings is divided into different semantic blocks; For consecutive paragraphs at heading levels, the semantic similarity between adjacent paragraphs is calculated. When the semantic similarity is lower than a preset similarity threshold, it is determined that a topic shift has occurred between adjacent paragraphs, and the paragraphs are split at that position. For each semantic block obtained after segmentation, verify whether the semantic block includes a complete knowledge unit. When a semantic block does not include a complete knowledge unit, adjust the boundary of the semantic block and incorporate the missing part from the adjacent semantic block until each semantic block includes a complete knowledge unit.

7. The method for unstructured industry documents to be imported and processed in a separate track and retrieved in a dual-track manner as described in claim 5, characterized in that, The cross-validation process when a semantic block matches candidate entities across multiple related fields includes: Obtain the matching results of semantic blocks on each associated field, determine whether the candidate entities matched by the same semantic block on different associated fields have associated records in the structured data table, and increase the confidence of the matching results when associated records exist.

8. The method for processing unstructured industry documents through database entry and dual-track retrieval as described in claim 1, characterized in that, The structured query includes: Based on the intent category determined by the route, the conditional parameters involved in the query statement are extracted into structured query parameters; A parameterized SQL query statement is constructed based on the extracted parameters. The query statement is executed on the target data table to obtain structured data results. When the number of returned results is zero, the system automatically switches to semantic search mode.

9. The method for processing unstructured industry documents through database entry and dual-track retrieval as described in claim 1, characterized in that, The semantic retrieval includes: The query input vector is embedded into a model and converted into a query vector. Cosine similarity is then calculated in a vector database. Obtain all semantic blocks in the vector database whose cosine similarity to the query vector is greater than or equal to a preset similarity threshold as preliminary search results; The first preset number of search results, sorted by similarity score from highest to lowest, are input into the semantic re-ranking model. The semantic re-ranking model calculates a relevance score for each input result, sorts the results by relevance score from highest to lowest, and then selects the second preset number of input results. The selected input results are then deduplicated.

10. The method for processing unstructured industry documents through database entry and dual-track retrieval as described in claim 1, characterized in that, After performing structured queries, semantic retrieval, or hybrid queries on the knowledge base based on the dual-track storage structure and soft association relationships, the process also includes: Periodically retrieve query logs from the production environment, and filter out zero-result queries where both structured and semantic searches return empty results, as well as low-similarity queries where the highest similarity in semantic search is lower than a preset similarity threshold; The selected query input vectors are embedded into the model and converted into query vectors. A density-based clustering algorithm is used to automatically cluster the query vectors using the cosine distance between them as a metric. The query closest to the cluster center in each cluster is selected as a representative example. The clusters are sorted from high to low according to the query frequency, and a knowledge gap report is generated, which includes the gap topic, representative query examples, query frequency, and suggested directions for supplementation.

11. The method for processing unstructured industry documents through database entry and dual-track retrieval as described in claim 1, characterized in that, Before performing structured queries, semantic retrieval, or hybrid queries on the knowledge base based on the dual-track storage structure and soft association relationships, a query preprocessing step is also included: Obtain the standard language space identifier of the knowledge base configuration, translate the query statement to the standard language space, and obtain the standard language version of the query statement; The query statements in the standard language version are processed for synonym normalization, term locking is performed, the domain-specific terminology list maintained by the knowledge base is obtained, and words belonging to the domain-specific terminology list are identified in the query statements.

12. The method for unstructured industry documents to be imported and processed in a separate database and retrieved using a dual-track system as described in claim 11, characterized in that... The preprocessed query statement is input into a three-tier progressive routing system to determine the query intent and query pattern, including: The first layer is regular expression keyword routing: it retrieves the pre-configured keyword rule table, and matches the query statement with the regular expressions in the keyword rule table in descending order of rule priority. When a match is successful, the corresponding query action is executed directly. When the first layer fails to match, the second layer centroid vector matching is entered: the centroid vectors of each intent category maintained by the system are obtained, the query statement is converted into a query vector, the cosine similarity with each centroid vector is calculated, and the intent category with the highest similarity is taken as the matching result. When the second layer fails to find a match, the system proceeds to the third layer: intelligent routing using the large language model. This involves constructing prompt words from the query statement and a small sample set as input to the large language model, which then outputs the intent classification result.

13. The method for unstructured industry documents to be imported and processed in a separate database and retrieved using a dual-track system as described in claim 1, characterized in that... The hybrid query steps: When the routing result corresponds to a mixed query mode, the structured query process and the semantic retrieval process are started in parallel to obtain the list of structured query results and the list of semantic retrieval results, respectively. For each result item in the structured query result list, obtain the rank number of that item in the structured query result list; for each result item in the semantic search result list, obtain the rank number of that item in the semantic search result list. For each unique document appearing in both result lists, its structured ranking contribution value is determined based on whether the document appears in the structured query result list, and its semantic ranking contribution value is determined based on whether the document appears in the semantic retrieval result list. The structured ranking contribution value and the semantic ranking contribution value are added together to obtain the final fusion score of the document. The results are then sorted and output in descending order of the final fusion score.

14. A device for the input, sorting, and dual-track retrieval of unstructured industry documents, characterized in that, The device for processing and retrieving unstructured industry documents by import and retrieval includes a memory and at least one processor. The memory stores instructions. The at least one processor invokes the instructions in the memory to cause the device for processing and retrieving unstructured industry documents by import and retrieval to perform each step of the method for processing and retrieving unstructured industry documents by import and retrieval as described in any one of claims 1-13.

15. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement each step of the unstructured industry document import and dual-track retrieval method as described in any one of claims 1-13.

Images