File analysis and knowledge recall method and device for large model question-answering system, equipment and storage medium

Abstract

The invention discloses a file analysis and knowledge recall method and device for a large model question-answering system, equipment and a storage medium. The method comprises the following steps: analyzing a document, and extracting fine-grained information; receiving a natural language query request, and generating a question text containing similar questions; performing keyword extraction and weight distribution on the question text to generate keyword weight mapping; calculating similarity scores of different documents and question texts by using keyword weights based on a BM25 algorithm, and preliminarily screening documents associated with similar questions; in the primarily screened document, accurately querying a specific fragment; and smoothing the specific segments, and returning the final result after sorting the final result according to the comprehensive scores of the segments. According to the method and the device, the operation cost of the question-answering system is reduced, the analysis capability of fine-grained information is improved, the user can accurately position required specific information during query, and the retrieval efficiency and accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the technical fields of natural language processing and information retrieval, and particularly relates to a file parsing and knowledge recall method, device, equipment, and storage medium for a large model question answering system. Background Art

[0002] With the rapid development of large language model technology, related application products based on this technology have received extremely high attention in the market. Among them, the knowledge question answering system, as an important application scenario, effectively makes up for the inherent defects of large models, such as being unable to obtain instant messages and being prone to hallucinations, and significantly improves the reliability and practicality of intelligent question answering.

[0003] In the knowledge question answering system, documents are usually retrieved based on the BM25 algorithm. For example, in the Chinese patent with the application number CN202210151143.7, it uses the BM25 algorithm to utilize pre-trained first and second natural language processing models to obtain the similarity scores between each candidate document and the question text, and obtains the retrieved documents according to the similarity scores, and pushes the retrieved documents as the document retrieval result to the user. However, the above document retrieval method has the following problems: it is necessary to pre-train two language processing models, increasing the operating cost of the question answering system; it is difficult to accurately identify and process variants of domain terms (such as "COVID-19" and "novel coronavirus"), resulting in poor scoring effects for words that have not been trained by the encoder; the computational complexity is large, increasing the latency of each user query and affecting the user experience; it only supports document content retrieval and lacks the ability to analyze fine-grained information, making it impossible for users to accurately locate the specific information they need during queries and reducing the retrieval efficiency; when there is a long query, the scoring of individual keywords is too high, interfering with the overall query result and reducing the retrieval accuracy. Summary of the Invention

[0004] Aiming at the defects existing in the prior art, the present application provides a file parsing and knowledge recall method, device, equipment, and storage medium for a large model question answering system to solve the technical problems such as high operating cost of the question answering system in the prior art, weak ability to process variants of domain terms, low retrieval efficiency and accuracy.

[0005] To achieve the above object of the invention, the technical solution provided by the present invention is as follows:

[0006] A file parsing and knowledge recall method for a large model question answering system includes the following steps:

[0007] S1. Parse the input document, extract the fine-grained information of each document, and store it in the database according to the predefined index mapping;

[0008] S2. Receive the natural language query request input by the user and generate a question text containing several relevant similar questions;

[0009] S3. Use multiple word segmentation methods to extract keywords and assign weights to the question text, generating a keyword weight mapping;

[0010] S4. Use the keyword weights based on the BM25 algorithm to calculate the similarity scores between different documents and the question text, and preliminarily screen the documents associated with the similar questions according to the relevance scores;

[0011] S5. Within the range of the initially screened documents, accurately query specific segments;

[0012] S6. Use the continuous interval annealing algorithm to smooth the adjacent matching segments of the specific segments, and sort the final results according to the comprehensive scores of the segments and then return.

[0013] In one implementation, step S1 specifically includes: parsing the basic information of the input document, splitting the document content into semantic blocks and slice windows at a granularity level, assigning labels to each part of the content, storing the parsed document information in the Elasticsearch retrieval engine, performing word segmentation on the text of the input document, setting field mapping, storing the basic document information in the top-level field, storing the text content after document chunking in the nested field, and recording the fine-grained data of each segment in the input document.

[0014] In one implementation, the basic information includes file name, title, abstract, text, and mind map; the labels include header, footer, title, text, table, picture, table of contents, and file name; storing in the Elasticsearch retrieval engine includes setting a global index in Elasticsearch, including the number of shards, backup, replica configuration, and configuration of the custom BM25 similarity algorithm; using ik_smart and ik_max_word to perform word segmentation on the text.

[0015] In one implementation, step S3 specifically includes: S31. Weight initialization stage:

[0016] S311. Use the ik_smart tokenizer to perform coarse-grained word segmentation and establish an initial weight dictionary;

[0017] S312. Execute the cumulative enhancement strategy for the repeatedly occurring terms: the weight increases by α each time it repeats, where α is a weight coefficient less than 1, and the weight of the i-th word segmentation in the ik_smart word segmentation set is calculated through (1):

[0018]

[0019] where n is the term wi The number of occurrences in the ik_smart word segmentation result; w i Indicates the i-th word segmentation in the ik_smart word segmentation set, boost(w i ) indicates w i The weight corresponding to the word segmentation;

[0020] S32. Fine-grained expansion stage:

[0021] S321. Use ik_max_word for fine-grained word segmentation to generate a candidate word set

[0022] S322. Filter stop words and low-value part-of-speech:

[0023] S323. Implement hierarchical weight inheritance: If the fine-grained term is completely included by the coarse-grained term, inherit its weight; for long terms not included, assign a basic weight of 0.3; for short terms, temporarily assign a basic value of 0.3;

[0024] S33. Keyword enhancement stage:

[0025] S331. Call jieba to extract the top N1 keywords and weights ranked from high to low;

[0026] S332. Perform non-linear enhancement on the existing terms according to (3):

[0027] boost new (w) = boost(w) × [1 + 0.2 × jieba_weight(w)](3)

[0028] where w is all the words extracted in S22, boost(w) represents the original weight of the word, boost new (w) represents the weight of the word after enhancement, and jieba_weight represents the jieba weight of the word w;

[0029] S34. Weight correction stage:

[0030] S341. Single-character word penalty mechanism, penalize single characters according to (4):

[0031]

[0032] S342. Synonym diffusion strategy, perform synonym diffusion on the word according to (5):

[0033]

[0034] where s is a synonym of w, and the weight of s is B times the weight of w, and B is a diffusion coefficient less than 1;

[0035] S35. Result Standardization: The result is rounded to two decimal places, and at the same time, low-confidence terms are filtered out.

[0036] In one embodiment, for the terms with a length greater than 1 in step S323, they are long terms, and for the terms with a length equal to 1, they are short terms. The j-th token in the ik_max_word tokenization set is achieved by calculating (2):

[0037]

[0038] where, W smart represents the ik_smart tokenization result set; w k represents a certain word in the ik_smart tokenization result set; w j represents the j-th token in the ik_max_word tokenization set, and w j is w k itself or its subset.

[0039] In one embodiment, step S4 specifically includes:

[0040] S41. Candidate Document Retrieval: Based on the BM25 algorithm, a preliminary retrieval model is constructed. For a query q ∈ Q and any document d ∈ D in the document set D, multi-dimensional retrieval matching is performed through the keyword set K. Let the retrieval parameter boost be the weight enhancement coefficient, and the candidate document score is calculated according to (6):

[0041] Score base (q, d) = BM25(q, d) × boost (6)

[0042] S42. Structured Field Weight Calculation: Establish a feature weight triple (a, b, c) ∈ R 3 corresponding to the multivariate correlation influence factors of the document file name, title field, and body content respectively. The structured feature scoring function is constructed according to (7):

[0043]

[0044] where, represents tf k the word frequency statistic in a specific field of document d;

[0045] S43. Comprehensive Relevance Ranking: The final relevance score is generated through linear weighted fusion, and the ranking function is defined according to (8):

[0046] Score final (q, d) = Score base (q, d) + η × [Score filename + Score title + Score content ​ (8)

[0048] Where Q is the query space, D is the document set, K is the keyword set, R is the real number field, η is the first empirical decay coefficient, and η is less than 1, which is used to balance the contribution of the basic retrieval score and the structured features;

[0049] Sorted in descending order according to Score final (q, d), the top N2 ranked documents are intercepted to form the candidate document set D', which is used as the high-quality candidate for subsequent processing.

[0050] In one implementation, step S5 specifically includes:

[0051] S51. Candidate segment retrieval: Based on the BM25 algorithm, a preliminary retrieval model is constructed. For any document d ∈ D in the query q ∈ Q and the document set D, multi-dimensional retrieval matching is performed through the keyword set K. Let the retrieval parameter boost be the weight enhancement coefficient, and the preliminary matching score is calculated according to (6):

[0052] Score base (q, d) = BM25(q, d) × boost(6);

[0053] S52. The original score is linearly mapped to the preset interval [ω min , ω max through the min-max normalization method, where ω min is 1.0, ω max is 1.2, and the original score linear mapping is calculated successively according to (9) and (10):

[0054]

[0056] Where μ is the second empirical decay coefficient, μ is less than 1, α(d) represents the result of the min-max normalization method, max(S) is the maximum value of the scores calculated in S51, min(S) is the minimum value of the scores calculated in S51, and s(d) represents the score of a certain document in the document set;

[0057] S53. Calculate the similarity between the user query and the document segment according to (11):

[0058] Score vector = λ × cosine(v q , v d )(11)

[0059] Where λ is the weight decay coefficient, v q represents the vector of the user's question, and v d represents the vector of the document segment;

[0060] S5. Aggregate all the calculated result values as the score for each document fragment. Finally, select the top N3 fragments ranked in descending order of scores as the final document fragments. The score of the document fragment is calculated according to (12):

[0061] Score final = MAX(Score struct , Score vector )(12).

[0062] In one implementation, the comprehensive score of the fragments in S6 is calculated according to (13):

[0063] s i ′ = s i × γ i-1 (13)

[0064] where i is any integer from 1 to n, n is the number of retrieved fragments recalled in the document, and γ represents the attenuation coefficient, and γ is less than 1.

[0065] This application also provides a file parsing and knowledge recall device for a large model question - answering system, including:

[0066] Document parsing module: used to parse the input document, extract the fine - grained information of each document, and store it in the database according to the predefined index mapping;

[0067] Receiving module: used to receive the natural language query request input by the user and generate a question text containing several related similar questions;

[0068] Keyword extraction module: used to extract keywords and assign weights to the question text using multiple word - segmentation methods to generate a keyword - weight mapping;

[0069] Initial screening module: used to calculate the similarity score between different documents and the question text based on the keyword weights using the BM25 algorithm, and initially screen the documents associated with the similar questions according to the relevance score;

[0070] Precise query module: used to precisely query specific fragments within the range of the initially screened documents;

[0071] Return retrieval module: used to smooth the adjacent matching fragments of the specific fragments using the continuous interval annealing algorithm, and return the final result sorted by the comprehensive score of the fragments, and input the answer required by the user.

[0072] This application also provides a device, including:

[0073] One or more processors;

[0074] A storage device for storing one or more computer programs;

[0075] When the computer program is executed by the processor, it causes the processor to perform the steps of the above method.

[0076] The present application also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the file parsing and knowledge recall method for a large model question answering system according to any one of claims 1 to 8.

[0077] Compared with the prior art, the present application has at least the following beneficial effects:

[0078] The file parsing and knowledge recall method, device, equipment and storage medium for a large model question answering system in the present application can retrieve documents without pre-training multiple language models, reducing the operation cost of the question answering system. It improves the analysis ability of fine-grained information, enables users to accurately locate the specific information required during query, and improves the retrieval efficiency and accuracy. During the process of keyword extraction and weight assignment for the question text, the single-word penalty mechanism reduces noise interference, enhances semantic understanding and improves the robustness of the question answering system. The synonym diffusion enhances semantic consistency, generalization ability and context understanding, and can optimize weight assignment, improve extraction performance and enhance system applicability. By comprehensively calculating the scores of keywords through multiple algorithms and strategies, it reduces and avoids the high computational cost and resource consumption of deep learning models, improves the calculation speed, and is more suitable for question answering scenarios that require quick response. The rule- and strategy-based method usually has clear logic, the decision-making process is more transparent, which is convenient for debugging and optimization, and can intuitively control the behavior of the system without retraining the model; in the case of insufficient data support, the rule- and strategy-based method can still run; the method that does not rely on model prediction usually does not require a large amount of labeled data, reducing the cost of data collection and annotation. It is relatively simple to implement, has a short development cycle, and is suitable for teams with limited resources. Adjusting rules and strategies is usually easier than tuning model parameters, reducing the long-term maintenance cost of the system. Brief Description of the Drawings

[0079] Figure 1 It is a schematic flowchart of the file parsing and knowledge recall method for a large model question answering system in an embodiment of the present application. Detailed Embodiments

[0080] To make the objectives, technical solutions, and advantages of the present invention clearer and more understandable, the present invention will be described below through specific embodiments shown in the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present invention.

[0081] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms "a", "the", and "said" used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0082] It should be understood that although the terms first, second, third, etc. may be used in this disclosure to describe various information, such information should not be limited to these terms and should not be construed as indicating or implying relative importance. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0083] In the description of the present invention, unless otherwise specified and defined, it should be noted that the terms "mounted", "connected", and "coupled" should be understood in a broad sense. For example, it may be a mechanical connection, or it may be the communication inside two components. It may be directly connected, or it may be indirectly connected through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms can be understood according to specific circumstances.

[0084] To better understand the technical solution of the present invention, the present invention will be described in detail below with reference to the accompanying drawings.

[0085] This embodiment provides a file parsing and knowledge recall method for a large model question answering system, including the following steps:

[0086] S1. Parse the input document, extract the fine-grained information of each document, and store it in the database according to the predefined index mapping;

[0087] S2. Receive the natural language query request input by the user and generate a question text containing several related similar questions;

[0088] S3. Use multiple word segmentation methods to extract keywords and assign weights to the question text to generate a keyword weight mapping;

[0089] S4. Use keyword weights based on the BM25 algorithm to calculate the similarity scores between different documents and the question text, and preliminarily screen the documents associated with similar questions according to the relevance scores;

[0090] S5. Within the range of the preliminarily screened documents, accurately query specific fragments;

[0091] S6. Use the continuous interval annealing algorithm to smooth the adjacent matching fragments of the specific fragments, and sort the final results according to the comprehensive scores of the fragments and then return.

[0092] Step S1 specifically includes: Parsing the input document including the basic information of the file name, title, abstract, text, and mind map, splitting the document content into semantic blocks and slice windows at a granularity, assigning labels to each part of the content, and the labels include header, footer, title, text, table, picture, table of contents, and file name. Store the parsed document information in the Elasticsearch retrieval engine, perform word segmentation on the text of the input document, set field mapping, store the basic document information in the top-level field, store the text content after the document is chunked in the nested field, and record the fine-grained data of each fragment in the input document.

[0093] Storing in the Elasticsearch retrieval engine includes setting global indexes in Elasticsearch including the number of shards, backups, replica configurations, and the configuration of the custom BM25 similarity algorithm to support subsequent retrieval queries. Use ik_smart and ik_max_word to perform word segmentation on the text. Use two word segmenters, ik_smart and ik_max_word, to perform word segmentation on the input text, and ensure the indexing and retrieval efficiency through a custom analyzer; in this embodiment, the term frequency saturation of BM25 is set to 1.2, and the length normalization strength is set to 0.75.

[0094] Step S3 specifically includes:

[0095] S31. Weight initialization stage:

[0096] S311. Use the ik_smart word segmenter to perform coarse-grained word segmentation and establish an initial weight dictionary;

[0097] S312. Execute the cumulative enhancement strategy for repeated terms: increase the weight by α each time it repeats. In this embodiment, α = 0.3, and the weight of the i-th word segmentation in the ik_smart word segmentation set is calculated through (1):

[0098]

[0099] where n is the number of times the term w i appears in the ik_smart word segmentation result; wi represents the i-th word segment in the ik_smart word segment set, boost(w i ) represents w i word segment corresponding weight;

[0100] S32, Fine-grained expansion stage:

[0101] S321, Use ik_max_word for fine-grained word segmentation to generate a candidate word set

[0102] S322, Filter stop words and low-value part-of-speech:

[0103] S323, Implement hierarchical weight inheritance: If a fine-grained term is completely included by a coarse-grained term, inherit its weight; for long terms not included, assign a basic weight of 0.3, and for short terms, temporarily assign a basic value of 0.3. In this embodiment, terms with a length greater than 1 are long terms, and terms with a length equal to 1 are short terms. The j-th word segment in the ik_max_word word segment set is implemented through calculation (2):

[0104]

[0105] Among them, W smart represents the ik_smart word segmentation result set; w k represents a certain word in the ik_smart word segmentation result set; w j represents the j-th word segment in the ik_max_word word segment set, w j is w k itself or its subset;

[0106] S33, Keyword enhancement stage:

[0107] S331, Call jieba to extract and assign the top N1 keywords and weights ranked from high to low. In this embodiment, N1 = 5;

[0108] S332, Perform non-linear enhancement on existing terms according to (3):

[0109] boost new (w) = boost(w) × [1 + 0.2 × jieba_weight(w)] (3)

[0110] Among them, w is all the words extracted in S32, boost represents the original weight of the word, boost new represents the weight of the word after enhancement, and jieba_weight represents the jieba weight of the word;

[0111] S34, Weight correction stage:

[0112] S341. Single-word Punishment Mechanism: Punish single words according to (4), that is, for keywords with a length of 1 (single word), reduce their weights to 1 / 3 of the original:

[0113]

[0114] S342. Synonym Diffusion Strategy: Diffuse synonyms of words according to (5):

[0115]

[0116] Among them, s is a synonym of w, and the weight of s is B times the weight of w. B is a diffusion coefficient less than 1. In this embodiment, B = 0.1;

[0117] S35. Result Standardization: Keep two decimal places for the result and filter out low-confidence terms at the same time.

[0118] The calculation amount of step S3 is small, and there is no need to train a model, which solves the interference caused by too high keyword scores to the overall query during long queries.

[0119] Step S4 specifically includes:

[0120] S41. Candidate Document Retrieval: Build a preliminary retrieval model based on the BM25 algorithm. For a query q ∈ Q and any document d ∈ D in the document set D, perform multi-dimensional retrieval and matching through the keyword set K. Let the retrieval parameter boost be the weight enhancement coefficient, and the preliminary matching score is calculated according to (6):

[0121] Score base (q, d) = BM25(q, d) × boost(6)

[0122] S42. Structured Field Weight Calculation: Establish a feature weight triple (a, b, c) ∈ R 3 Respectively corresponding to the multi-dimensional correlation influence factors of the document file name, title field, and body content, construct a structured feature scoring function according to (7):

[0123]

[0124] Among them, represents tf k The word frequency statistic in a specific field of document d;

[0125] S43. Comprehensive Relevance Ranking: Generate the final relevance score through linear weighted fusion, and the ranking function is defined according to (8):

[0126] Score final (q, d) = Score base (q, d) + η × [Score filename + Scoretitle +Score content (8)

[0128] Among them, Q is the query space, D is the document set, K is the keyword set, R is the real number field, η is the first empirical decay coefficient, η is less than 1, and is used to balance the contribution of the basic retrieval score and the structured features; in this embodiment, η = 0.7;

[0129] Sorted in descending order according to Score final (q, d), the top N2 documents are intercepted to form the candidate document set D', which is used as the high-quality candidate for subsequent processing. In this embodiment, N2 = 5.

[0130] Step S5 specifically includes:

[0131] S51. Candidate segment retrieval: Based on the BM25 algorithm, a preliminary retrieval model is constructed. For any document d ∈ D in the query q ∈ Q and the document set D, multi-dimensional retrieval matching is performed through the keyword set K. Let the retrieval parameter boost be the weight enhancement coefficient, and the preliminary matching score is calculated according to (6):

[0132] Score base (q, d) = BM25(q, d) × boost(6);

[0133] S52. Linearly map the original score to the preset interval [ω min , ω max , ω min is 1.0, ω max is 1.2. The linear mapping of the original score is calculated according to (9) and (10) in sequence:

[0134]

[0135] Among them, μ is the second empirical decay coefficient, μ is less than 1. In this embodiment, μ = 0.7, α(d) represents the result of the min-max normalization method, max(S) is the maximum value of the scores calculated in S51, min(S) is the minimum value of the scores calculated in S51, and s(d) represents the score of a certain document in the document set.

[0136] S53. Calculate the similarity between the user query and the document segment according to (11):

[0137] Score vector = λ × cosine(v q , v d )(11)

[0138] Among them, λ is the weight decay coefficient, v q ​The vector representing the user's question, v d The vector representing the document fragment;

[0139] S54. Aggregate all the calculated result values as the score for each document fragment, and finally select the top N3 fragments ranked in descending order of the score as the final document fragments. In this embodiment, N3 = 20, and the score of the document fragment is calculated according to (12).

[0140] Score final = MAX(Score struct , Score vector )(12)

[0141] The comprehensive score of the fragments in S6 is calculated according to (13):

[0142] s i ′ = s i × γ i-1 (13)

[0143] Where i is any integer from 1 to n, n is the number of retrieved fragments recalled in the document, γ represents the attenuation coefficient, γ is less than 1, and in this embodiment, γ = 0.96.

[0144] By enhancing logical coherence in S6, the purpose of the annealing algorithm is to perform score attenuation on multiple retrieved fragments from the same document, suppress the concentrated display of similar fragments in the sorting results, improve the result diversity, and return the final results sorted according to the comprehensive score.

[0145] This application also provides a file parsing and knowledge recall device for a large model question - answering system, including:

[0146] Document parsing module: used to parse the input document, extract the fine - grained information of each document, and store it in the database according to the predefined index mapping;

[0147] Receiving module: used to receive the natural language query request input by the user and generate a question text containing several related similar questions;

[0148] Keyword extraction module: used to extract keywords and assign weights to the question text using multiple word - segmentation methods to generate a keyword - weight mapping;

[0149] Initial screening module: used to calculate the similarity score between different documents and the question text based on the keyword weights using the BM25 algorithm, and initially screen the documents associated with the similar questions according to the relevance score;

[0150] Precise query module: used to precisely query specific fragments within the range of the initially screened documents;

[0151] Return retrieval module: It is used to smooth the adjacent matching segments of specific segments by using the continuous interval annealing algorithm, sort the final results according to the comprehensive scores of the segments, and return the answers required by the user.

[0152] This application also provides a device, including:

[0153] One or more processors;

[0154] A storage device for storing one or more computer programs;

[0155] When the computer program is executed by the processor, it enables the processor to execute the steps of the above method.

[0156] This application also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. It is characterized in that the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the file parsing and knowledge recall method for the large model question answering system as claimed above.

[0157] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them; although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that: it is still possible to modify the specific implementation manners of the invention or perform equivalent replacements for some technical features; without departing from the spirit of the technical solutions of the present invention, they should all be covered within the scope of the technical solutions claimed by the present invention.

Claims

1. A file parsing and knowledge recall method for large model Q&A systems, characterized in that, The steps include the following: S1. Parse the input documents, extract the fine-grained information of each document, and store it in a database according to a predefined index mapping; S2. Receive the natural language query request input by the user and generate a question text containing several relevant similar questions; S3. Use multiple word segmentation methods to extract keywords and assign weights to the question text to generate a keyword weight mapping; S4. Use the keyword weights based on the BM25 algorithm to calculate the similarity scores between different documents and the question text, and preliminarily screen the documents associated with the similar questions according to the relevance scores; S5. Within the range of the initially screened documents, accurately query specific fragments; S6. Use the continuous interval annealing algorithm to smooth the adjacent matching fragments of the specific fragments, and sort the final results according to the comprehensive scores of the fragments and then return them.

2. The file parsing and knowledge recall method for the large model Q&A system according to claim 1, characterized in that Step S1 specifically includes: parsing the basic information of the input documents, splitting the document content into semantic blocks and slice windows at a granularity level, assigning labels to each part of the content, storing the parsed document information in an Elasticsearch retrieval engine, performing word segmentation on the text of the input documents, setting field mappings, storing the basic document information in the top-level fields, storing the text content after document chunking in the nested fields, and recording the fine-grained data of each fragment in the input documents.

3. The file parsing and knowledge recall method for the large model question answering system according to claim 2, wherein The basic information includes file name, title, abstract, body text, and mind map; the labels include header, footer, title, body text, table, picture, table of contents, and file name; storing in the Elasticsearch retrieval engine includes setting a global index in Elasticsearch, including the number of shards, backups, replica configurations, and configurations of a custom BM25 similarity algorithm; using ik_smart and ik_max_word to perform word segmentation on the text.

4. The method for document parsing and knowledge recall for large model Q&A system according to claim 3, characterized in that, Step S3 specifically includes: S31. Weight initialization stage: S311. Use the ik_smart tokenizer to perform coarse-grained word segmentation and establish an initial weight dictionary; S312. Execute an accumulative enhancement strategy for repeatedly occurring terms: the weight increases by α each time it repeats, where α is a weight coefficient less than 1, and the weight of the i-th word segmentation in the ik_smart word segmentation set is calculated by formula (1); where n is the number of times the term w i appears in the ik_smart word segmentation result; w i represents the i-th word segment in the ik_smart word segmentation set, and boost(w i ) represents w i the weight corresponding to the word segment; S32. Fine-grained expansion stage: S321. Use ik_max_word to perform fine-grained word segmentation to generate a candidate word set; S322. Filter stop words and low-value part-of-speech; S323. Implement hierarchical weight inheritance: if the fine-grained term is completely included by the coarse-grained term, inherit its weight; long terms not included are given a basic weight of 0.3; short terms are temporarily given a basic value of 0.3; S33. Keyword enhancement stage: S331. Call jieba to extract and assign the top N1 keywords and weights ranked from high to low; S332. Perform non-linear enhancement on the existing terms according to formula (3); boost new (w) = boost(w) × [1 + 0.2 × jieba_weight(w)] (3) Among them, w is all the words extracted by S22, boost(w) represents the original weight of the word, and boost new (w) represents the weight of the word after enhancement, and jieba_weight represents the jieba weight of the word w; S34. Weight correction stage: S341. Single-character word penalty mechanism, perform penalty on single characters according to formula (4); S342. Synonym diffusion strategy, perform synonym diffusion on words according to formula (5); Among them, s is a synonym of w, and the weight of s is B times the weight of w, where B is a diffusion coefficient less than 1; S35. Result standardization: The result is reserved to two decimal places, and at the same time, low-confidence terms are filtered.

5. The file parsing and knowledge recall method for the large model question answering system according to claim 4, characterized in that, In step S323, terms with a length greater than 1 are long terms, and terms with a length equal to 1 are short terms. The j-th term in the ik_max_word word segmentation set is implemented by calculating (2): Among them, W smart represents the ik_smart word segmentation result set; w k represents a word in the ik_smart word segmentation result set; w j represents the j-th word segmentation in the ik_max_word word segmentation set, and w j is w k itself or its subset.

6. The method for document parsing and knowledge recall for a large model question answering system according to claim 4, wherein Step S4 specifically includes: S41. Candidate document retrieval: Based on the BM25 algorithm, a preliminary retrieval model is constructed. For a query q ∈ Q and any document d ∈ D in the document set D, multi-dimensional retrieval matching is performed through the keyword set K. Let the retrieval parameter boost be the weight enhancement coefficient, and the candidate document score is calculated according to (6): Score base (q,d) = BM25(q,d) × boost(6) S42. Structured field weight calculation: Establish a feature weight triple (a, b, c) ∈ R 3 which respectively correspond to the multivariate relevance impact factors of the document file name, the title field, and the body content, and construct a structured feature scoring function according to (7): Among them, it represents tf k The word frequency statistic of a specific field in document d; S43. Comprehensive relevance ranking: The final relevance score is generated through linear weighted fusion, and the ranking function is defined according to (8): Score final (q,d) = Score base (q,d) + η × [Score filename + Score title + Score content ​ (8) Among them, Q is the query space, D is the document set, K is the keyword set, R is the real number field, η is the first empirical decay coefficient, and η is less than 1, which is used to balance the contribution of the basic retrieval score and the structured features; Sort by Score final (q, d) in descending order, and intercept the top N2 documents to form the candidate document set D', which serves as high-quality candidates for subsequent processing.

7. The file parsing and knowledge recall method for the large model question answering system according to claim 5, wherein Step S5 specifically includes: S51. Candidate fragment retrieval: Based on the BM25 algorithm, a preliminary retrieval model is constructed. For a query q ∈ Q and any document d ∈ D in the document set D, multi-dimensional retrieval matching is performed through the keyword set K. Let the retrieval parameter boost be the weight enhancement coefficient, and the preliminary matching score is calculated according to (6): Score base (q, d) = BM25(q, d) × boost(6); S52. Linearly map the original score to the preset interval [ω min , ω max by the min-max normalization method. ω min is 1.0, ω max is 1.2, and the linear mapping of the original score is calculated successively according to (9) and (10): Among them, μ is the second empirical attenuation coefficient, μ is less than 1, α(d) represents the result of the min-max normalization method, max(S) is the maximum value of the scores calculated in S51, min(S) is the minimum value of the scores calculated in S51, and s(d) represents the score of a certain document in the document set; S53. Calculate the similarity between the user query and the document fragment according to (11): Score vector = λ × cosine(v q , v d )(11) Among them, λ is the weight decay coefficient, v q represents the vector of the user's question, v d represents the vector of the document fragment; S54. Aggregate all the calculated result values as the score of each document fragment. Finally, select the top N3 fragments ranked in descending order of the score as the final document fragments, and the score of the document fragment is calculated according to (12): Score final = MAX(Score struct , Score vector )(12).

8. The method for document parsing and knowledge recall for large model Q&A systems according to claim 5, wherein The comprehensive score of the fragment in S6 is calculated according to (13): s i ′ = s i × γ i-1 (13) Among them, i is any integer from 1 to n, n is the number of retrieved fragments recalled in the document, and r represents the decay coefficient, and r is less than 1.

9. A file parsing and knowledge recall device for a large model question answering system, characterized in that, Including: Document parsing module: Used to parse the input document, extract the fine-grained information of each document, and store it in the database according to the predefined index mapping; Receiving module: Used to receive the natural language query request input by the user and generate a question text containing several related similar questions; Keyword extraction module: Used to extract keywords and assign weights to the question text using multiple word segmentation methods to generate a keyword weight mapping; Preliminary screening module: Used to calculate the similarity score between different documents and the question text based on the keyword weights using the BM25 algorithm, and preliminarily screen the documents associated with the similar questions according to the relevance score; Precise query module: Used to precisely query specific fragments within the range of the initially screened documents; Return retrieval module: Used to smooth the adjacent matching fragments of the specific fragments using the continuous interval annealing algorithm, and sort the final results according to the comprehensive score of the fragments and then return to input the answer required by the user.

10. A device, characterized in that, Including: One or more processors; Storage device, used to store one or more computer programs; When the computer program is executed by the processor, it causes the processor to execute the steps of any one of claims 1 to 8.

11. A storage medium, characterized in that, The storage medium is a computer-readable storage medium for computer-readable storage, characterized in that the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the file parsing and knowledge recall method for the large model question-answering system according to any one of claims 1 to 8.

Images