Method and system for retrieving DOCX document content based on keywords

Abstract

The invention belongs to the technical field of text processing, and particularly relates to a method and system for retrieving DOCX document content based on keywords, which comprises the following steps: analyzing an Office Open XML structure of a DOCX document, combining with multi-dimensional features such as style names, and utilizing a title classification score model to accurately distinguish a title and a text, so that a semantic hierarchical structure of the document is effectively reserved; and secondly, a multi-level semantic extension mechanism is introduced, and a Sension-BERT, a HowNet knowledge base and a Word2Vec model are fused, so that intelligent extension of synonyms and synonyms of keywords is realized, and the recall rate and semantic understanding ability of retrieval are remarkably improved. And in addition, a BM25 model is combined with paragraph length normalization and structure position weight to calculate a correlation score, so that retrieval results are sorted more accurately and reasonably. The construction of the reverse index is combined with the position coding and compression optimization strategy, and the retrieval efficiency and the storage performance are both considered.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of text processing, and particularly relates to a method and system for retrieving DOCX document content based on keywords, a terminal and a storage medium. BACKGROUND

[0002] With the popularity of digital office, the DOCX format document generated by Microsoft Word has become one of the most mainstream document formats in enterprises, educational institutions and scientific research fields. However, in the face of the growing number of electronic documents, how to quickly and accurately retrieve the required information from a large number of DOCX documents has become a technical problem to be solved in the current document processing field.

[0003] The current common document retrieval scheme mainly has the following limitations: first, although the traditional full-text search engine (such as the system based on Lucene) can realize basic keyword matching, it cannot effectively analyze the complex internal structure of the DOCX document (such as the title level, table data, style information, etc.), resulting in a lack of semantic hierarchy of the retrieval results. Secondly, most of the existing retrieval tools use literal matching strategies, which are difficult to cope with the complexity of Chinese semantics, especially the lack of recognition ability for synonyms, near-synonyms and variant forms of professional terms. In addition, the traditional method mainly relies on the edit distance algorithm when dealing with fuzzy queries, and fails to combine the context semantics for intelligent expansion, so that the retrieval results often contain a large amount of irrelevant content.

[0004] It is particularly important to note that the DOCX document uses the Office Open XML (OOXML) standard for storage, and contains a large amount of XML structured data inside. The existing technical solutions usually convert the DOCX document into pure text for retrieval, which will cause the loss of style information, resulting in the inability to distinguish the importance difference between the title and the body. SUMMARY

[0005] In view of the above deficiencies of the prior art, the application provides a method, system, terminal and storage medium for retrieving DOCX document content based on keywords.

[0006] In the first aspect, the application provides a method for retrieving DOCX document content based on keywords, comprising: S1, loading the DOCX file through the document analysis module, traversing all paragraphs and tables, extracting the paragraph text and style information, and classifying the paragraphs into titles or bodies according to at least one of the characteristics of the style name, font size and bold attribute, to form structured text data; S2, text preprocessing of the structured text data: removing non-English characters and redundant spaces, using a word segmentation tool to segment the remaining text, and loading a stop word table to filter meaningless words to obtain the purified word sequence; S3. Construct an inverted index based on word sequences, establishing a one-to-one mapping relationship between keywords and their occurrence positions, where the position must include at least the paragraph number and the offset within the paragraph. S4. Receive keywords input by the user, use a natural language processing model to perform semantic expansion on the keywords, obtain a set of synonyms and / or near-synonyms of the original keywords, and form an expanded keyword list; S5. Perform multi-pattern matching in the inverted index based on the expanded keyword list. The multi-pattern matching includes at least one of exact matching, fuzzy matching, and synonym matching to obtain the initial set of matching paragraphs. S6. Calculate the relevance score for each paragraph in the initial set of matched paragraphs, and output the sorting results after sorting them in descending order of relevance score. S7. Highlight the keywords in the sorting results in the corresponding paragraphs, and extract the matching sentences and their adjacent sentences to generate a summary. The length of the summary is dynamically determined by the maximum number of sentences configured by the user. S8. Export the highlighted paragraphs and summaries as HTML, TXT, or PDF formats to persist the search results.

[0007] Further improvements to this technical solution include step S1, which includes: S11. Load the DOCX file at the specified path through the document parsing module, use the python-docx library to parse the Office Open XML structure of the document, generate a document object model tree, which contains all paragraph elements and table elements, and ensures the integrity of the document structure. S12. Traverse each paragraph element in the document object model tree, extract the text content of the paragraph, and obtain the style attribute information of the paragraph. The style attribute information includes at least the style name, font size, and bold attribute. The style name is extracted from the name attribute of the paragraph style, the font size is extracted from the font size attribute, and the bold attribute is extracted from the font weight attribute. S13. Based on the extracted style attribute information, calculate the title classification score for each paragraph, and classify the paragraph into title or body text according to the comparison result between the score and the preset threshold. The formula for calculating the title classification score is as follows: ; in, This indicates the heading category score for paragraph p; It is an indicator function. When the style name of paragraph p The value is 1 if it belongs to the predefined heading style set T, and 0 otherwise. This indicates the font size of paragraph p, in points. represents the bold attribute of paragraph p, which is 1 if bold and 0 otherwise; represents the base font size, which is a preset constant; , , is a weight coefficient; if , paragraph p is classified as a title, otherwise as a text, where θ is a preset score threshold.

[0008] Further improvements of the technical solution include that step S2 comprises: S21, denoising the structured text data, removing non-English characters and extra spaces by regular expression matching and character encoding recognition, and retaining valid text content; S22, performing word segmentation on the denoised text using a Chinese word segmentation tool, calculating the optimal word segmentation path by dictionary matching and hidden Markov model, and dividing the text into a word sequence; S23, loading a stop word list to filter the word sequence, combining word frequency statistics and semantic weight calculation to determine the removal threshold of meaningless words: ; wherein, represents the filtering weight of the word w; is the word frequency; N is the total number of words; is the document frequency containing the word w; is the total number of documents; when , the word w is filtered as a meaningless word, is a preset filtering threshold.

[0009] Further improvements of the technical solution include that step S3 comprises: S31, position encoding the purified word sequence, assigning each word a unique paragraph number and paragraph offset, and establishing a word position mapping table; the position encoding uses a segmented linear mapping function: ; wherein, represents the position coordinates of the word ; p is the paragraph number; is the paragraph offset; is the average paragraph length; is the sequential index of the word in the entire text; S32, constructing an inverted index based on the word position mapping table, using hash mapping and linked list structure to store the mapping relationship from the keyword to the occurrence position; the definition formula of the inverted index item is: ; wherein, represents the inverted index item of the keyword t; is the paragraph number; is the paragraph offset; is the frequency of occurrence of the keyword at position , m is the total number of paragraphs containing the keyword, and n is the number of keyword occurrence positions in a single paragraph; S33, the inverted index is compressed and optimized, and a multi-level cache mechanism is established to improve retrieval efficiency; the index compression rate calculation formula is: ; wherein, represents the index compression rate; is the size of the compressed index; is the original index size; the cache hit rate optimization formula is: ; wherein, is the cache hit rate; is the number of cache hits; is the total number of accesses.

[0010] Further improvements of the technical solution include that step S4 comprises: S41, receiving the keyword string input by the user, standardizing and preprocessing the keyword, including converting the keyword to a unified lowercase form, removing preset characters and unnecessary spaces, and using a Chinese word segmentation tool to segment the composite keyword into single word units to form a standardized keyword sequence; S42, using a pre-trained natural language processing model to perform semantic analysis on the standardized keyword sequence, and using a multi-level semantic expansion strategy to obtain a synonym and near-synonym set for each keyword, the multi-level semantic expansion strategy comprising the following calculation process: First expansion: using a Sentence-BERT sentence encoding model to calculate the semantic similarity between the keyword and the candidate word, the semantic similarity calculation formula being: ; wherein, A and B are the vector representations of the keyword and the candidate word, respectively; is the cosine similarity value; when , the candidate word is added to the expansion set; is a preset similarity threshold; Second expansion: combining the HowNet synonym knowledge base to obtain explicit synonyms of the keyword and verifying the validity of the synonyms; Third expansion: using a Word2Vec word vector model to calculate the cosine similarity between the keyword and the candidate word in the vector space trained on the domain corpus, and selecting the top K most similar words as the near-synonyms, K being a preset expansion number; S43, the synonym and near-synonym set obtained by multi-level semantic expansion is de-duplicated and merged to form an expanded keyword list, and each expanded keyword is assigned a weight, and the weight calculation formula is: ; wherein, represents the weight of the expanded keyword e; is the semantic similarity of the expanded keyword e and the original keyword k; is the expansion level coefficient; and is the weight coefficient.

[0011] Further improvements of the technical solution are that step S5 includes: S51, parse the expanded keyword list, and construct a query expression tree according to the logical operators input by the user, wherein the logical operators include AND and OR, the leaf nodes of the query expression tree represent single keywords, the internal nodes represent logical operations, and the correct operator priority is ensured; S52, for each keyword node in the query expression tree, perform multi-pattern matching in the inverted index, and the multi-patterns include exact matching, fuzzy matching and synonym matching; the fuzzy matching uses edit distance similarity calculation: ; wherein, represents the fuzzy matching similarity of the words and ; represents the edit distance; represents the character length of the words ; when , it is considered as matching; is a preset fuzzy matching threshold; The synonym matching uses semantic similarity calculation: ; wherein, represents the semantic similarity of the words and ; represents the word vector representation of the words ; when , it is considered as matching; is a preset synonym matching threshold; S53, combine the matching results according to the logical operations of the query expression tree, take the intersection of paragraph sets for AND operation, take the union set for OR operation, and calculate the matching strength score of each paragraph: ; wherein, represents the matching strength score of paragraph p; r is the total number of extended keywords; is the weight of the extended keyword ; is an indicator function, taking value 1 when paragraph p contains keyword , otherwise 0; finally, the paragraph with matching strength score higher than a preset threshold is selected as the initial matching paragraph set.

[0012] Further improvement of the technical solution is that step S6 includes: S61, for each paragraph in the initial matching paragraph set, calculate the original relevance score based on keyword matching, which is calculated by combining the term frequency of the keyword in the paragraph and the inverse document frequency, wherein the term frequency represents the number of occurrences of the keyword in the paragraph, and the inverse document frequency represents the rarity of the keyword in the entire document set; the original relevance score is calculated by using the BM25 model, and the formula is: ; wherein, represents the original relevance score of paragraph p; represents the i-th extended keyword, and r is the total number of extended keywords; represents the inverse document frequency of keyword , and the calculation formula is , wherein M is the total number of documents in the document set; is the number of documents containing keyword ; represents the term frequency of keyword in paragraph p; represents the length of paragraph p; represents the average length of all paragraphs in the document set; and b are adjustable parameters, wherein controls the saturation of the term frequency, and is usually taken as 1.2 to 2.0, and b controls the influence of paragraph length normalization, and is usually taken as 0.5 to 0.8; S62, the original relevance score is subjected to paragraph length normalization processing to eliminate the bias of long paragraphs due to high term frequency, and to ensure the comparability of scores of paragraphs with different lengths; the normalization processing adopts the length weight adjustment formula: ; wherein, represents the normalized relevance score; is the normalization intensity coefficient; represents the exponential function, which is used to smooth the influence of length difference; S63, adjust the normalized score of the structural position weight of the paragraph, wherein the weight of the title paragraph is higher than that of the text paragraph, and calculate a final relevance score, and output the ranking result in descending order of the score; the final relevance score calculation formula is: ; wherein, represents the final relevance score of paragraph p; is an indicator function, which takes the value of 1 when the paragraph p is classified as a title, otherwise 0; is a title weight bonus coefficient; when ranking, the paragraphs are arranged in descending order of the value from high to low, and the final search result list is generated.

[0013] In a second aspect, the present application provides a system for searching DOCX document content based on keywords, comprising: A document parsing module is used to load the DOCX file, traverse all paragraphs and table elements in the document, extract the text content and style information of each paragraph, and classify the paragraphs into titles or texts according to at least one of the characteristics of style name, font size, and bold attribute, and form structured text data; A text preprocessing module is connected with the document parsing module, and is used to preprocess the structured text data, including removing non-English characters and redundant spaces, using a word segmentation tool to process text segmentation, and loading a stop word table to filter meaningless words, and obtaining a purified word sequence; An inverted index construction module is connected with the text preprocessing module, and is used to construct an inverted index based on the purified word sequence, and establish a one-to-one mapping relationship from keywords to occurrence positions, wherein the position information at least includes paragraph number and paragraph internal offset; A semantic expansion module is used to receive user input keywords, and uses a natural language processing model to perform semantic analysis on the keywords, to obtain a synonym and / or near-synonym set of the original keywords, and form an expanded keyword list; A multi-mode matching module is connected with the semantic expansion module and the inverted index construction module, and is used to perform multi-mode matching in the inverted index according to the expanded keyword list, wherein the matching mode at least includes one of exact matching, fuzzy matching and synonym matching, and an initial matching paragraph set is obtained; A relevance calculation module is connected with the multi-mode matching module, and is used to calculate a relevance score for each paragraph in the initial matching paragraph set, and generate a ranking result in descending order of the score; A result display module is connected with the relevance calculation module, and is used to highlight mark the keywords in the corresponding paragraph in the ranking result, and extract the matching sentence and its adjacent sentences to generate an abstract, wherein the length of the abstract is dynamically determined by the maximum number of sentences configured by the user; ​The result exporting module, connected with the result displaying module, is used for exporting the highlighted paragraph and the abstract in HTML, TXT or PDF format, so as to realize the persistent storage of the search result.

[0014] In a third aspect, the present application provides a terminal, comprising: a processor and a memory, The memory is used for storing a computer program, The processor is used for calling and running the computer program from the memory, so that the terminal executes the method of the terminal described above.

[0015] In a fourth aspect, the present application provides a computer storage medium, wherein the computer readable storage medium stores instructions, when the instructions are run on a computer, the computer executes the method described in the above aspects.

[0016] The present application has the following beneficial effects: The present application accurately distinguishes the title and the body by deeply analyzing the Office Open XML structure of the DOCX document, combining the multi-dimensional features such as style name, font size and bold attribute, and using the title classification score model, effectively retains the semantic hierarchical structure of the document, and overcomes the semantic loss problem caused by text flattening in traditional methods. Secondly, the multi-level semantic expansion mechanism is introduced, the Sentence-BERT, HowNet knowledge base and Word2Vec model are fused, the synonym and near-synonym of the keyword are intelligently expanded, and the recall rate and semantic understanding ability of the retrieval are significantly improved. Thirdly, the BM25 model is used to calculate the relevance score combined with the paragraph length normalization and the structure position weight (such as title weighting), so that the ranking of the retrieval result is more accurate and reasonable. The construction of the inverted index combines the position coding and compression optimization strategy, and takes into account the retrieval efficiency and storage performance. In addition, the system supports highlight display, dynamic abstract generation and multi-format export, which greatly improves the user interaction experience. The overall scheme realizes the whole process optimization from structure analysis, semantic understanding to intelligent sorting, significantly improves the accuracy, intelligent level and practicability of the DOCX document retrieval. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, brief introductions will be given below to the drawings needed to be used in the embodiments or prior art descriptions. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0018] Figure 1 A schematic flowchart of a method according to an embodiment of the present application.

[0019] Figure 2 A schematic block diagram of a system according to an embodiment of the present application.

[0020] Figure 3 A structural schematic diagram of a terminal provided by an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the objectives, characteristics and advantages of the present application more obvious and easy to understand, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the specific embodiments. Obviously, the embodiments described below are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terminology used in the description of the present application herein only for the purpose of describing specific embodiments and is not intended to limit the present application.

[0023] The method for retrieving DOCX document content based on a keyword provided by the embodiment of the present application is executed by a computer device, and accordingly, the system for retrieving DOCX document content based on a keyword runs in the computer device.

[0024] Figure 1 is a schematic flowchart of the method of an embodiment of the present application. Wherein, Figure 1 The execution subject can be a system for retrieving DOCX document content based on a keyword. The order of the steps in the flowchart can be changed according to different needs, and some can be omitted.

[0025] As Figure 1 indicated, the method comprises: S1, loading a DOCX file through a document parsing module, traversing all paragraphs and tables, extracting paragraph text and style information, and classifying the paragraphs into titles or texts according to at least one of the characteristics of style name, font size and bold attribute, to form structured text data; S2, text preprocessing on the structured text data: removing non-English characters and redundant spaces, using a word segmentation tool to segment the remaining text, and loading a stop word table to filter meaningless words, to obtain a purified word sequence; S3, constructing an inverted index based on the word sequence, establishing a one-to-one mapping relationship from the keyword to the occurrence position, and the position at least including paragraph number and paragraph internal offset; S4, receiving a keyword input by a user, using a natural language processing model to perform semantic expansion on the keyword, obtaining a synonym and / or near-synonym set of the original keyword, and forming an expanded keyword list; S5, performing multi-mode matching in the inverted index according to the extended keyword list, the multi-mode at least including one of exact matching, fuzzy matching and synonym matching, to obtain an initial matching paragraph set; S6, calculating a relevance score for each paragraph in the initial matching paragraph set and outputting a sorting result in descending order of the relevance score; S7, highlighting the keyword in the corresponding paragraph in the sorting result, and extracting a matching sentence and adjacent sentences before and after the matching sentence to generate an abstract, the length of the abstract being dynamically determined by a maximum number of sentences configured by a user; S8, exporting the highlighted paragraph and the abstract in HTML, TXT or PDF format to complete the persistence of the retrieval result.

[0026] In order to facilitate the understanding of the application, the principle of the method for retrieving DOCX document content based on keywords provided by the application is described below, combined with the process of retrieving DOCX document content based on keywords in the embodiment, to further describe the method for retrieving DOCX document content based on keywords provided by the application.

[0027] Firstly, step S1 includes: S11, loading the DOCX file of the specified path through the document parsing module, parsing the Office Open XML structure of the document by using the python-docx library, generating a document object model tree, the model tree containing all paragraph elements and table elements, and ensuring the integrity of the document structure; S12, traversing each paragraph element in the document object model tree, extracting the text content of the paragraph, and obtaining the style attribute information of the paragraph, the style attribute information at least including the style name, the font size and the bold attribute, wherein the style name is extracted from the name attribute of the paragraph style, the font size is extracted from the font size attribute, and the bold attribute is extracted from the font weight attribute; S13, calculating the title classification score of each paragraph based on the extracted style attribute information, and classifying the paragraph as a title or a body according to the comparison result of the score and a preset threshold, wherein the calculation formula of the title classification score is: ; Wherein, represents the title classification score of the paragraph p; is an indicator function, , when the style name of the paragraph p belongs to the pre-defined title style set T, the value is 1, otherwise the value is 0; represents the font size of the paragraph p, the unit is pound; represents the bold attribute of the paragraph p, if bold, the value is 1, otherwise the value is 0; represents the reference font size, which is a preset constant; , , is a weight coefficient; if , the paragraph p is classified as a title, otherwise as a body, where θ is a preset score threshold.

[0028] The python-docx library (version 0.8.11, which is compatible with the OOXML standard and supports all versions of DOCX 2007-2021) is used as the core parsing tool, and it can be installed by pip install python-docx==0.8.11. If the document to be parsed is in the old.doc format, it needs to be converted to.docx format by the unoconv tool (version 0.9.0) first, with the conversion command being unoconv -f docx source file.doc. Ensure that the input document is in.docx format.

[0029] Loading logic: Get the absolute path (D: / document_repo / technical document.docx) through the file path construction function get_doc_path(doc_name), and call docx.Document(docx_path) to load the document. Exception handling: For the three common exceptions docx.opc.exceptions.PackageNotFoundError (file damage or illegal format), FileNotFoundError (path error), and PermissionError (insufficient permissions), the processing flow of "record error log (including document path, error time, error details) → return empty document object → terminate current parsing task" is executed respectively. The log storage path is. / log / doc_parse_error.log, which is convenient for subsequent problem troubleshooting.

[0030] After loading the document, the following two verifications are performed to ensure the integrity of the OOXML structure: Element integrity: Check whether doc.element.body contains at least one paragraph element (w:p tag) or table element (w:tbl tag). If both are empty, it is determined as "empty document" and a warning is output. Attribute integrity: Randomly select 5% of the paragraph elements and verify whether they contain core sub-tags such as w:style (style), w:r (text run block). If the missing rate exceeds 10%, it is determined as "structurally damaged document", triggering the reloading or format repair process (calling the docx.oxml.ns module to repair missing tags).

[0031] Iterate through all the sub-elements under the document body (w:body) using doc.element.body.iter(), distinguish the element type by the element tag suffix (p for paragraph, tbl for table), and store them in the elements = [] list, each list item being a tuple (element type, element object) (e.g. ("paragraph", para_obj), ("table", tbl_obj)).

[0032] Text extraction: For each paragraph object para, call para.text.strip() to get the text content. If the text is empty (length < 1), mark it as "empty paragraph" and skip the subsequent style extraction. Style name extraction: Get the style name by para.style.name. If para.style is None (style information missing), assign the default value "Normal". Font size extraction: Iterate through the paragraph's text runs and take the font size (run.font.size.pt) of the first non-empty run in "pounds (pt)". If all run font sizes are None, assign the default value of 12pt (common font size for Chinese documents). Bold attribute extraction: Determine the run.font.bold attribute of the first non-empty run. If it is True or None (default bold when not explicitly set in some documents), take the value 1. If it is False, take the value 0.

[0033] Recursive table parsing: For each table object tbl, iterate through its rows (tbl.rows) and cells (row.cells), and call cell.text.strip() to extract the text of each cell. Merge them into "table sub-paragraphs" (format: "Table-Row-Column: Text Content", e.g. "Table1-2-3: Retrieval accuracy ≥ 90%"). Context association: Associate "table sub-paragraphs" with the "associated content" of the previous non-empty paragraph where the table is located, ensuring that the table text is semantically coherent with the context, e.g. the table is located after paragraph P5, and the table sub-paragraph is stored as a sub-item of P5's "content" field.

[0034] Score calculation: For each paragraph p, substitute the above formula to calculate ; Type determination: If (i.e. ), determine it as "title" and further subdivide the level according to the score, as a first-level title, as a second-level title, For tertiary title; if , determine as "text".

[0035] Mixed style paragraph: if a paragraph contains both title style and non-title style runs, take the style of the run with a proportion ≥ 50% as .

[0036] Short paragraph determination: if the paragraph text length < 10 characters and , even , still determine as "potential title", which needs manual review (review prompts can be output through the system interface).

[0037] Secondly, step S2 includes: S21, denoising the structured text data, removing non-English characters and redundant spaces by regular expression matching and character encoding recognition, and retaining valid text content; S22, performing word segmentation processing on the denoised text using a Chinese word segmentation tool, segmenting the text into a word sequence by dictionary matching and hidden Markov model calculation of the optimal segmentation path; S23, loading a stop word list to filter the word sequence, combining word frequency statistics and semantic weight calculation to determine the removal threshold of meaningless words: ; Among them, represents the filtering weight of the word w; is the word frequency; N is the total number of words; is the document frequency containing the word w; is the total number of documents; when , the word w is filtered as a meaningless word, is a preset filtering threshold.

[0038] The noise to be removed includes: Special characters: emojis, symbols (such as ￥, %, #, @), full / half special symbols (such as _, —, ~); Redundant spaces: 2 or more consecutive spaces (including full-width space " "); Non-visible characters: line feed, tab (\t), carriage return (\r); Encoding exception characters: garbled characters due to encoding incompatibility.

[0039] The chardet library (version 3.0.4, installed through pip install chardet==3.0.4) is used to detect the text encoding; Execution flow: Traverse the text field of each paragraph in the structured text data output by S1, call chardet.detect(text.encode()) to obtain the encoding type (e.g. "utf-8", "gbk", "gb2312"); if the encoding is not "utf-8", call text.encode(encoding='utf-8',errors='replace').decode('utf-8') to convert it to UTF-8 encoding, where errors='replace' ensures that encoding error characters are replaced with a preset character before being processed uniformly.

[0040] Regular expression: The pattern for matching non-Chinese and English characters and numbers is r'[^\u4e00-\u9fa5a-zA-Z0-9]'; Execution steps: ① Call re.sub(pattern=r'[^\u4e00-\u9fa5a-zA-Z0-9]', repl=' ', text) to replace all non-Chinese and non-numeric characters with a half-width space. ② Call re.sub(pattern=r'\s+', repl=' ', text) to merge consecutive spaces (including half-width spaces converted from full-width spaces) into a single space. ③ Call text.strip() to remove leading and trailing spaces; if the length of the processed text is <1 (only containing spaces or an empty string), mark it as "invalid text" and discard it.

[0041] Preserve key symbols: If the text contains technical parameter symbols (e.g. "≥", "≤", "%"), they can be preserved by extending the regular expression r'[^\u4e00-\u9fa5a-zA-Z0-9≥≤%]', which needs to be enabled in the system configuration file config.ini. Long text segmentation: For paragraphs longer than 1000 characters, segment them by punctuation (., ;) and remove noise paragraph by paragraph to avoid high memory usage in single processing (threshold can be configured by max_text_length=1000).

[0042] Use the jieba segmentation library (version 0.42.1, installed by pip install jieba==0.42.1), which supports custom dictionaries and HMM models, suitable for Chinese technical document segmentation scenarios.

[0043] Dictionary format: Adopting "word frequency word class" three column format (for example, "DOCX document 20 n" "inverted index 15 n" "semantic expansion 12 v"), where "word frequency" is based on 5000 documents retrieval field corpus statistics, "word class" refers to "Chinese information processing word segmentation specification"; Loading mode: Call jieba.load_userdict('domain_dict.txt') to load the domain dictionary (path is. / dict / domain_dict.txt), which has higher priority than the default jieba dictionary, ensuring that technical terms are not segmented (for example, "document parsing module" is not segmented into "document / parsing / module").

[0044] Adopting "precise mode + HMM auxiliary", call jieba.cut(text, cut_all=False, HMM=True), where: cut_all=False: Turn off full mode to avoid over-segmentation (for example, "search system" is not segmented into "search / system / system"); HMM=True: Enable HMM model to handle out-of-vocabulary words (for example, "Sentence-BERT", "BM25 model" and other foreign technical terms).

[0045] Generate initial word segmentation candidate set based on dictionary matching, calculate the probability of each candidate word sequence through HMM model, and select the sequence with the highest probability as the optimal path; HMM parameters: State set is {B (beginning of word), M (middle of word), E (end of word), S (single word)}, and the observation set is Chinese characters; Example: For the text "based on keyword DOCX search system", the initial candidate set contains "based / keyword / of / DOCX / search / system" "based / keyword / of / DOCX / search / system" and so on, and after HMM calculation, the "based / keyword / of / DOCX / search / system" with the highest probability is selected as the final word sequence.

[0046] English term reservation: If the segmentation result contains consecutive English characters (for example, "DOCX", "BERT"), call re.match(r'[a-zA-Z0-9]+', word) to detect, and if it is determined to be an English term, the original word is reserved and not segmented; Number and unit combination: For "200ms" "92%" and other number + unit combinations, match through regular expression r'[0-9]+[a-zA-Z%≤≥]+', and merge into a single word (for example, "200ms" is not segmented into "200 / ms").

[0047] Basic stop word list: Derived from open-source standard stop word lists (Harbin Institute of Technology stop word list, Baidu stop word list, Sichuan University stop word list), it contains a total of 1208 general meaningless words, such as "的", "了", "在", "和", "是", "我", "你"; Domain stop word list: Customized for the document retrieval domain, it contains 32 high-frequency meaningless words, such as "文档", "文件", "模块", "系统", "方法", "步骤" (these words appear very frequently in retrieval domain documents but have no actual retrieval discrimination); Dynamic stop word list: Updated in real-time based on user retrieval logs. If the retrieval click-through rate (number of clicks / number of retrievals) of a certain word is < 5% for 7 consecutive days, it is automatically added to the dynamic stop word list, updated once a week, and initially empty.

[0048] Traverse the word segmentation results of all documents and count using collections.Counter() ; Construct a "word - document ID" mapping table, count the number of document IDs where each word appears, and obtain ; For each word w, substitute it into the formula to calculate ; If (that is, ), or w exists in the three-level stop word list, it is determined as a "meaningless word" and excluded; otherwise, it is retained in the purified word sequence; Result storage: The purified word sequence is associated with the corresponding paragraph ID and document ID, and stored in JSON format (for example, {"doc_id":"doc_001","para_id":"P1","word_seq":["基于","关键词","DOCX","检索系统"]}).

[0049] Next, step S3 includes: S31. Perform position encoding on the purified word sequence, assign a unique paragraph number and an offset within the paragraph to each word, and establish a word position mapping table; The position encoding uses a piecewise linear mapping function: ; Among them, represents the position coordinates of the word ; p is the paragraph number; is the offset within the paragraph; is the average paragraph length; is the sequential index of the word in the full text; S32. Construct an inverted index based on the word position mapping table, and use a hash map and a linked list structure to store the mapping relationship from keywords to their occurrence positions; The definition formula of the inverted index entry is: ; in, This represents the inverted index item for the keyword t; Paragraph numbering; This is the offset within the paragraph; For keywords in position The frequency of occurrence of the keyword is m, where m is the total number of paragraphs containing the keyword, and n is the number of positions where the keyword appears in a single paragraph. S33. Compress and optimize the inverted index, and establish a multi-level caching mechanism to improve retrieval efficiency; the formula for calculating the index compression rate is: ; in, Indicates the index compression ratio; This is the size of the compressed index; The original index size is used; the cache hit rate optimization formula is: ; in, For cache hit rate; This represents the number of times the cache was hit. This represents the total number of visits.

[0050] It uses JSON Lines (.jsonl) format for storage, with each line corresponding to the position information of a word (e.g., {"doc_id":"doc_20251022_001","word":"keyword","p":1,"o_i":2,"i":2,"L":2}), supports reading and writing line by line to avoid excessive memory consumption; Exception handling: If the number of words in a paragraph is 0 (invalid text removed by S2), skip the paragraph numbering assignment and ensure that p increases continuously (for example, if P2 is an invalid paragraph, P1 corresponds to p=1, and P3 corresponds to p=2).

[0051] Hash mapping: Python's built-in dict type is used (average search time complexity O(1)), the key is the keyword t (string type, case sensitive, e.g. "DOCX" ≠ "docx"), and the value is a linked list (simulated using Pythonlist, which supports dynamic node appending). Linked list nodes: Each node is a tuple type with the format (p_j, list_o, f_jk) (e.g., (1, [3],1) indicates that the keyword is in paragraph 1, offset 3, and frequency 1).

[0052] Initialize index: Create an empty hash map: `inverted_index={}`; Traverse the position mapping table: Read the .jsonl file generated by S31 line by line, and extract the word (t) and p (p) of each word. ), o_i (i.e. ); Update index logic: If t is not in inverted_index: create a new key-value pair, and initialize the linked list to [(p_j,[o_i], 1)] (first occurrence, frequency 1); If t is already in inverted_index: Traverse its linked list nodes and check if it exists. Consistent nodes: If it exists: will The f_jk is appended to the list_o of this node and updated to len(list_o); If it does not exist: append a new node (p_j, [o_i], 1) to the linked list; Index verification: After the index is built, randomly select 10% of the keywords and check whether the p_j and list_o in their index entries are consistent with the position mapping table to ensure that there are no omissions or errors.

[0053] Duplicate words: If the same word appears consecutively in the same paragraph (e.g., "search search"), it is recorded according to the actual offset, and f_jk is accumulated normally; Mixed Chinese and English vocabulary: For example, “DOCX search”, the S2 segmentation results are used as independent keywords t, and its index terms are logically consistent with Chinese vocabulary.

[0054] Delta encoding is used for paragraph numbering in inverted indexes. The "orderly increasing" characteristic can significantly reduce the number of bytes to be stored.

[0055] Regarding paragraph numbering For each keyword t in the linked list, store the first paragraph number. Subsequent paragraph numbers are stored as "current" -Previous (difference) For example: original Sequence [1,2,5] → Compressed and stored as [1, 1(2-1), 3(5-2)].

[0056] For offset If the offsets within a paragraph are ordered (e.g., [3,6,8]), similarly store the first offset. Subsequent storage difference (e.g., [3,3 (6-3),2 (8-6)]); frequency : no compression, direct storage (due to , compression yields no significant benefit).

[0057] Hit rate optimization strategy: L1 cache update: every time a keyword t is retrieved, if it hits L1, update its "last access time"; if it misses L1 but hits L2, move t to L1 (if L1 is full, remove the keyword with the earliest "last access time"); L2 cache update: if t misses L1 and L2, load it from the disk index and store it in L2 (if L2 is full, remove the keyword with the earliest "last access time"); High-frequency word statistics: every morning, statistics of the previous day's retrieval log are updated to update the keyword list of L1 and L2 (to ensure that high-frequency words are stored in L1 first).

[0058] By using a piecewise linear mapping function to convert "paragraph number + offset" into a unique position coordinate, the defect of the prior art that "only recording paragraph ID cannot locate specific words within the paragraph" is avoided. For example, when the user searches for "DOCX", the position coordinate can be used to directly locate the 3rd word in P1, without the need to traverse the entire paragraph, and the positioning accuracy is improved from "paragraph level" to "word level".

[0059] In addition, step S4 includes: S41, receiving the keyword string input by the user, standardizing and preprocessing the keywords, including converting the keywords to a unified lowercase form, removing preset characters and unnecessary spaces, and using a Chinese word segmentation tool to segment complex keywords into single word units to form a standardized keyword sequence; S42, using a pre-trained natural language processing model to perform semantic analysis on the standardized keyword sequence, and using a multi-level semantic expansion strategy to obtain a synonym and near-synonym set for each keyword, the multi-level semantic expansion strategy including the following calculation process: First-level expansion: use the Sentence-BERT sentence encoding model to calculate the semantic similarity between the keyword and the candidate word, and the semantic similarity calculation formula is: ; Where A and B are the vector representations of the keyword and the candidate word, respectively; is the cosine similarity value; when , the candidate word is added to the expansion set; is a preset similarity threshold; Second-level expansion: combine the HowNet synonym knowledge base to obtain the explicit synonyms of the keyword, and verify the validity of the synonyms; Three-level expansion: using the vector space obtained by training the Word2Vec word vector model on the domain corpus, calculate the cosine similarity between the key words and the candidate words, select the top K words with the highest similarity as the synonyms, K is the preset expansion number; S43, the synonym and synonym set obtained by multi-level semantic expansion is de-duplicated and merged to form an expanded keyword list, and each expanded keyword is assigned a weight, and the weight calculation formula is: ; Among them, represents the weight of the expanded keyword e; is the semantic similarity between the expanded word e and the original keyword k; is the expansion level coefficient; and is the weight coefficient.

[0060] Calling keyword_str.lower() converts the user input keyword string (for example, "DOCX document retrieval" "retrieval system") to all lowercase form ("docx document retrieval" "retrieval system"), avoiding matching omissions due to case differences (for example, "DOCX" and "docx").

[0061] Pre-set character removal: Define "preset characters" as: non-Chinese and English characters, meaningless separators (such as "_" "-" "#" "@"), and special symbols (such as "!" "?" ";" ); Use the regular expression r'[^\u4e00-\u9fa5a-zA-Z0-9]' to match the preset characters, and call re.sub(pattern, repl='', keyword_str) to remove them; Example: input "DOCX-Document Retrieval!" → processed as "docx document retrieval".

[0062] Excess space cleaning: call re.sub(r'\s+', repl=' ',keyword_str).strip(), merge consecutive spaces (including full-width spaces) into a single space, and remove the leading and trailing spaces; If the length of the processed string is <1 (only contains spaces), prompt the user "Please enter a valid keyword" and terminate the process.

[0063] Compound keyword segmentation and disassembly: Call jieba.cut(keyword_str, cut_all=False, HMM=True) to combine the document retrieval domain dictionary (containing "docx document" "retrieval system" and other terms) loaded in step S2 to split the compound keyword into single word units; Rule: If the length of the split word is < 2 (e.g. "of" "in"), and not in the domain core term list, remove the word; finally form the "standardized keyword sequence" (list form); Example: Compound keyword "docx document retrieval system" → split into ["docx document", "retrieval system"] (remove meaningless word, keep domain term).

[0064] Exception handling: Empty input processing: If the user does not input keywords (string length is 0), trigger the front-end prompt "keywords cannot be empty" and re-receive input; Single character input processing: If the input is a single character (e.g. "check" "find"), automatically supplement the domain common collocation (e.g. "check" "retrieve"), form the candidate keyword sequence for user selection, avoid the expansion effect due to short input.

[0065] First-level expansion: Sentence-BERT semantic similarity expansion.

[0066] Choose the paraphrase-multilingual-MiniLM-L12-v2 model in the sentence-transformers library (version 2.2.2, installed through pip install sentence-transformers==2.2.2), which supports Chinese semantic encoding, has small parameter size (about 120MB) and fast inference speed (single sentence encoding <10ms); Candidate word source: Based on the "document retrieval domain corpus" (contains 5000 DOCX documents of purified word sequences, total vocabulary of 100,000 words) to build a candidate word pool to ensure that the candidate words are strongly related to the domain.

[0067] Second-level expansion: HowNet synonym knowledge base expansion.

[0068] Use HowNet 2024 official version (contains 1.5 million word semantic relationships, covering office documents, retrieval domain terms); Access method: Deploy HowNet dictionary file locally (path. / knowledge / HowNet_2024.dat), call the dictionary query interface through the pynetdicom library (version 2.0.2) to avoid network dependence.

[0069] Synonym query: For each word in the standardized keyword sequence, call HowNet.query_synonyms(word) to get explicit synonyms (e.g. "retrieve" → "find" "search", "DOCX document" → "DOCX file"). Effectiveness verification: Verification rule: count the frequency of the synonym in the "document retrieval field corpus", if the frequency ≥ 5 times (indicating that the word is commonly used in the field), it is determined as "effective synonym"; otherwise, it is removed (for example, the synonym "sought" of "search" appears only 2 times in the corpus, and is removed); Example: the synonym "DOCX file" of the keyword "DOCX document" appears 32 times in the corpus → determined as effective, added to the "secondary expansion set".

[0070] Third-level expansion: Word2Vec field word vector expansion.

[0071] Near-synonym screening logic: Default expansion quantity K: value is 5 (tested, K=5 can balance between "expansion breadth" and "precision", avoiding excessive expansion leading to redundancy); Execution steps: Call model.wv.most_similar(word, topn=K) to get the top K near-synonyms of the keyword token and the corresponding cosine similarity; If the near-synonym already exists in the primary / secondary expansion set, only update the highest similarity value; otherwise, add it to the "third-level expansion set".

[0072] De-duplication and merging logic: De-duplication rule: use "exact match of word string" for de-duplication (for example, "docx file" and "DOCX file" are considered the same word, as S41 has unified the lowercase, so the strings are directly compared); Merging strategy: merge the primary, secondary, and tertiary expansion sets into an "initial expansion set", keeping the highest semantic similarity value for each word (for example, a word exists in both the primary and secondary sets, take the higher similarity from the two).

[0073] Final expansion keyword list output: Output format: use a list to store, each element is a tuple (expanded keyword, weight value, original keyword), sorted in descending order of weight value (to facilitate the use of high-weight words first in subsequent matching); Example: the expansion list of the original keyword "docx document" is [("docx file", 0.895, "docx document"), ("word document", 0.836, "docx document"), ("office document", 0.752, "docx document")].

[0074] Then, step S5 includes: S51, parse the extended keyword list, construct a query expression tree according to the logical operators input by the user, wherein the logical operators include AND and OR, the leaf nodes of the query expression tree represent single keywords, the internal nodes represent logical operations, and the operator priority is ensured to be correct; S52, for each keyword node in the query expression tree, perform multi-mode matching in the inverted index, the multi-mode includes exact matching, fuzzy matching and synonym matching; the fuzzy matching uses edit distance similarity calculation: ; Wherein, represents the fuzzy matching similarity of the words and ; represents the edit distance; represents the character length of the words ; when , it is considered to be matched; is a preset fuzzy matching threshold; The synonym matching uses semantic similarity calculation: ; Wherein, represents the semantic similarity of the words and ; represents the word vector representation of the words ; when , it is considered to be matched; is a preset synonym matching threshold; S53, combine the matching results according to the logical operations of the query expression tree, take the intersection of paragraph sets for AND operation, take the union set for OR operation, and calculate the matching strength score of each paragraph: ; Wherein, represents the matching strength score of the paragraph p; r is the total number of extended keywords; is the weight of the extended keyword ; is an indicator function, which takes the value of 1 when the paragraph p contains the keyword , otherwise 0; finally, the paragraph set with a matching strength score higher than the corresponding preset threshold is selected as the initial matching paragraph set.

[0075] Logical operator rules: Supported operators: only include AND (logical and, high priority), OR (logical or, low priority), and the operators must be in uppercase (to avoid confusion with keywords, for example, "and" as a keyword); Priority rules: The priority of AND is higher than that of OR (e.g., "a AND b OR c" is equivalent to "(a AND b) OR c"); Input format: The user needs to input in the format of "keyword 1 operator keyword 2 [operator keyword 3...]", e.g., "DOCX document AND retrieve OR search"; if no operator is input, it is defaulted to be processed as OR (e.g., "DOCX document retrieve" is equivalent to "DOCX document OR retrieve").

[0076] Adopt the pyparsing library of Python (version 3.1.1, installed by pip install pyparsing==3.1.1), which supports customizing grammar rules and avoids the complexity of manual parsing; Expression tree structure: Leaf node: The data structure is LeafNode(keyword, weight), where keyword is the extended keyword (e.g., "docx file"), and weight is the weight assigned in step S4 (e.g., 0.94); Internal node: The data structure is InternalNode(op, left, right), where op is the logical operator (AND / OR), and left / right are the child nodes (which can be leaf nodes or other internal nodes); Example: For "DOCX document AND retrieve OR search", the expression tree structure is: InternalNode(op="OR",left=InternalNode(op="AND",left=LeafNode("docx document",0.895),right=LeafNode("retrieve",0.82)), right=LeafNode ("search",0.94)).

[0077] Load the list of (extended keyword, weight, original keyword) output by step S4 and construct a "keyword-weight" mapping dictionary (e.g., {"docx file":0.94,"retrieve":0.82}); User input parsing: Call pyparsing.Word(pyparsing.alphanums+"中文") to define the keyword grammar, and pyparsing.CaselessLiteral("AND")|pyparsing.CaselessLiteral("OR") to define the operator grammar; Grammar construction: query = pyparsing.OneOrMore(LeafNode_expr +(InternalNode_expr + LeafNode_expr)[...]), where LeafNode_expr matches extended keywords, InternalNode_expr matches operators; Exception handling: If the user inputs a word not in the extended keyword list (e.g. "invalid keyword"), prompt "Keyword not in the extended list, please re-enter"; If an invalid operator is input (e.g. "NOT" "XOR"), prompt "Only AND / OR operators are supported, please correct"; If the expression syntax is incorrect (e.g. "DOCX document AND OR retrieval"), prompt "Logical expression syntax error, please check operator position".

[0078] Exact match (priority execution): Core logic: directly compare whether the extended keyword and the keyword in the inverted index are exactly the same (since step S4 has been standardized to lowercase, there is no need for case adaptation); Execution steps: For the keyword e of the leaf node, check whether the key e exists in the inverted index inverted_index; If it exists, extract all paragraph number sets corresponding to the keyword P_exact = {p_j | (p_j,[...],...) ∈ I(e)}; If it does not exist, the exact match result is an empty set, and proceed to the subsequent synonym matching.

[0079] Use the python-Levenshtein library (version 0.21.1, installed through pip install python-Levenshtein==0.21.1), call Levenshtein.distance(w1, w2) to get the edit distance; Execution steps: Traverse all keywords w_idx in the inverted index; Calculate Sim_fuzzy(e, w_idx), if Sim_fuzzy≥θ_f, extract the paragraph number set P_fuzzy corresponding to w_idx; De-duplication: if P_fuzzy and P_exact have overlapping paragraphs, only keep one.

[0080] Word vector acquisition: call the Sentence-BERT model loaded in step S to encode the extended keyword e and the inverted index keyword w_idx, and obtain v_e and v_idx; Similarity calculation: calculate Sim_syn(e, w_idx) according to the formula, and if Sim_syn ≥ θ_s, extract the paragraph number set P_syn corresponding to w_idx; Matching order: first perform exact matching, then perform synonym matching, and finally perform fuzzy matching to avoid repeated processing (paragraphs that have been matched will not participate in subsequent pattern matching); Result merging: the final matching paragraph set of a single extended keyword is P_e = P_exact ∪ P_syn ∪ P_fuzzy.

[0081] Result merging of logical operations (intersection / union): Data structure: the paragraph set uses the set type of Python (supports efficient intersection and union operations); Merging rules: If the internal node is AND: merge the paragraph set P_left of the left child node and the paragraph set P_right of the right child node, and take the intersection P_merge = P_left ∩ P_right (only keep paragraphs that contain both keywords on both sides); If the internal node is OR: take the union P_merge = P_left ∪ P_right (keep paragraphs that contain any keyword on one side); Recursive merging: recursively perform merging from the leaf node to the root node of the expression tree, and finally obtain the total paragraph set P_total of the root node (all possible matching paragraphs).

[0082] Initial matching paragraph set screening Score calculation: traverse each paragraph p in P_total, and for all extended keywords , judge whether p is in , and calculate according to the formula; Screening rules: if , add p to the "initial matching paragraph set"; if , remove the paragraph (considered as insufficient correlation); Result storage: the initial matching paragraph set is stored in the format of "paragraph number-matching strength score-matched keyword list" (for example, {"p1": {"score":1.78, "matched_keywords":["docx file", "search"]}}), which is convenient for relevance sorting in subsequent step S6.

[0083] Finally, step S6 includes: S61, for each paragraph in the initial matching paragraph set, calculate an original relevance score based on keyword matching, which is calculated in combination with the term frequency of the keyword in the paragraph and the inverse document frequency, wherein the term frequency represents the number of occurrences of the keyword in the paragraph, and the inverse document frequency represents the rarity of the keyword in the entire document set; the original relevance score is calculated using the BM25 model, and the formula is: ; wherein, represents the original relevance score of the paragraph p; represents the i-th extended keyword, and r is the total number of extended keywords; represents the inverse document frequency of the keyword , and the calculation formula is , wherein M is the total number of documents in the document set; is the number of documents containing the keyword ; represents the term frequency of the keyword in the paragraph p; represents the length of the paragraph p; represents the average length of all paragraphs in the document set; and b are adjustable parameters, wherein controls the saturation of the term frequency, and is usually valued at 1.2 to 2.0, and b controls the influence of paragraph length normalization, and is usually valued at 0.5 to 0.8; S62, the original relevance score is subjected to paragraph length normalization processing to eliminate the bias of long paragraphs due to high term frequency, and to ensure the comparability of scores of paragraphs of different lengths; the normalization processing uses a length weight adjustment formula: ; wherein, represents the normalized relevance score; is a normalization intensity coefficient; represents an exponential function for smoothing the influence of length difference; S63, the normalized score is adjusted in combination with the structural position weight of the paragraph, wherein the weight of the title paragraph is higher than that of the body paragraph, and a final relevance score is calculated, and the ranking result is output in descending order of score; the final relevance score calculation formula is: ; wherein, represents the final relevance score of the paragraph p; is an indicator function, which takes a value of 1 when the paragraph p is classified as a title, and 0 otherwise; is a title weight addition coefficient; when ranking, the paragraphs are ranked according to The values ​​are sorted in descending order from highest to lowest to generate the final search results list.

[0084] IDF calculation: First, we need to calculate the sum of all documents in the document set, M (e.g., if a user uploads 100 documents, then M=100) and... (Traverse the S3 inverted index and count the number of contained elements) The number of document IDs), calculated using the Pythonmath.log function (natural logarithm); if (If the keyword does not appear in any document), then Taking 0.5 avoids the logarithm being meaningless; word frequency Keywords from the S3 inverted index The corresponding paragraph p Field retrieval (e.g.) =“docx file” in p=1 ); Paragraph length |p|: Extracted from the “paragraph ID-clean word sequence” mapping output by S2 (e.g., the clean word sequence length is 4 when p=1). Average length avgdl: Iterate through all paragraphs of all documents, sum up |p| and divide by the total number of paragraphs (for example, if there are 100 documents with a total of 1200 paragraphs and a total length of 6360, then avgdl=6360 / 1200=5.3).

[0085] For each initially matched paragraph p, substitute into the formula to calculate ; Sort in descending order: by Sort from high to low, if If they are equal, they will be sorted in ascending order by paragraph number (the paragraphs that appear earlier in the document will be displayed first). Output: Generate a structured list of "paragraph ID-final score-matched keywords-paragraph type" (e.g., {"para_id":"p1","final_score":4.29,"matched_keywords":["docx file","search"],"type":"h1"}) as input to S7.

[0086] Highlighting execution logic: Load the “Paragraph ID-Final Score-Matching Keywords” list output by S6 (e.g., the matching keywords for p1 are ["docx file", "retrieval", "search"]) and the “Paragraph ID-Original Text” mapping stored in S1 (e.g., the original text for p1 is “Keyword-based DOCX retrieval system”). Keyword positioning: call re.findall(keyword, text, flags=re.IGNORECASE) (ignore case) on the paragraph original text to locate the start / end position of each matching keyword in the text; Style embedding: If HTML format: replace the matching keyword with <mark style="background-color:#ffeb3b;color:#000;">Keywords< / mark> ; If TXT format: replace with

keyword

[0087] Matching sentence recognition and context expansion: 1. Matching sentence definition and recognition tool.

[0088] Matching sentence: refers to a complete sentence in the paragraph original text that "contains at least one expanded keyword (or its synonym / fuzzy matching word)"; Sentence segmentation tool: use the snownlp library (version 0.12.3, install through pip install snownlp==0.12.3), call SnowNLP(text).sentences to segment the paragraph text into independent sentences (support Chinese punctuation sentence segmentation, such as "。、;!?").

[0089] 2. Context expansion rules (based on user-configured maximum sentence number).

[0090] User configuration parameters: max_sentence_num (maximum number of summary sentences, value range 1-10 sentences, default 3 sentences, set through system interface or configuration file); Expansion logic: Mark all matching sentences (recorded as match_sent_idx, e.g. the 2nd sentence is a matching sentence, then match_sent_idx=[1], index starts from 0) for the paragraph segmented sentence list sent_list; Expand sentences before and after each matching sentence: If len(sent_list) ≤ max_sentence_num: extract all sentences as a summary; If len(sent_list) > max_sentence_num: expand by "take 1 sentence before + matching sentence + take 1 sentence after" (default 3 sentences). If the matching sentence is at the beginning / end of the sentence, expand backward / forward first (for example, if the matching sentence is the first sentence, then take "the first + second + third sentences"). Deduplication: If multiple matching sentences contain duplicate sentences after expansion, only one duplicate is retained.

[0091] Dynamic summary generation and output: Abstract format: Output in a structured format as "Paragraph ID - Highlighted Paragraph Text - Abstract Text - Final Score", where the abstract text must be marked with "Matching Sentences" (e.g., using "... (prefix) Example: Summary of p1 (max_sentence_num=3): The highlighted text reads: "A DOCX retrieval system based on keywords." Abstract text: A DOCX retrieval system based on keywords (the paragraph contains only one sentence, so it was extracted directly).

[0092] Export in HTML format (supports web browsing, retains highlighting styles): Template definition: Create a Jinja2 template file result_template.html and define a fixed structure: html Preview <!DOCTYPE html> <meta charset="UTF-8"> <title>DOCX search results< / title> <style>mark {background-color: #ffeb3b; color: #000;}< / style> <h1>Search results summary (total of {{total_para}} paragraphs)< / h1> {% for para in result_list %} <h3>Paragraph {{para.para_id}} (score: {{para.final_score|round(3)}})< / h3> <strong>Highlighted paragraph:< / strong> {{para.highlighted_text}} <strong>Dynamic summary:< / strong> {{para.summary}} {% endfor %} Data rendering: The "paragraph ID-highlighted text-summary-score" list output by S7 is used as result_list, passed to the template, and jinja2.Template.render() is called to generate complete HTML content; File saving: Call `with open("search results.html", "w", encoding="utf-8") as f:f.write(html_content)` to save to the user-specified path (the default path is `. / export / `).

[0093] Persistent storage and exception handling: Storage path: By default, the . / export / folder is created in the user's current working directory. If the folder does not exist, it will be created automatically by calling os.makedirs(". / export / "). File name rules: Name files in the format "Search Results_YYYYMMDD_HHMMSS" (e.g., "Search Results_20251022_153045.html") to avoid file overwriting; Error handling: If the user-specified path does not have write permissions (PermissionError), automatically switch to the system temporary directory (tempfile.gettempdir()) and prompt the user; if the exported file is too large (>100MB), prompt "The file size is large, it is recommended to export in batches".

[0094] In some embodiments, the system 200 for retrieving DOCX document content based on keywords may include multiple functional modules composed of computer program segments. The computer programs for each program segment in the system 200 for retrieving DOCX document content based on keywords may be stored in the memory of a computer device and executed by at least one processor to perform (see details). Figure 1 (Description) Functionality for retrieving DOCX document content based on keywords.

[0095] In this embodiment, the system 200 for retrieving DOCX document content based on keywords can be divided into multiple functional modules according to the functions it performs, such as... Figure 2 As shown. The functional modules may include: a document parsing module 210, a text preprocessing module 220, an inverted index construction module 230, a semantic expansion module 240, a multi-pattern matching module 250, a relevance calculation module 260, a result display module 270, and a result export module 280. The module referred to in this invention is a series of computer program segments that can be executed by at least one processor and perform a fixed function, stored in memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0096] Specifically, the document parsing module 210 is configured to load a DOCX file, traverse all paragraphs and table elements in the document, extract text content and style information of each paragraph, and classify the paragraphs into a title or a body according to at least one of the following features: a style name, a font size, and a bold attribute, to form structured text data; the text preprocessing module 220 is connected to the document parsing module and is configured to preprocess the structured text data, including removing non-English characters and redundant spaces, performing word segmentation processing on the text by using a word segmentation tool, and loading a stop word table to filter meaningless words, to obtain a purified word sequence; the inverted index construction module 230 is connected to the text preprocessing module and is configured to construct an inverted index based on the purified word sequence, and establish a one-to-one mapping relationship from a keyword to a position, wherein the position information at least includes a paragraph number and a paragraph internal offset; the semantic expansion module 240 is configured to receive a keyword input by a user, perform semantic analysis on the keyword by using a natural language processing model, obtain a synonym and / or near-synonym set of the original keyword, and form an expanded keyword list; the multi-mode matching module 250 is connected to the semantic expansion module and the inverted index construction module, and is configured to perform multi-mode matching in the inverted index according to the expanded keyword list, wherein the matching mode at least includes one of the following: exact matching, fuzzy matching, and synonym matching, to obtain an initial matching paragraph set; the relevance calculation module 260 is connected to the multi-mode matching module, and is configured to calculate a relevance score for each paragraph in the initial matching paragraph set, and generate a sorting result in descending order of the scores; the result display module 270 is connected to the relevance calculation module, and is configured to highlight the keyword in the corresponding paragraph in the sorting result, and extract a matching sentence and adjacent sentences before and after the matching sentence to generate an abstract, wherein the length of the abstract is dynamically determined according to a maximum sentence number configured by the user; and the result export module 280 is connected to the result display module, and is configured to export the highlighted paragraph and the abstract in an HTML, TXT or PDF format, to realize persistent storage of the retrieval result.

[0097] Figure 3 A structure schematic diagram of a terminal 300 is provided for the embodiment of the present application, and the terminal 300 can be used to execute the method for retrieving DOCX document content based on a keyword provided by the embodiment of the present application.

[0098] The terminal 300 can include a processor 310, a memory 320 and a communication module 330. These components communicate through one or more buses. Those skilled in the art can understand that the structure of the server shown in the figure does not constitute a limitation to the present application. It can be a bus structure or a star structure. It can also include more or fewer components than shown in the figure, or combine some components, or have a different component arrangement.

[0099] The memory 320 can be used to store the execution instructions of the processor 310, and the memory 320 can be implemented by any type of volatile or non-volatile storage terminal or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the terminal 300 can execute part or all of the steps in the following method embodiments.

[0100] The processor 310 is the control center of the storage terminal, connects various parts of the entire electronic terminal through various interfaces and lines, executes the software programs and / or modules stored in the memory 320, and calls the data stored in the memory, so as to execute various functions of the electronic terminal and / or process data. The processor can be composed of an integrated circuit (IC), for example, can be composed of a single packaged IC, or can be composed of a plurality of packaged ICs with the same function or different functions. For example, the processor 310 can only include a central processing unit (CPU). In the embodiments of the present application, the CPU can be a single operation core or can include multiple operation cores.

[0101] The communication module 330 is used to establish a communication channel, so that the storage terminal can communicate with other terminals. The communication module 330 receives user data sent by other terminals or sends user data to other terminals.

[0102] The present application also provides a computer storage medium, wherein the computer storage medium can store a program, and the program can include part or all of the steps in the embodiments provided by the present application when executed. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0103] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present application can be implemented by means of software plus necessary universal hardware platforms. Based on such an understanding, the technical solutions in the embodiments of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a USB flash disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and the like, and includes a plurality of instructions for causing a computer terminal (which can be a personal computer, a server, or a second terminal, a network terminal, or the like) to execute all or part of the steps of the methods described in the embodiments of the present application.

[0104] In the present specification, the same or similar parts among various embodiments can be referred to each other. In particular, for the terminal embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0105] In several embodiments provided in the present application, it should be understood that the disclosed system and method can be implemented in other ways. For example, the system embodiments described above are merely schematic, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed modules can be indirect coupling or communication connection through some interfaces, systems or modules, and can be electrical, mechanical or other forms.

[0106] The modules described as separate components can or can not be physically separated, and the components displayed as modules can or can not be physical modules, that is, they can be located in one place, or can be distributed on a plurality of network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0107] In addition, the functional modules in each of the embodiments of the present application can be integrated in one processing module, or each module can exist physically independently, or two or more modules can be integrated in one module.

[0108] Although the present application has been described in detail with reference to the preferred embodiments, it should be understood that the application is not limited to those preferred embodiments. Various modifications and equivalents can be made by those skilled in the art without departing from the spirit and scope of the application. Any and all modifications and equivalents are intended to be included within the scope of the present application.

Claims

1. A method for retrieving DOCX document content based on keywords, characterized in that, include: S1. Load the DOCX file through the document parsing module, traverse all paragraphs and tables, extract the text and style information of each paragraph, and classify the paragraphs into headings or body text based on at least one feature of style name, font size, and bold attribute to form structured text data. S2. Preprocess the structured text data: Remove non-Chinese and English characters and extra spaces, use a word segmentation tool to segment the remaining text, and load a stop word list to filter meaningless words to obtain a purified word sequence. S3. Construct an inverted index based on word sequences, establishing a one-to-one mapping relationship between keywords and their occurrence positions, where the position must include at least the paragraph number and the offset within the paragraph. S4. Receive keywords input by the user, use a natural language processing model to perform semantic expansion on the keywords, obtain a set of synonyms and / or near-synonyms of the original keywords, and form an expanded keyword list; S5. Perform multi-pattern matching in the inverted index based on the expanded keyword list. The multi-pattern matching includes at least one of exact matching, fuzzy matching, and synonym matching to obtain the initial set of matching paragraphs. S6. Calculate the relevance score for each paragraph in the initial set of matched paragraphs, and output the sorting results after sorting them in descending order of relevance score. S7. Highlight the keywords in the sorting results in the corresponding paragraphs, and extract the matching sentences and their adjacent sentences to generate a summary. The length of the summary is dynamically determined by the maximum number of sentences configured by the user. S8. Export the highlighted paragraphs and summaries as HTML, TXT, or PDF formats to persist the search results.

2. The method for retrieving DOCX document content based on keywords according to claim 1, characterized in that, Step S1 includes: S11. Load the DOCX file at the specified path through the document parsing module, use the python-docx library to parse the Office Open XML structure of the document, generate a document object model tree, which contains all paragraph elements and table elements, and ensures the integrity of the document structure. S12. Traverse each paragraph element in the document object model tree, extract the text content of the paragraph, and obtain the style attribute information of the paragraph. The style attribute information includes at least the style name, font size, and bold attribute. The style name is extracted from the name attribute of the paragraph style, the font size is extracted from the font size attribute, and the bold attribute is extracted from the font weight attribute. S13. Based on the extracted style attribute information, calculate the title classification score for each paragraph, and classify the paragraph into title or body text according to the comparison result between the score and the preset threshold. The formula for calculating the title classification score is as follows: ； in, This indicates the heading category score for paragraph p; It is an indicator function. When the style name of paragraph p The value is 1 if it belongs to the predefined heading style set T, and 0 otherwise. This indicates the font size of paragraph p, in points. This indicates the bold attribute of paragraph p. The value is 1 if the paragraph is bold, and 0 otherwise. This represents the base font size, which is a preset constant. , , These are the weighting coefficients; if If θ is the correct score, paragraph p will be classified as a heading; otherwise, it will be classified as body text, where θ is a preset score threshold.

3. The method for retrieving DOCX document content based on keywords according to claim 1, characterized in that, Step S2 includes: S21. Denoise the structured text data by using regular expression matching and character encoding recognition to remove non-Chinese and English characters and extra spaces, while retaining the valid text content. S22. Use a Chinese word segmentation tool to segment the denoised text, and calculate the optimal segmentation path through dictionary matching and hidden Markov model to divide the text into word sequences. S23. Load the stop word list to filter the word sequence, and calculate the threshold for removing meaningless words by combining word frequency statistics and semantic weights: ； in, Indicates the filtering weight of word w; N represents word frequency; N represents the total number of words. The document frequency containing the word 'w'; The total number of documents; when The word 'w' is filtered as meaningless. This is the preset filtering threshold.

4. The method for retrieving DOCX document content based on keywords according to claim 3, characterized in that, Step S3 includes: S31. Perform positional encoding on the purified word sequence, assigning a unique paragraph number and intra-paragraph offset to each word, and establishing a word positional mapping table; the positional encoding uses a piecewise linear mapping function: ； in, Words The position coordinates; p is the paragraph number; This is the offset within the paragraph; This represents the average paragraph length. A sequential index of words throughout the text; S32. Construct an inverted index based on a word position mapping table, using hash mapping and linked list structure to store the mapping relationship between keywords and their occurrence positions; the definition formula for an inverted index item is: ； in, This represents the inverted index item for the keyword t; Paragraph numbering; This is the offset within the paragraph; For keywords in position The frequency of occurrence of the keyword is m, where m is the total number of paragraphs containing the keyword, and n is the number of positions where the keyword appears in a single paragraph. S33. Compress and optimize the inverted index, and establish a multi-level caching mechanism to improve retrieval efficiency; the formula for calculating the index compression rate is: ； in, Indicates the index compression ratio; This is the size of the compressed index; The original index size is used; the cache hit rate optimization formula is: ； in, For cache hit rate; This represents the number of times the cache was hit. This represents the total number of visits.

5. The method for retrieving DOCX document content based on keywords according to claim 4, characterized in that, Step S4 includes: S41. Receive the keyword string input by the user, perform standardized preprocessing on the keywords, including converting the keywords into a uniform lowercase form, removing preset characters and extra spaces, and using a Chinese word segmentation tool to divide the compound keywords into individual word units to form a standardized keyword sequence. S42. Perform semantic analysis on the standardized keyword sequence using a pre-trained natural language processing model, and use a multi-level semantic expansion strategy to obtain the synonyms and near-synonyms for each keyword. The multi-level semantic expansion strategy includes the following calculation process: Level 1 Extension: Use the Sentence-BERT sentence encoding model to calculate the semantic similarity between keywords and candidate words. The formula for calculating semantic similarity is: ； Where A and B are vector representations of keywords and candidate words, respectively; The cosine similarity value; when Candidate words are added to the expanded set at that time; Set a preset similarity threshold; Second-level extension: Combine the HowNet thesaurus to obtain explicit synonyms for keywords and verify the effectiveness of the synonyms; Level 3 expansion: Using the Word2Vec word vector model trained on the domain corpus, calculate the cosine similarity between the keywords and candidate words, and select the top K words with the highest similarity as synonyms, where K is the preset expansion number; S43. The sets of synonyms and near-synonyms obtained from multi-level semantic expansion are deduplicated and merged to form an expanded keyword list, and a weight is assigned to each expanded keyword. The weight calculation formula is as follows: ； in, This indicates the weight of the extended keyword 'e'; The semantic similarity between the extended word 'e' and the original keyword 'k'; For expansion level coefficients; and These are the weighting coefficients.

6. The method for retrieving DOCX document content based on keywords according to claim 5, characterized in that, Step S5 includes: S51. Parse the extended keyword list and construct a query expression tree based on the logical operators input by the user. The logical operators include AND and OR. The leaf nodes of the query expression tree represent a single keyword, and the internal nodes represent logical operations. Ensure that the operator precedence is correct. S52. For each keyword node in the query expression tree, perform multi-pattern matching in the inverted index. The multi-pattern matching includes exact matching, fuzzy matching, and synonym matching; fuzzy matching uses edit distance similarity calculation. ； in, Words and Fuzzy matching similarity; Indicates edit distance; Words The length of the character; when This is considered a match; Preset fuzzy matching threshold; Synonym matching uses semantic similarity calculation: ； in, Words and semantic similarity; Words Word vector representation; when This is considered a match; Set a preset synonym matching threshold; S53. Merge the matching results based on the logical operations of the query expression tree. For AND operations, take the intersection of the paragraph sets; for OR operations, take the union of the paragraph sets; and calculate the matching strength score for each paragraph. ； in, This represents the matching strength score of paragraph p; r is the total number of extended keywords; To expand keywords The weights; As an indicator function, when paragraph p contains keywords The value is 1 when the match is strong and 0 otherwise; the final selection is based on a match strength score higher than the corresponding preset threshold. The paragraphs form the initial set of matching paragraphs.

7. The method for retrieving DOCX document content based on keywords according to claim 6, characterized in that, Step S6 includes: S61. For each paragraph in the initial set of matched paragraphs, calculate the raw relevance score based on keyword matching. This score is calculated by combining the term frequency and inverse document frequency of the keyword in the paragraph, where term frequency represents the number of times the keyword appears in the paragraph, and inverse document frequency represents the rarity of the keyword in the entire document set. The raw relevance score is calculated using the BM25 model, with the following formula: ； in, This represents the original relevance score of paragraph p; This represents the i-th extended keyword, where r is the total number of extended keywords; Keywords The inverse document frequency is calculated using the following formula: , where M is the total number of documents in the document collection; For keywords The number of documents; Keywords Word frequency in paragraph p; Indicates the length of paragraph p; This represents the average length of all paragraphs in the document collection; and b are adjustable parameters, where The saturation value for controlling word frequency is typically set to 1.2 to 2.0, while b controls the effect of paragraph length normalization, typically set to 0.5 to 0.

8. S62. The original relevance scores are normalized by paragraph length to eliminate bias caused by higher word frequencies in long paragraphs and ensure the comparability of scores for paragraphs of different lengths. The normalization process uses a length-weighted adjustment formula: ； in, This represents the normalized correlation score; This is the normalized intensity coefficient; This represents an exponential function used to smooth out the effects of length differences. S63. Adjust the normalized score based on the structural position weight of the paragraphs, with the title paragraph having a higher weight than the body paragraphs. Calculate the final relevance score and output the results in descending order of score. The formula for calculating the final relevance score is: ； in, This represents the final relevance score of paragraph p; This is an indicator function that takes a value of 1 when paragraph p is classified as a heading, and 0 otherwise; Add a weighting factor to the title; when sorting, paragraphs will be ranked according to... The values ​​are sorted in descending order from highest to lowest to generate the final search results list.

8. A system for retrieving DOCX document content based on keywords, characterized in that, include: The document parsing module is used to load DOCX files, traverse all paragraphs and table elements in the document, extract the text content and style information of each paragraph, and classify the paragraphs into headings or body text based on at least one feature of style name, font size, and bold attribute, forming structured text data. The text preprocessing module, connected to the document parsing module, is used to preprocess structured text data, including removing non-Chinese and English characters and extra spaces, using a word segmentation tool to segment the text, and loading a stop word list to filter meaningless words, resulting in a purified word sequence. The inverted index building module, connected to the text preprocessing module, is used to build an inverted index based on the purified word sequence and establish a one-to-one mapping relationship between keywords and their occurrence positions. The position information includes at least the paragraph number and the offset within the paragraph. The semantic extension module is used to receive keywords input by the user, perform semantic analysis on the keywords using a natural language processing model, obtain a set of synonyms and / or near-synonyms of the original keywords, and form an extended keyword list; The multi-pattern matching module, connected to the semantic extension module and the inverted index construction module, is used to perform multi-pattern matching in the inverted index based on the extended keyword list. The matching pattern includes at least one of exact matching, fuzzy matching and synonym matching to obtain an initial set of matching paragraphs. The relevance calculation module, connected to the multi-pattern matching module, is used to calculate the relevance score for each paragraph in the initial set of matched paragraphs and generate sorted results in descending order of the scores. The results display module, connected to the relevance calculation module, is used to highlight keywords in the ranking results in the corresponding paragraphs and extract matching sentences and their adjacent sentences to generate a summary. The length of the summary is dynamically determined by the maximum number of sentences configured by the user. The results export module, connected to the results display module, is used to export highlighted paragraphs and summaries as HTML, TXT, or PDF formats, enabling persistent storage of search results.

9. A terminal, characterized in that, include: processor; Memory used to store the processor's execution instructions; The processor is configured to perform the method according to any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images