Automatic indexing method and system based on large language model

Abstract

The invention relates to the technical field of data processing, and discloses an automatic indexing method and system based on a large language model. The method comprises the following steps: carrying out structured analysis on a document to obtain text data and page mapping, obtaining a word list matching entry set by utilizing word list forward matching, inputting an uncovered text interval into a large language model to extract new entries, and normalizing to obtain a model extraction entry set, the two sets are fused, a comprehensive weight value is calculated according to the word frequency value, the inverse document frequency value, the hierarchy weighting coefficient and the position weighting coefficient, the first K entries are screened, sorting is conducted according to pinyin, page numbers are added, and an indexing document is generated. According to the method, the normalization, the coverage integrity and the weight calculation accuracy of the index entries are improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an automatic indexing method and system based on a large language model. Background Technology

[0002] Indexing dissertations is a crucial task in library information organization. Traditional indexing relies on manual methods, where professional indexers read the full text of the dissertation, manually extract index terms based on a standardized thesaurus, and number the pages. This requires indexers to possess solid subject knowledge and indexing experience, accurately identifying core concepts, technical terms, research methods, and other valuable vocabulary from the dissertation, and standardizing them into index terms conforming to the thesaurus standards. In recent years, with the development of automatic indexing technology, researchers have attempted to use statistical methods such as TF-IDF and TextRank to automatically extract keywords from text as index terms. These methods assess the importance of words by calculating their frequency of occurrence and distribution within a document set, significantly improving processing efficiency compared to manual indexing. Furthermore, rule-based automatic indexing methods compare the dissertation text word-by-word with a professional thesaurus, identifying thesaurus terms appearing in the text as index terms. This method ensures the standardization and authority of the extracted terms, avoiding the non-standard terminology issues that arise with statistical methods.

[0003] However, existing automatic indexing technologies have many shortcomings. While statistical keyword extraction methods can identify high-frequency words, they cannot distinguish between specialized terms and common high-frequency words. This results in a large number of useless keywords such as "research," "methods," and "analysis" mixed in with the extracted index entries. Furthermore, this method relies on word frequency statistics and lacks semantic understanding, often failing to identify emerging terms and core concepts that appear less frequently in the text but have significant academic value, leading to incomplete index coverage. Automatic indexing methods based on thesaurus matching, while ensuring the standardization of entries, face the problem of lagging thesaurus updates. New theoretical concepts and technical terms constantly emerge in the academic research field, and these new terms have not yet been included in authoritative thesauruses. This causes thesaurus matching methods to fail to identify innovative content in papers, resulting in significant omissions in the indexing results. Furthermore, existing methods lack the ability to perceive document structure when processing long dissertations, mixing texts from different logical levels such as titles, abstracts, main text, and references together for processing. They fail to distinguish the differences in importance of terms in different positions. Terms appearing in titles and abstracts are obviously more reflective of the core theme of the paper than terms in references, but existing methods assign the same weight to terms in all positions, resulting in a decline in index quality. Summary of the Invention

[0004] This application provides an automatic indexing method and system based on a large language model, which solves the problems of existing automatic indexing technologies being unable to integrate vocabulary standardization and semantic understanding capabilities, and unable to comprehensively consider statistical features and knowledge features when calculating term weights. By combining a hybrid indexing mechanism of vocabulary coarse screening and large language model refinement with the TWTF four-dimensional fusion scoring algorithm, the standardization, coverage completeness and weight calculation accuracy of indexed terms are improved.

[0005] Firstly, this application provides an automatic indexing method based on a large language model, the automatic indexing method based on a large language model comprising: Step S1: Perform structured parsing on the document to be indexed, dividing the document content into heading area, abstract area and chapter area according to logical hierarchy, and obtaining structured text data and page number mapping relationship of each area; Step S2: Perform a forward matching scan on the structured text data using the vocabulary, identify terms that are completely consistent with the vocabulary and record their page number positions to obtain a set of vocabulary matching terms; Step S3: Extract text intervals not covered by the word list matching term set from the structured text data, input the text intervals along with the preceding and following context into the large language model for semantic analysis, and output newly identified terms from the large language model and perform similarity matching with the word list to complete the normalization, thereby obtaining the term set extracted by the model; Step S4: Merge the term matching set and the term extraction set from the model to form a candidate term set. Calculate the term frequency, inverse document frequency, level weighting coefficient, and position weighting coefficient for each candidate term based on its frequency of occurrence in the document, its distribution within the document, its term hierarchy attribute, and its position type. Multiply the four parameters to obtain the comprehensive weight value for each term. Sort the terms in descending order of weight value and select the top K terms to obtain the target indexing term set. Step S5: Reorder the target index term set according to the alphabetical order of the term pinyin, attach the page number string corresponding to each term to each term, and generate a structured index document.

[0006] Secondly, this application provides an automatic indexing system based on a large language model, the automatic indexing system based on a large language model comprising: The parsing module is used to perform structured parsing of the document to be indexed, dividing the document content into title area, summary area and chapter area according to logical hierarchy, and obtaining structured text data and page number mapping relationship of each area; The matching module is used to perform a forward matching scan on the structured text data using a vocabulary list, identify terms that are completely consistent with the vocabulary list and record their page number positions, thereby obtaining a set of vocabulary list matching terms. The analysis module is used to extract text intervals from the structured text data that are not covered by the vocabulary matching term set, input the text intervals along with the preceding and following context into the large language model for semantic analysis, and output newly identified terms from the large language model and perform similarity matching with the vocabulary to complete the normalization, thereby obtaining the term set extracted by the model. The calculation module is used to merge the vocabulary matching term set and the term set extracted by the model to form a candidate term set. Based on the frequency of each candidate term in the document, the distribution of the document, the vocabulary hierarchy attribute and the position type, the term frequency value, the inverse document frequency value, the hierarchy weighting coefficient and the position weighting coefficient are calculated respectively. The four parameters are multiplied to obtain the comprehensive weight value of each term. After sorting the terms in descending order of weight value, the top K terms are selected to obtain the target indexing term set. The sorting module is used to reorder the target index term set according to the alphabetical order of the term pinyin, attach the page number string corresponding to each term to each term, and generate a structured indexing document.

[0007] Thirdly, an automatic indexing device based on a large language model is provided, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor invokes the instructions in the memory to cause the automatic indexing device based on a large language model to execute the aforementioned automatic indexing method based on a large language model.

[0008] Fourthly, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform the above-described automatic indexing method based on a large language model.

[0009] The technical solution provided in this application solves the problem of existing technologies treating paper text as a flat character stream, which makes it impossible to distinguish the importance of terms in different positions, by dividing the document content into title area, abstract area and chapter area according to logical hierarchy and establishing page number mapping relationship through a structured parsing step. This enables the subsequent indexing process to assign differentiated weight scores to terms at different levels based on the document structure. Terms in the title and abstract receive higher position weighting coefficients due to their high degree of summarization of the paper's topic, thus occupying a higher position in the final index ranking. This structure awareness capability significantly improves the matching degree between the index results and the core content of the paper. The step of performing forward matching scans on structured text data using a professional thesaurus achieves efficient thesaurus retrieval through a Trie tree index structure. Compared to the linear search method of traversing the thesaurus record by record, the prefix compression characteristic of the Trie tree reduces the matching time complexity from O(n×m) to O(m), where n is the size of the thesaurus and m is the length of the text. This efficiency improvement is particularly significant when dealing with large-scale thesaurus containing millions of entries. At the same time, thesaurus matching ensures that the extracted entries fully conform to the standard expressions of authoritative thesaurus, avoiding non-standard problems such as word form changes and synonyms caused by statistical methods, and providing a unified vocabulary benchmark for subsequent cross-document retrieval. The step of extracting text regions not covered by the vocabulary matching term set and inputting them into a large language model for semantic analysis uses difference operations to accurately locate text content that the vocabulary cannot cover. This selective processing mechanism allows the large language model to process only about 20% to 30% of the difficult segments in the whole text. Compared with the solution of inputting the whole text into the model indiscriminately, the token consumption is reduced by more than 70%, significantly reducing API call costs. More importantly, by providing 500 characters of context before and after the uncovered segments, the large language model's ability to understand the semantics of the segments is enhanced, enabling it to accurately identify emerging terms and implicit concepts, making up for the shortcomings of vocabulary update lag. Furthermore, the edit distance algorithm is used to perform similarity matching between the output terms of the model and the vocabulary to complete the normalization process. This not only preserves the model's ability to discover new terms, but also ensures the normalization of the output terms, achieving an organic integration of semantic understanding and vocabulary standardization.

[0010] The process involves merging a vocabulary matching set and a model-extracted vocabulary set, and calculating a comprehensive weight value based on frequency of occurrence, document distribution, vocabulary level, and position type. The TWTF algorithm is used to achieve a four-dimensional fusion scoring of statistical, knowledge, structural, and semantic features. The term frequency and inverse document frequency reflect the statistical importance of the term within the document. The hierarchical weighting coefficient reflects the specificity of the term within the subject knowledge system. Third-level specific terms receive a weighting coefficient of 1.5 due to their strong retrieval targeting, while first-level subject categories are penalized with a coefficient of 0.6 due to their broadness. The position weighting coefficient reflects the term's positional weighting. Strategically positioned within the document structure, heading entries receive a weight of 2.0 due to their highly condensed representation of the paper's theme, while body text entries maintain a baseline weight of 1.0. The multiplication of these four parameters allows the comprehensive weight value to fully measure the indexing value of each entry, avoiding the problem of high-frequency function words ranking higher due to reliance solely on word frequency in existing technologies. It also solves the deficiency that statistics alone cannot reflect the patterns of subject knowledge. The dynamic filtering mechanism, which calculates the retention threshold K based on the ratio of the total number of words in the document to the preset density, ensures that the index density is adapted to the document length, avoiding the problems of overly dense indexing of short documents and overly sparse indexing of long documents. The steps involved reordering the target index term set according to alphabetical order using pinyin and appending page number strings to generate a structured index document. Pinyin sorting ensured an alphabetical arrangement that conformed to national indexing standards, while tone-based secondary sorting guaranteed a deterministic order for homophones. Continuous page number compression converted the page number sequence into a start-end format, saving storage space in the index document and facilitating quick location of concentrated term distribution areas for readers. The encapsulation of structured data objects supports multiple output formats to meet the needs of different application scenarios, and fully preserves all attribute information such as term text, category, page number, and weight. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of an embodiment of the automatic indexing method based on a large language model in this application. Figure 2 This is a schematic diagram of one embodiment of the automatic indexing system based on a large language model in this application. Figure 3 This is a schematic block diagram of the automatic indexing device based on a large language model in an embodiment of the present invention. Detailed Implementation

[0013] This application provides an automatic indexing method and system based on a large language model. The terms first, second, third, fourth, etc. (if present) in the specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms include or have, and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0014] For ease of understanding, the specific process of the embodiments of this application is described below. Please refer to [link / reference]. Figure 1 One embodiment of the automatic indexing method based on a large language model in this application includes: Step S1: Perform structured parsing on the document to be indexed, dividing the document content into heading area, abstract area and chapter area according to logical hierarchy, and obtaining structured text data and page number mapping relationship of each area; The process involves using a document parsing library to identify the format and extract content from the thesis files, parsing the binary document stream into a plain text stream. Simultaneously, information such as the thesis title, author name, university name, and total page count are extracted from the document metadata and stored in the database. The parsed plain text stream is then used to identify chapter boundary markers in the document through regular expression pattern matching, matching feature strings such as Chapter X, abstract, and keywords to locate the start and end positions of each logical level. Each logical level is assigned a unique structure identifier, recording the start and end character offsets of that level in the original text. Page number mapping is achieved by calculating the relative position of each level's text within the full text; the starting character offset of a level is divided by the total number of characters in the full text and then multiplied by the total number of pages to obtain the starting page number of that level. All structured information is encapsulated as a JSON-formatted data object array, with each element corresponding to a logical level and containing a structure identifier, text content, starting page number, and ending page number.

[0015] Step S2: Perform a forward matching scan on the structured text data using the vocabulary, identify terms that are completely consistent with the vocabulary and record their page number positions to obtain a set of vocabulary matching terms; The process involves batch loading terminology records from a subject-specific thesaurus database. Each record contains the terminology text, thesaurus hierarchical attributes, and terminology category. The thesaurus hierarchy is divided into three levels: the first level represents the broad subject category, the second level represents the professional field, and the third level represents specific terms. A Trie tree index structure is constructed for each term to accelerate matching and searching. Each node in the Trie tree stores a single Chinese character, and the path from the root node to a leaf node constitutes the complete term. The leaf nodes store the terminology identifier and hierarchical attributes. Forward matching scanning employs the maximum matching algorithm. The matching pointer and window length are initialized, and a substring is extracted from the matching pointer position to perform a character-by-character path search in the Trie tree. When a match is successful, the terminology text, identifier, and hierarchical attributes are recorded. Calculating the page number of the term requires reading the character offset of the matching pointer. The hierarchy and starting page number of this offset are retrieved through the page number mapping relationship. The relative offset within the hierarchy is calculated and divided by the average number of characters per page to obtain the relative page number offset. This relative page number offset is then added to the starting page number of the hierarchy to obtain the absolute page number value. After a successful match, the pointer moves forward by the character length of the term to continue matching; if a match fails, the window size is reduced and the matching is retried. After traversing all chapter texts, the page number value list for each term is sorted in ascending order, and continuous page number compression is performed. The continuous page number segments are converted into start-end format to obtain the set of matched terms and page number positions.

[0016] Step S3: Extract text intervals from the structured text data that are not covered by the vocabulary matching term set, input the text intervals along with the preceding and following context into the large language model for semantic analysis, and have the large language model output newly identified terms and perform similarity matching with the vocabulary to complete the normalization, thus obtaining the term set extracted by the model; The process involves calculating uncovered text intervals based on a vocabulary matching set. The character offset interval of each term in the document is read, and all intervals are sorted in ascending order of their starting positions. An interval merging algorithm is then executed; if the starting position of the current interval is less than or equal to the ending position of the previous interval, they are merged into one large interval. After merging, a list of covered intervals is obtained. The total number of document text intervals is then compared with the list of covered intervals to extract uncovered text intervals. The start and end positions of each uncovered interval are recorded to obtain a set of uncovered text fragments. Each text fragment is read from this set, and 500 characters before and after it are extracted as context and concatenated to obtain the input text. This input text is combined with a pre-defined term extraction hint template and sent to the large language model via an API interface. The hint template includes task instructions, output format requirements, and vocabulary hierarchy constraints, guiding the model to extract index terms from the text fragments and output JSON format data. The system receives JSON-formatted term data returned by the large language model, parses the text content and category attributes of each term, performs similarity matching with terms in the vocabulary database using the edit distance algorithm, calculates the number of character editing operations between the input term and the vocabulary term, and determines the term as a similar term when the edit distance is less than or equal to a preset threshold. The system then replaces the original term text with a standardized term from the vocabulary database and inherits the vocabulary hierarchy attributes, thus completing the standardization process and obtaining the term set extracted by the model.

[0017] Step S4: Combine the term matching set and the term extraction set from the model to form a candidate term set. Calculate the term frequency, inverse document frequency, level weighting coefficient, and position weighting coefficient for each candidate term based on its frequency of occurrence in the document, its distribution within the document, its term hierarchy attribute, and its position type. Multiply these four parameters to obtain the comprehensive weight value for each term. Sort the terms in descending order of weight value and select the top K terms to obtain the target indexing term set. The process involves several steps: First, the term set matched to the thesaurus and the term set extracted by the model are deduplicated and merged. For records with identical term text, the one with the higher confidence score is retained, resulting in a candidate term set. The candidate term set is then iterated through to count the frequency of each term in the structured text data. This frequency is divided by the total number of words in the document to obtain the term frequency value. Next, the number of chapters containing the term is counted. This number is divided by the total number of chapters containing the term, incremented by one, and the logarithm is taken to obtain the inverse document frequency value. This incrementing operation avoids cases where the denominator is zero. Based on the term's thesaurus hierarchy attribute, the corresponding hierarchy weighting coefficient is retrieved from a pre-defined three-level weight matrix. This three-level weight matrix defines the weight differences between terms at different levels. Finally, based on the position type of each term in the structured text data, the corresponding position weighting coefficient is retrieved from a pre-defined position weight matrix. Position types include title areas, abstract areas, chapter titles, and body text. The position weight matrix assigns different importance weights to different position types. The term frequency, inverse document frequency, hierarchical weighting coefficient, and positional weighting coefficient of each term are multiplied together to obtain a comprehensive weight value. The candidate term set is then sorted in descending order according to the comprehensive weight value. The retention threshold K is calculated based on the ratio of the total number of words in the document to a preset density ratio. The density ratio defines the standard for how many words to retain one index term. The K terms with the highest weight ranking are selected to obtain the target index term set.

[0018] Step S5: Reorder the target index term set according to the alphabetical order of the term's pinyin, attach the corresponding page number string to each term, and generate a structured index document.

[0019] The process involves several steps: First, the text of each term in the target indexing term set is converted using a pinyin conversion library to generate corresponding pinyin strings, which serve as sorting keywords. The target indexing term set is then sorted lexicographically in ascending order by pinyin strings. When pinyin is the same, a secondary sort is performed based on tone values ​​to obtain a pinyin-ordered term list. Next, the page number values ​​of each term are read from the pinyin-ordered term list and sorted in ascending order. The sorted page number list is then processed for continuous page number recognition and compression, converting continuous page number segments into start-end page number formats (e.g., page number sequence 3, 4, 5, 6 is converted to 3-6). Non-continuous page numbers remain independent, resulting in the page number strings for each term. Finally, the text content, term category, page number strings, and comprehensive weight values ​​of each term in the pinyin-ordered term list are encapsulated into structured data objects. These objects are then used to generate a structured indexing document containing a term index table and metadata information, based on a preset index document format template. The document format supports plain text, rich text, and JSON output formats to meet the needs of different application scenarios.

[0020] In one specific embodiment, step S1 includes: The document parsing library is called to perform format recognition on the document to be indexed, convert the document binary stream into a plain text stream and extract the document title, author and page number information to obtain the document metadata and full text content; By matching the feature patterns of chapter titles in the full text content using regular expressions, the text boundaries of the title layer, abstract layer, keyword layer and each chapter layer are identified, and a unique structural identifier is assigned to each level. Based on the character offset of each level of text in the full text content, the starting page number and ending page number of each level are calculated in combination with the total number of pages and the total number of characters in the document. The structure identifiers, text content, start page number, and end page number of each level are encapsulated into an array of JSON objects to obtain structured text data and the page number mapping relationship of each area.

[0021] Specifically, the format recognition of the thesis documents is performed using either the Apache PDFBox or Apache POI library. For PDF documents, the PDFBox PDDocument class is called to load the file and extract the text content page by page. For Word documents, the POI XWPFDocument class is called to parse the document structure and read the paragraph text. The format conversion process decodes the document's binary byte stream into a Unicode-encoded plain text character stream. At the same time, the thesis title field, author name field, and total page number field are extracted from the document attribute object. This metadata is written to the document information table in the database as a retrieval index, and the plain text character stream is used as the full-text content for subsequent structured segmentation processing.

[0022] Regular expression matching employs pattern recognition technology to locate chapter boundaries in the full-text content. For the title level, the matching pattern is the first line of large font or the first line of bold text. For the abstract level, the matching pattern is consecutive paragraphs following the abstract keywords until the keywords are encountered or the "Chapter 1" marker is found. For the keyword level, the matching pattern is a list of phrases separated by semicolons or commas after the keyword marker. For each chapter level, the matching pattern is "Chapter 1," "Chapter 2," or chapter numbers such as 1., 2., etc. After matching a level boundary, the starting and ending character positions of that level in the full-text content are recorded. The starting position is the index value of the matched marker character in the text stream, and the ending position is the starting position of the next level minus one or the end of the text stream. Each identified level is assigned a structural identifier using a naming rule of level type plus sequence number. For example, the identifier for the title level is title_1, the identifier for the abstract level is abstract_1, the identifier for Chapter 1 is chapter_1, and the identifier for Chapter 2 is chapter_2. These identifiers are used as unique locating keys in subsequent data references.

[0023] The starting and ending page numbers for each level are calculated using the mapping relationship between character offsets and page numbers. The calculation logic is as follows: divide the starting character offset of a level by the total number of characters in the full text to obtain the relative position ratio of that level in the document; multiply this ratio by the total number of pages in the document to obtain a floating-point value for the starting page number; and then round up to obtain the integer value for the starting page number. For example, if the starting character offset of a chapter is 12500, the total number of characters in the full text is 50000, and the total number of pages in the document is 100, then the relative position ratio is 12500 divided by 50000, which equals 0.25; multiplying this by 100 gives 25.0; and rounding up gives the starting page number as 25. The ending page number is calculated using the same method: divide the ending character offset by the total number of characters in the full text, multiply by the total number of pages in the document, and then round down to obtain the integer value for the ending page number. This calculation method is based on the assumption that characters are evenly distributed in the document. In practical applications, statistical verification shows that the error is within 1 to 2 pages, which meets the accuracy requirements for indexing.

[0024] The JSON object array is encapsulated using a nested data structure to represent the complete information of each level. Each element of the array is a JSON object containing a `structure_id` field storing the structure identifier string, a `level_type` field storing the level type enumeration value, a `text_content` field storing the plain text content of that level, a `start_page` field storing the starting page number integer value, a `end_page` field storing the ending page number integer value, a `start_offset` field storing the starting character offset, and an `end_offset` field storing the ending character offset. All level objects are arranged in document order to form the JSON array. The first element of the array corresponds to the heading level, the second element to the summary level, the third element to the keyword level, and subsequent elements correspond to the chapter levels in sequence. This JSON array is stored as structured text data in a cache table in the database. Simultaneously, the starting and ending page numbers of each level form a page number mapping table, indexed using the structure identifier as the primary key. Subsequent steps utilize this table to quickly convert character offsets to page number values.

[0025] In one specific embodiment, step S2 includes: Load term records from the subject thesaurus database, and construct a Trie tree index structure from the text content, thesaurus hierarchical attributes, and term category of each term. Each node of the tree stores a Chinese character. Initialize the matching pointer position and the matching window length, extract a substring with a length equal to the matching window length from the beginning position of the chapter text in the structured text data, and perform a path lookup in the Trie tree index structure to determine if a complete match exists; When a match is successful, the text content, term identifier, and term hierarchy attributes of the matched term are recorded. Based on the character offset of the matching pointer position in the structured text data and the page number mapping relationship, the page number value of the term is calculated. The matching pointer position is then moved forward by the character length of the matched term and the matching continues. After traversing all chapters of the structured text data to complete the matching scan, the page number value list of each term is sorted in ascending order and compressed into consecutive page numbers to generate a page number string, thus obtaining the set of matched terms and their corresponding page number positions.

[0026] Specifically, batch queries are performed from the term table in the subject-specific thesaurus database. SQL statements are used to select all records in the term text field, thesaurus hierarchy field, and term category field, loading them into memory. Each thesaurus record contains complete term attribute information. The construction process of the Trie tree index structure involves traversing the loaded term list, splitting the text content of each term character by character, and creating or searching for child nodes along the character sequence starting from the root node of the Trie tree. Each node corresponds to a Chinese character and stores a pointer to the next character node. For example, the insertion process for the term "deep learning" involves first creating or searching for the deep byte node under the root node, then creating or searching for the degree byte node under the deep byte node, and so on until the end byte node. The end byte node is marked as a leaf node and stores the term's identifier, thesaurus hierarchy attribute, and term category information. When multiple terms share the same prefix character, existing node paths are reused. For example, "deep learning" and "deep neural network" share the node path for the two characters "deep." This prefix compression structure reduces memory usage and accelerates the matching and searching speed.

[0027] The matching pointer position is initialized to zero to indicate that scanning starts from the first character of the chapter text, and the matching window length is initialized to fifteen to attempt matching by extracting fifteen Chinese characters at a time. The plain text content of a chapter object is obtained from the `text_content` field of the structured text data JSON array. A string slicing function is used to extract a substring of length equal to the matching window length, starting from the matching pointer position. When performing path lookup in the Trie tree index structure, characters of the substring are read one by one starting from the root node. A node matching the character is searched in the child node list of the current node. If a match is found, the search continues to the next character in that child node; otherwise, the match fails and an empty result is returned. A complete match is determined when all characters of the substring are successfully traversed and the finally reached node is marked as a leaf node. At this point, the stored term identifier, term hierarchy attributes, and term category information are read from the leaf node as the matching result.

[0028] After a successful match, the recorded term text is reconstructed directly from the Trie tree path. All characters along the path from the root node to the leaf node are concatenated sequentially to obtain the complete term text. The term identifier and term hierarchy attribute are read from the attribute fields of the leaf nodes. The term identifier is used to uniquely identify the term in subsequent steps, and the term hierarchy attribute indicates whether the term belongs to a first-level, second-level, or third-level category. Calculating the page number where the term appears requires reading the value of the current matching pointer position. This value represents the relative character offset of the term's first character in the chapter text. Adding this relative offset to the chapter's starting character offset in the full text yields the term's absolute character offset in the full text. By querying the page number mapping table, the hierarchical structure identifier to which the term belongs is found based on the absolute character offset. The starting page number and starting character offset of that level are read, and the offset difference of the term relative to the starting position of the level is calculated. This difference is divided by the average number of characters per page of that level to obtain the relative page number offset. The relative page number offset is rounded down and then added to the starting page number of the level to obtain the absolute page number value where the term appears. The step size of the matching pointer position moving forward is equal to the length of the successfully matched term characters. For example, after matching four characters of "deep learning", the pointer position is increased by four, the window length is reset to fifteen, and the next round of matching continues from the new position. If the match fails, the window length is decreased by one to fourteen, and a substring is extracted to try to match again. The window length is decreased sequentially until the window length is one and there is still no match. Then, the matching pointer position is moved forward one character and the window length is reset to fifteen. This sliding window mechanism traverses each character position of the chapter text.

[0029] After traversing all chapter objects in the structured text data, each matched term accumulates a list of page number values, recording the page numbers of all occurrences of that term in the document. The page number value list is then sorted in ascending order, arranging the page numbers in ascending sequence. This sorted page number sequence facilitates the identification of consecutive page number segments. The logic for continuous page number compression is to traverse the sorted page number list, comparing the difference between two adjacent page number values. If the difference equals one, the page numbers are consecutive, and the starting and ending page numbers of the consecutive page number segment are recorded. If the difference is greater than one, the page numbers are not consecutive, the compression result of the previous segment is output, and a new page number segment is started. For example, the page number sequence 3, 4, 5, 6, 8, 10, 11 is compressed to generate the page number strings 3-6, 8, 10-11. The consecutive segments 3-6 and 10-11 are represented using the start and end page number format, while the non-consecutive page number 8 remains in its independent form. The text content, term identifier, term hierarchy attribute, and page number string of all terms are combined to form a term matching term set. This set is stored in the index result table of the database with the term identifier as the primary key, and the page number string is used as the page number position information field of the term for subsequent queries.

[0030] In one specific embodiment, the page number value of the word's occurrence is calculated based on the character offset of the matching pointer position in the structured text data and the page number mapping relationship. The matching pointer position is then moved forward by the character length of the matching word before continuing the matching process. This includes: Read the character offset of the matched term in the structured text data, and query the logical hierarchy identifier to which the character offset belongs and the starting page number of that hierarchy; The relative offset of the term within that level is calculated by taking the difference between the character offset and the starting character offset of that level. Calculate the average number of characters per page based on the total number of characters and page number span of the level. Divide the relative offset by the average number of characters per page and round down to obtain the relative page number offset of the entry within the level. The starting page number of the level is added to the relative page number offset to obtain the absolute page number value where the term appears. The matching pointer position is then increased by the character length value of the matching term, the matching window length is reset, and the matching process continues.

[0031] Specifically, the processed character offset is the relative position of the current matching pointer within the chapter text. This needs to be converted to the absolute character offset of the term within the full text before page number location can be performed. The conversion method involves reading the `start_offset` field value of the chapter object from the JSON array of the structured text data as the chapter's starting offset, and adding the matching pointer position value to the chapter's starting offset to obtain the term's absolute offset within the full text. The page number mapping table is queried using an interval matching algorithm. It determines whether the absolute offset satisfies the condition of being greater than or equal to the `start_offset` of a certain level and less than or equal to the `end_offset` of that level. The level that meets the condition is the level to which the term belongs. The `structure_id` and `start_page` fields are extracted from the record of that level for subsequent calculations.

[0032] The relative offset is calculated through subtraction, subtracting the absolute character offset of the term from the starting character offset of its level. The difference represents the number of character intervals the term is within that level of text from the starting position. This relative offset needs to be mapped to a relative page number offset to determine the term's page number position within the level. The mapping process relies on the average number of characters per page as a conversion benchmark.

[0033] The average number of characters per page is calculated by dividing the total number of characters at that level by the page span. The total number of characters equals the end character offset of that level minus the start character offset, and the page span equals the end page number of that level minus the start page number plus one. For example, if a chapter has a start offset of 10000, an end offset of 15000, a start page number of 20, and an end page number of 29, then the total number of characters is 5000, the page span is 10, and the average number of characters per page is 500. Dividing the relative offset of the entry by the average number of characters per page yields a floating-point number for the relative page offset. Rounding this floating-point number down removes the decimal part, leaving only the integer part, gives the page offset of the entry relative to the start page number of the level.

[0034] The absolute page number of an entry is obtained through addition, by adding the starting page number of its level to the relative page number offset. The result is the actual page number of the first character of the entry in the document. This page number is recorded in the entry's page number list as the page number marker for the entry's first occurrence. Updating the matching pointer position increments the current pointer position value by the character length of the matching entry. The character length is calculated by counting the number of characters in the entry's text. For example, if the character length of the entry "deep learning" is 4, the pointer position is incremented by 4 to point to the next character position after the entry. The matching window length is reset to the initial value of 15. Starting from the updated pointer position, a new substring is extracted and the matching search continues in the Trie tree. This process is repeated until the entire chapter's text content has been traversed.

[0035] In one specific embodiment, step S3 includes: Traverse the text of each chapter of the structured text data, mark the covered text segments based on the character offset interval of each term in the vocabulary matching term set, and generate a list of covered intervals through the interval merging algorithm; Perform a difference operation between the total text intervals of the structured text data and the list of covered intervals to extract the uncovered text intervals and record the start and end positions of each uncovered interval to obtain a set of uncovered text fragments. Read each text fragment from the never-covered text fragment set, extract 500 characters before and after each text fragment as context and concatenate them to obtain the input text, combine the input text with the preset word extraction prompt template and send it to the large language model through the API interface; The system receives JSON-formatted vocabulary data returned by the large language model, parses the text content and category attributes of each vocabulary, performs similarity matching with the vocabulary in the vocabulary database using the edit distance algorithm, and replaces the original vocabulary text with the vocabulary-standardized vocabulary and inherits the vocabulary-level attributes when the edit distance is less than or equal to the threshold, thus obtaining the set of vocabulary extracted by the model.

[0036] Specifically, the character offsets of each term are read from the vocabulary matching term set. Each offset corresponds to a text interval, with the starting position of the interval being the offset value and the ending position being the starting position plus the length of the term's characters. The interval merging algorithm sorts all intervals in ascending order of their starting positions and then compares adjacent intervals sequentially. If the starting position of the current interval is less than or equal to the ending position of the previous interval, they are merged into a larger interval. The starting position of the new interval is set to the smaller value, and the ending position is set to the larger value, ultimately generating a list of non-overlapping covered intervals.

[0037] The difference operation subtracts the covered intervals from the total intervals of the entire text to obtain the uncovered intervals. The algorithm initializes the start position of the uncovered intervals to zero, traverses the list of covered intervals, and if the start position of the covered interval is greater than the start position of the uncovered interval, there is an uncovered segment between the two. The start and end positions of the segment are recorded and added to the set of uncovered text segments. The start position of the uncovered interval is updated to the end position of the covered interval plus one and the processing continues.

[0038] When extracting a segment from an uncovered text fragment, 500 characters are truncated from the beginning of the fragment as the preceding context, and 500 characters are truncated from the end of the fragment as the following context. These are then concatenated to obtain the input text containing the context. The preset prompt template includes task descriptions and output format requirements, specifying that the returned JSON array contains term and category fields. After filling the template with the input text, it is sent to the Large Language Model API interface via an HTTP POST request.

[0039] The JSON response returned by the large language model contains an array of terms. Parsing the array, we read the term and category fields of each element. The edit distance algorithm calculates the number of character operations between model terms and vocabulary terms. Operations include insertion, deletion, and replacement. If the edit distance is less than or equal to a threshold, the terms are considered similar. The original term text is replaced with a vocabulary-standardized term that inherits the vocabulary hierarchy attributes, resulting in the set of terms extracted by the model.

[0040] In one specific embodiment, step S4 includes: The deduplication and merging operation is performed on the vocabulary matching set and the model-extracted set of terms. For records with the same term text, the one with the higher confidence score is retained to obtain the candidate term set. Iterate through each term in the candidate term set, count the number of times the term appears in the structured text data, divide by the total number of words in the document to get the term frequency value, count the number of chapters containing the term, divide the total number of chapters by the number of chapters containing the term plus one, and take the logarithm to get the inverse document frequency value. Based on the terminology level attributes of each term, the corresponding level weighting coefficient is queried from the preset three-level weighting matrix; based on the position type of each term in the structured text data, the corresponding position weighting coefficient is queried from the preset position weighting matrix. The term frequency, inverse document frequency, hierarchical weighting coefficient, and positional weighting coefficient of each term are multiplied together to obtain a comprehensive weight value. The candidate term set is sorted in descending order according to the comprehensive weight value. The retention threshold K is calculated based on the ratio of the total number of words in the document to the preset density. The K terms with the highest weight ranking are selected to obtain the target index term set.

[0041] Specifically, when merging the vocabulary matching set and the model-extracted set, duplicate records are determined by comparing the text fields of the entries. If the text content of two entries is exactly the same, they are considered duplicates. The confidence score comparison rule is that the confidence score of the vocabulary matching entries is fixed at 1.0 because it comes from an authoritative vocabulary list, and the confidence score of the model-extracted entries is equal to the relevance score output by the large language model. The entry record with the higher confidence score is retained, while the other is deleted. The merged candidate entry set does not contain duplicate entries.

[0042] The term frequency (TNF) is calculated by iterating through all chapters of the structured text data, using a string matching function to count the occurrences of a term in each chapter, and summing the occurrences across all chapters to obtain the total number of occurrences of the term in the entire text. The total word count of the document is obtained by performing Chinese word segmentation on the entire text content and counting the number of words in the segmentation results. The TNF is then divided by the total number of occurrences of the term to obtain the TNF value. The inverse document frequency (INF) is calculated by counting the number of chapters containing the term. Each chapter is iterated through to determine if the term appears in that chapter; if so, the chapter count is incremented. The total number of chapters containing the term is then divided by the sum of the number of chapters containing the term plus one (incrementing to avoid a zero denominator), and the logarithm to base 10 is taken as the INF value.

[0043] The three-level weight matrix pre-defines a mapping relationship where first-level terms correspond to a weight coefficient of 0.6, second-level terms to 1.0, and third-level terms to 1.5. The corresponding level-weighted coefficient is obtained by querying this matrix based on the term's terminology level attribute value. The position weight matrix pre-defines a mapping relationship where the title area corresponds to a weight of 2.0, the abstract area to 1.8, the chapter title to 1.5, and the body text to 1.0. The position type is determined by querying the term's occurrence position in the structured text data, and the weight value corresponding to the position type with the highest weight among all occurrence positions is taken as the position weighted coefficient for that term.

[0044] The overall weight value is obtained by multiplying four parameters: term frequency, inverse document frequency, hierarchical weighting coefficient, and positional weighting coefficient. This value reflects the overall importance of the term in the document. Candidate terms are sorted in descending order of their overall weight values, with terms having higher weight values ​​appearing first. The retention threshold K is calculated by dividing the total number of words in the document by a preset density ratio. A density ratio of 200 is typically set to retain one index term for every 200 words. For example, if the total number of words in the document is 50,000, then K equals 250. The top K terms after sorting form the target index term set, which contains the K terms with the highest weights as the final index terms.

[0045] In one specific embodiment, step S5 includes: For each term in the target indexed term set, call the pinyin conversion library to generate the corresponding pinyin string, and use the pinyin string as the sorting key; The target indexed term set is sorted in ascending lexicographical order by the pinyin string. When the pinyin is the same, it is sorted in a secondary order by the tone value to obtain a pinyin ordered term list. Iterate through each entry in the ordered list of pinyin entries, read the page number value list of the entry and sort it in ascending order, perform continuous page number recognition and compression on the sorted page number value list, convert the continuous page number segment into the start and end page number format, and obtain the page number string of each entry; The text content, category, page number string, and comprehensive weight value of each entry in the ordered list of pinyin entries are encapsulated into structured data objects, and a structured index document containing an entry index table and metadata information is generated according to a preset index document format template.

[0046] Specifically, this step calls the Pinyin4j library or a similar pinyin conversion tool to perform pinyin conversion on the term text. The conversion process reads each Chinese character in the term, queries a Chinese character pinyin mapping table to obtain the pinyin representation of that character, and uses the most common pronunciation as the default pinyin for polyphonic characters. The pinyin of all characters in the term is then concatenated in order to form a complete pinyin string. For example, the term "deep learning" is converted to the pinyin string "shenduXueXi," and this pinyin string is stored in the pinyin field of the term record as a sorting keyword.

[0047] The dictionary sorting process arranges entries in ascending lexicographical order from a to z according to the alphabetical order of the pinyin strings. The comparison rule is to compare the pinyin strings character by character, with entries starting with the smaller character listed first. If the first characters are the same, the second character is compared, and so on. Tone values ​​are extracted from the pinyin strings by identifying tone markers: first tone corresponds to 1, second tone to 2, third tone to 3, fourth tone to 4, and neutral tone to 0. When two entries have identical pinyin strings, their tone value sequences are compared, with the entry having the smaller tone value listed first. This secondary sorting results in an ordered list of pinyin entries.

[0048] The ascending sort of the page number value list arranges the page numbers from smallest to largest by comparing their numerical values. The difference between adjacent page numbers after sorting facilitates the identification of consecutive page number segments. The logic for consecutive page number identification is to traverse the sorted page number list, calculate the difference between any two adjacent page numbers, and mark them as consecutive if the difference equals 1. This process continuously counts the start and end page numbers of consecutive page numbers. Compression converts consecutive page number segments into a start-end page number format. For example, the page number sequence 3, 4, 5, 6 is compressed into 3-6. Non-consecutive independent page numbers retain their original numerical form. All compressed page number segments and independent page numbers are connected with commas to form a page number string such as 3-6, 8, 10-11.

[0049] The encapsulation of structured data objects uses the term text content, term category, page number string, and comprehensive weight value as attribute fields, organizing the data structure in JSON or XML format. The index document format template defines the document's layout structure, including a document title area displaying the thesis index, a metadata area displaying the thesis title and author information, and a term index table listing each term and its page number row by row. The generation process iterates through the alphabetically ordered list of terms, filling the attribute values ​​of each term into the corresponding positions in the template, ultimately outputting a structured indexed document containing complete index information. The document format supports three output types: TXT plain text, DOCX rich text, and JSON structured data.

[0050] The automatic indexing method based on a large language model in the embodiments of this application has been described above. The automatic indexing system based on a large language model in the embodiments of this application is described below. Please refer to [link to relevant documentation]. Figure 2 One embodiment of the automatic indexing system based on a large language model in this application includes: The parsing module is used to perform structured parsing of the document to be indexed, dividing the document content into title area, summary area and chapter area according to logical hierarchy, and obtaining structured text data and page number mapping relationship of each area; The matching module is used to perform a forward matching scan on the structured text data using a vocabulary list, identify terms that are completely consistent with the vocabulary list and record their page number positions, thereby obtaining a set of vocabulary list matching terms. The analysis module is used to extract text intervals from the structured text data that are not covered by the vocabulary matching term set, input the text intervals along with the preceding and following context into the large language model for semantic analysis, and output newly identified terms from the large language model and perform similarity matching with the vocabulary to complete the normalization, thereby obtaining the term set extracted by the model. The calculation module is used to merge the vocabulary matching term set and the term set extracted by the model to form a candidate term set. Based on the frequency of each candidate term in the document, the distribution of the document, the vocabulary hierarchy attribute and the position type, the term frequency value, the inverse document frequency value, the hierarchy weighting coefficient and the position weighting coefficient are calculated respectively. The four parameters are multiplied to obtain the comprehensive weight value of each term. After sorting the terms in descending order of weight value, the top K terms are selected to obtain the target indexing term set. The sorting module is used to reorder the target index term set according to the alphabetical order of the term pinyin, attach the page number string corresponding to each term to each term, and generate a structured indexing document.

[0051] above Figure 2 The automatic indexing system based on a large language model in this embodiment of the invention will be described in detail from the perspective of modular functional entities. The automatic indexing device based on a large language model in this embodiment of the invention will be described in detail from the perspective of hardware processing.

[0052] Reference Figure 3 This invention also provides an automatic indexing device based on a large language model. This automatic indexing device can be a server, and its internal structure can be as follows: Figure 3 As shown, the large language model-based automatic indexing device includes a processor, memory, display screen, input device, network interface, and database connected via a system bus. The processor, designed for computing and control, provides computational and control capabilities. The memory of the large language model-based automatic indexing device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the large language model-based automatic indexing device stores the data corresponding to this embodiment. The network interface of the large language model-based automatic indexing device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the above-described method.

[0053] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the automatic indexing device based on a large language model to which the present invention is applied.

[0054] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when the instructions are executed on a computer, cause the computer to perform the steps of the automatic indexing method based on a large language model.

[0055] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0056] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an automatic indexing device based on a large language model (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0057] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An automatic indexing method based on a large language model, characterized in that, The method includes: Step S1: Perform structured parsing on the document to be indexed, dividing the document content into heading area, abstract area and chapter area according to logical hierarchy, and obtaining structured text data and page number mapping relationship of each area; Step S2: Perform a forward matching scan on the structured text data using the vocabulary, identify terms that are completely consistent with the vocabulary and record their page number positions to obtain a set of vocabulary matching terms; Step S3: Extract text intervals not covered by the word list matching term set from the structured text data, input the text intervals along with the preceding and following context into the large language model for semantic analysis, and output newly identified terms from the large language model and perform similarity matching with the word list to complete the normalization, thereby obtaining the term set extracted by the model; Step S4: Merge the term matching set and the term extraction set from the model to form a candidate term set. Calculate the term frequency, inverse document frequency, level weighting coefficient, and position weighting coefficient for each candidate term based on its frequency of occurrence in the document, its distribution within the document, its term hierarchy attribute, and its position type. Multiply the four parameters to obtain the comprehensive weight value for each term. Sort the terms in descending order of weight value and select the top K terms to obtain the target indexing term set. Step S5: Reorder the target index term set according to the alphabetical order of the term pinyin, attach the page number string corresponding to each term to each term, and generate a structured index document.

2. The automatic indexing method based on a large language model according to claim 1, characterized in that, Step S1 includes: The document parsing library is called to perform format recognition on the document to be indexed, convert the document binary stream into a plain text stream and extract the document title, author and page number information to obtain the document metadata and full text content; By matching the chapter title feature patterns in the full text content using regular expressions, the text boundaries of the title layer, abstract layer, keyword layer and each chapter layer are identified, and a unique structural identifier is assigned to each level. Based on the character offset of each level of text in the full text content, the starting page number and ending page number of each level are calculated in combination with the total number of pages and the total number of characters in the document. The structure identifiers, text content, start page number, and end page number of each level are encapsulated into an array of JSON objects to obtain the structured text data and the page number mapping relationship of each region.

3. The automatic indexing method based on a large language model according to claim 1, characterized in that, Step S2 includes: Load term records from the subject thesaurus database, and construct a Trie tree index structure from the text content, thesaurus hierarchical attributes, and term category of each term. Each node of the tree stores a Chinese character. Initialize the matching pointer position and the matching window length, extract a substring with a length equal to the matching window length from the beginning position of the chapter text of the structured text data, and perform a path lookup in the Trie tree index structure to determine whether a complete match exists; When a match is successful, the text content, term identifier, and term hierarchy attributes of the matched term are recorded. Based on the character offset of the matching pointer position in the structured text data and the page number mapping relationship, the page number value of the term is calculated. The matching pointer position is then moved forward by the character length of the matched term before the matching continues. After traversing all chapters of the structured text data to complete the matching scan, the page number value list of each term is sorted in ascending order and compressed into consecutive page numbers to generate a page number string, thus obtaining the set of matched terms and their corresponding page number positions.

4. The automatic indexing method based on a large language model according to claim 3, characterized in that, The step of calculating the page number value of the term based on the character offset of the matching pointer position in the structured text data and the page number mapping relationship, and then moving the matching pointer position forward by the character length of the matching term before continuing the matching includes: Read the character offset of the matching term in the structured text data, and query the logical hierarchy structure identifier to which the character offset belongs and the starting page number of the hierarchy; The difference between the character offset and the starting character offset of the level is calculated to obtain the relative offset of the term within the level. The average number of characters per page is calculated based on the total number of characters and page number span of the level. The relative offset is divided by the average number of characters per page and rounded down to obtain the relative page number offset of the entry within the level. The starting page number of the level is added to the relative page number offset to obtain the absolute page number value where the term appears. The matching pointer position is then increased by the character length value of the matching term, the matching window length is reset, and the matching process continues.

5. The automatic indexing method based on a large language model according to claim 1, characterized in that, Step S3 includes: Traverse each chapter of the structured text data, mark the covered text segments based on the character offset interval of each term in the vocabulary matching term set, and generate a list of covered intervals through an interval merging algorithm; Perform a difference operation between the total text intervals of the structured text data and the list of covered intervals to extract the uncovered text intervals and record the start and end positions of each uncovered interval to obtain a set of uncovered text fragments. Read each text fragment from the uncovered text fragment set, extract 500 characters before and after the text fragment as context and splice them to obtain the input text, combine the input text with the preset word extraction prompt template and send it to the large language model through the API interface; The system receives JSON-formatted term data returned by the large language model, parses the text content and category attributes of each term, performs similarity matching with terms in the vocabulary database using the edit distance algorithm, and replaces the original term text with a standard term from the vocabulary database and inherits the vocabulary hierarchy attributes when the edit distance is less than or equal to the threshold, thus obtaining the term set extracted by the model.

6. The automatic indexing method based on a large language model according to claim 1, characterized in that, Step S4 includes: The word list matching word set and the model extracted word set are deduplicated and merged. For records with the same word text, the one with the higher confidence score is retained to obtain the candidate word set. Traverse each term in the candidate term set, count the number of times the term appears in the structured text data, divide by the total number of words in the document to obtain the term frequency value, count the number of chapters containing the term, divide the total number of chapters by the number of chapters containing the term, add one, and take the logarithm to obtain the inverse document frequency value. Based on the term hierarchy attribute of each term, the corresponding hierarchy weighting coefficient is queried from the preset three-level weighting matrix; based on the position type of each term in the structured text data, the corresponding position weighting coefficient is queried from the preset position weighting matrix. The term frequency value, inverse document frequency value, hierarchical weighting coefficient, and positional weighting coefficient of each term are multiplied together to obtain a comprehensive weight value. The candidate term set is sorted in descending order according to the comprehensive weight value. The retention threshold K is calculated based on the ratio of the total number of words in the document to the preset density. The K terms with the highest weight ranking are selected to obtain the target index term set.

7. The automatic indexing method based on a large language model according to claim 1, characterized in that, Step S5 includes: For each term in the target indexed term set, a pinyin conversion library is called to generate a corresponding pinyin string, and the pinyin string is used as a sorting keyword. The target indexed term set is sorted in ascending lexicographical order according to the pinyin string. When the pinyin is the same, it is sorted in a secondary order according to the tone value to obtain a pinyin ordered term list. Traverse each entry in the ordered list of pinyin entries, read the page number value list of the entry and sort it in ascending order, perform continuous page number recognition and compression on the sorted page number value list, convert the continuous page number segment into the start and end page number format, and obtain the page number string of each entry; The text content, category, page number string, and comprehensive weight value of each entry in the ordered list of pinyin entries are encapsulated into a structured data object, and the structured index document containing the entry index table and metadata information is generated according to a preset index document format template.

8. An automatic indexing system based on a large language model, characterized in that, For implementing the automatic indexing method based on a large language model as described in any one of claims 1-7, the automatic indexing system based on a large language model comprises: The parsing module is used to perform structured parsing of the document to be indexed, dividing the document content into title area, summary area and chapter area according to logical hierarchy, and obtaining structured text data and page number mapping relationship of each area; The matching module is used to perform a forward matching scan on the structured text data using a vocabulary list, identify terms that are completely consistent with the vocabulary list and record their page number positions, thereby obtaining a set of vocabulary list matching terms. The analysis module is used to extract text intervals from the structured text data that are not covered by the vocabulary matching term set, input the text intervals along with the preceding and following context into the large language model for semantic analysis, and output newly identified terms from the large language model and perform similarity matching with the vocabulary to complete the normalization, thereby obtaining the term set extracted by the model. The calculation module is used to merge the vocabulary matching term set and the term set extracted by the model to form a candidate term set. Based on the frequency of each candidate term in the document, the distribution of the document, the vocabulary hierarchy attribute and the position type, the term frequency value, the inverse document frequency value, the hierarchy weighting coefficient and the position weighting coefficient are calculated respectively. The four parameters are multiplied to obtain the comprehensive weight value of each term. After sorting the terms in descending order of weight value, the top K terms are selected to obtain the target indexing term set. The sorting module is used to reorder the target index term set according to the alphabetical order of the term pinyin, attach the page number string corresponding to each term to each term, and generate a structured indexing document.

9. An automatic indexing device based on a large language model, characterized in that, It includes a memory and a processor, the memory storing a computer program that can run on the processor, and the processor executing the computer program to implement the automatic indexing method based on a large language model as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is run by a processor, it causes the processor to execute the automatic indexing method based on a large language model as described in any one of claims 1 to 7.

Images