An index table generation method of a book data structure based on artificial intelligence

Abstract

The application discloses an index table generation method of a book data structure based on artificial intelligence. The method comprises the following steps: acquiring a book full text, performing data preprocessing on the book full text, acquiring a preprocessing text, performing word segmentation processing on the preprocessing text, acquiring a word segmentation set, and assigning a book type label according to the word segmentation set; filtering the word segmentation set according to the assigned book type label, and extracting a first-level item keyword; determining a second-level item keyword corresponding to the first-level item keyword according to the first-level item keyword; and generating a book index table according to the first-level and second-level item keywords. The application can automatically adapt to different types of books and accurately generate a book index table.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of book data processing, and particularly relates to a book data structure index directory generation method based on artificial intelligence. BACKGROUND

[0002] In the prior art, a book index directory is usually generated by collecting chapter titles of a predetermined style in a book. However, this method has high requirements for the style of the book. In the prior art, there is also a method for generating a book index directory by content analysis. However, this method lacks adaptability to different types of books, resulting in inaccurate generation of the index directory. Therefore, how to adapt to different types of books and accurately generate a book index directory has become a problem to be solved in the field. SUMMARY

[0003] The application provides a book data structure index directory generation method based on artificial intelligence, which comprises the following steps:

[0004] S101, acquiring a book full text, performing data preprocessing on the book full text to obtain a preprocessing text, performing word segmentation processing on the preprocessing text to obtain a word segmentation set, assigning a book type label according to the word segmentation set, the data preprocessing is used for maximum word segmentation on the book full text against a dictionary and removing invalid words, and the word segmentation processing is used for word segmentation processing on the preprocessing text to improve the word segmentation accuracy;

[0005] S102, screening the word segmentation set according to the assigned book type label and extracting a first item keyword;

[0006] S103, determining a second item keyword corresponding to the first item keyword according to the first item keyword, specifically including determining a search range of the second item keyword and then screening the second item keyword;

[0007] S104, generating a book index directory according to the first item keyword and the second item keyword.

[0008] Further, in step S101, the data preprocessing on the book full text specifically comprises:

[0009] obtaining a preprocessing text ,

[0010]

[0011] wherein, is the preprocessing text, is a book text, is a dictionary parameter, is a forward maximum matching function, The word segmentation cleaning function is used to remove invalid words such as stop words.

[0012] Further, the word segmentation processing on the preprocessed text in step S101 specifically includes:

[0013] First, the mark sequence corresponding to the preprocessed text is obtained

[0014]

[0015] wherein, is the preprocessed text, is the mark sequence corresponding to the preprocessed text, indicating the word position state corresponding to each word, such as the beginning of a word, the middle of a word, the end of a word, and a single word, is the sequence position of the word in the preprocessed text, is a normalization factor, is the number of transition feature functions, is the parameter corresponding to each transition feature function, is the transition feature function, is the number of state feature functions, is the parameter corresponding to each state feature function, is the state feature function, is the mark value corresponding to the position of the i-th word, is the mark value corresponding to the position of the i-th word, is the Viterbi function. Further, the word segmentation set is obtained

[0016]

[0017]

[0018] wherein, is the word segmentation set, is a mapping function, is the above mark sequence. Thus, the mapping relationship between the mark and the word is found by traversing the mark sequence, and the word segmentation set is obtained.

[0019] Further, in step S101, according to the word segmentation set, the book type label is assigned, specifically including:

[0020] The book type label is obtained

[0021]

[0022] wherein, is the book type label, is the total number of books, ​​​​​the number of each type of book, the word at the corresponding position in the word set, for each type of book, a statistical function for counting the number of occurrences of each word in the type of book, the length of the word set, the number of book types.

[0023] Further, in step S102, according to the assigned book type label, the word set is filtered to extract the first-level entry keyword, which specifically includes:

[0024] extracting the first-level entry keyword ,

[0025]

[0026] the first-level entry keyword list, the word set, the first-level entry keyword dictionary corresponding to each type of book, the keyword matching function, which is used to traverse the word set to find the corresponding first-level entry keyword.

[0027] Further, in step S103, the search range of the second-level entry keyword is determined, which specifically includes:

[0028] determining the search range sequence of the second-level entry keyword corresponding to the first-level entry keyword ,

[0029]

[0030] wherein, is the first-level entry keyword, is the word set, is the length of the word set, is the number of first-level entry keywords, is the search function, which is used to find the position of the first-level entry keyword in the word set, and the search range of the second-level entry keyword is from the current first-level entry keyword to the appearance of the next first-level entry keyword;

[0031] filtering the second-level entry keyword, which specifically includes:

[0032] obtaining the second-level entry keyword according to the search range sequence of the second-level entry keyword ,

[0033]

[0034] wherein, is the second-level entry keyword, is the search range for searching the second-level entry keyword according to the first-level entry keyword, is the association relationship between the first-level entry keyword and the second-level entry keyword, indicates the association relationship between the first-level entry keyword and the second-level entry keyword . is a matching function for extracting the second-level entry keyword related to the first-level entry keyword.

[0035] Further, the multi-level entry keywords including third-level or above are obtained by progressively performing the search range and the screening of the second-level entry keyword according to the explicit second-level entry keyword search range.

[0036] The present application also relates to a system using the above-mentioned index catalog generation method based on the artificial intelligence-based book data structure, which comprises:

[0037] a processing module for obtaining a book full text, performing data preprocessing on the book full text, obtaining a preprocessed text, performing word segmentation processing on the preprocessed text, obtaining a word segmentation set, assigning a book type label according to the word segmentation set, the data preprocessing being used for maximum word segmentation on the book full text against a dictionary and removing invalid words, and the word segmentation processing being used for word segmentation processing on the preprocessed text to improve the word segmentation accuracy;

[0038] a first-level screening module for screening the word segmentation set according to the assigned book type label and extracting a first-level entry keyword;

[0039] a second-level screening module for determining a second-level entry keyword corresponding to the first-level entry keyword according to the first-level entry keyword, specifically including explicitly defining a second-level entry keyword search range and then screening the second-level entry keyword;

[0040] a generation module for generating a book index catalog according to the first-level and second-level entry keywords.

[0041] The present application also relates to a computer program product comprising a computer program, which is executed by a processor to execute the above-mentioned index catalog generation method based on the artificial intelligence-based book data structure.

[0042] The application also relates to a computer readable storage medium for storing a computer program, which is executed by a processor to execute the AI-based index table generation method of a book data structure.

[0043] The technical scheme of the application can automatically adapt to different types of books and accurately generate an index table of the books by automatically assigning a book type label and obtaining a first entry keyword and a second entry keyword in association with the book type. DETAILED DESCRIPTION

[0044] The application will be further described below. The following examples are only used to more clearly illustrate the technical scheme of the application and cannot be used to limit the protection scope of the application. It should be pointed out that the following detailed description is exemplary and is intended to provide further description of the application.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It must be noted that as used herein, the terms are intended to be interpreted as being open-ended, and specifically each and every one of the terms should be accorded the broadest interpretation under the circumstances, except to the extent otherwise explicitly provided. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0046] Embodiment 1 of the application relates to an AI-based index table generation method of a book data structure, which comprises the following steps:

[0047] In S101, the full text of a book is obtained, data preprocessing is performed on the full text of the book to obtain preprocessed text, the preprocessed text is subjected to word segmentation processing to obtain a word segmentation set, and a book type label is assigned according to the word segmentation set.

[0048] The data preprocessing is used to perform maximum word segmentation on the full text of the book against a dictionary and remove invalid words, and the word segmentation processing is used to perform word segmentation processing on the preprocessed text to improve the accuracy of word segmentation.

[0049] The book type may include, for example, an academic research type, a technical manual type, a history and humanities type, a leisure and entertainment type, an education and learning type, and the like.

[0050] The data preprocessing on the full text of the book specifically includes obtaining preprocessed text ,

[0051]

[0052] in, For preprocessing text, For book text, For dictionary parameters, It is a positive maximum matching function. This is a word segmentation and cleaning function used to remove invalid words such as stop words.

[0053] The specific steps of word segmentation for preprocessed text include first obtaining the tag sequence corresponding to the preprocessed text. ,

[0054]

[0055] in, For preprocessing text, To preprocess the text, a token sequence is used to represent the word position status of each character, such as word beginning, word middle, word ending, or standing alone as a word. For the sequence position of a word in the preprocessed text, For normalization factor, The number of transition characteristic functions, The parameters corresponding to each transition feature function, For the transition characteristic function, The number of state characteristic functions. The parameters corresponding to each state feature function For state characteristic functions, For the first The tag value corresponding to the position of each character. For the first The tag value corresponding to the position of each character. Let be the Viterbi function. Thus, the most likely sequence of markers is obtained.

[0056] Get the word segmentation set again ,

[0057]

[0058] in, For word segmentation set, For mapping functions, The above is the tag sequence. Therefore, by traversing the tag sequence, the mapping relationship between tags and words is found, resulting in a word segmentation set.

[0059] Based on the word segmentation set, assign book type tags, specifically including obtaining book type tags. ,

[0060]

[0061] in, For book type tags, For the total number of books, For the quantity of each type of book, The word at the corresponding position in the word set. For each book type, This is a statistical function used to count the number of times each word appears in books of this type. The length of the word segmentation set, This represents the number of book types.

[0062] S102, Based on the assigned book type tags, filter the word segmentation set and extract the first-level entry keywords. .

[0063] Taking academic research books as an example, extract the following primary keywords: Introduction, Literature Review, Methodology, Conclusion, and References. Specifically, extract the primary keywords... Specifically:

[0064]

[0065] This is a list of keywords for the first-level entries. For word segmentation set, A dictionary of first-level keywords corresponding to book types. This is a keyword matching function used to iterate through the word segmentation set. Find the corresponding first-level entry keywords.

[0066] S103. Based on the primary entry keywords, determine the corresponding secondary entry keywords. Specifically, first clarify the search scope of the secondary entry keywords, and then filter the secondary entry keywords. Taking academic research books as an example, the secondary entry keywords corresponding to the primary entry keyword "Introduction" include the specific content of the research background, the specific content of the research purpose and significance, the specific content of the research methods, etc.

[0067] The above clarifies the search scope of keywords for secondary entries, specifically including determining the first... The search scope sequence of secondary keyword entries corresponding to each primary keyword entry. ,

[0068]

[0069] in, For the first One primary entry keyword, For word segmentation set, The length of the word segmentation set, The number of keywords for the first-level entries. To find the function, for finding the location of the primary entry keyword in the segmentation set, the secondary entry keyword search range is from the current primary entry keyword to the next primary entry keyword.

[0070] The above screening secondary entry keyword specifically includes obtaining the secondary entry keyword according to the search range sequence of the secondary entry keyword ,

[0071]

[0072] Among them, is the th secondary entry keyword, is the search range when the th secondary entry keyword is found according to the primary entry keyword, is the association relationship between the primary entry keyword and the secondary entry keyword, indicates the association relationship between the primary entry keyword and the secondary entry keyword , is a matching function, which is used to extract the secondary entry keyword related to the primary entry keyword.

[0073] S104, according to the primary and secondary entry keywords, generates a book index directory.

[0074] Again, taking the academic research type book as an example, the generated book index directory is as follows,

[0075] Introduction (primary entry keyword)

[0076] • Research background (secondary entry keyword)

[0077] • Research purpose and significance (secondary entry keyword)

[0078] • Research method (secondary entry keyword)

[0079] Literature review (primary entry keyword)

[0080] • Theoretical framework (secondary entry keyword)

[0081] • Findings of previous research (secondary entry keyword)

[0082] • Research gap (secondary entry keyword)

[0083] Methodology (primary entry keyword)

[0084] • Data collection (secondary entry keyword)

[0085] • Data analysis method (secondary entry keyword)

[0086] • Study design (secondary entry keyword)

[0087] Results (primary entry keyword)

[0088] • Data analysis results (secondary entry keyword)

[0089] • Charts and statistics (secondary entry keyword)

[0090] • Discussion (secondary entry keyword)

[0091] Conclusion (primary entry keyword)

[0092] • Study summary (secondary entry keyword)

[0093] • Research contribution (secondary entry keyword)

[0094] • Future research directions (secondary entry keyword)

[0095] References (primary entry keyword)

[0096] • Books (secondary entry keyword)

[0097] • Journal articles (secondary entry keyword)

[0098] • Web resources (secondary entry keyword)

[0099] Appendices (primary entry keyword)

[0100] • Survey questionnaires (secondary entry keyword)

[0101] • Data tables (secondary entry keyword)

[0102] • Additional charts (secondary entry keyword)

[0103] The index directory can also include multi-level entry keywords, which include tertiary or higher levels, and the search range and screening method of the secondary entry keywords are progressively implemented to obtain multi-level entry keywords.

[0104] Embodiment 2 of the present application relates to a system using the index directory generation method of the artificial intelligence-based book data structure of embodiment 1, comprising:

[0105] The processing module is used for acquiring a book full text, performing data preprocessing on the book full text, acquiring a preprocessing text, performing word segmentation processing on the preprocessing text, acquiring a word segmentation set, assigning a book type label according to the word segmentation set, the data preprocessing is used for maximum word segmentation on the book full text against a dictionary, and invalid words are removed, and the word segmentation processing is used for word segmentation processing on the preprocessing text, and improves the word segmentation accuracy;

[0106] The first screening module is used for screening the word segmentation set according to the assigned book type label, and extracting a first item keyword;

[0107] The second screening module is used for determining a second item keyword corresponding to the first item keyword according to the first item keyword, and specifically includes the following steps of: determining a search range of the second item keyword, and then screening the second item keyword;

[0108] The generating module is used for generating a book index directory according to the first item keyword and the second item keyword.

[0109] Embodiment 3 of the present application relates to a computer program product, the computer program product comprises a computer program, the computer program is executed by a processor, and is used for executing the index directory generation method of the book data structure based on artificial intelligence in embodiment 1.

[0110] Embodiment 4 of the present application relates to a computer readable storage medium, the computer readable storage medium is used for storing a computer program, the computer program is executed by a processor, and is used for executing the index directory generation method of the book data structure based on artificial intelligence in embodiment 1.

[0111] As described above, only the preferred embodiments of the present application, it should be noted that, for those skilled in the art, without departing from the technical principles of the present application, can make several improvements and modifications, these improvements and modifications should also be considered as the protection scope of the present application.

Claims

1. A method for generating an index catalog based on an artificial intelligence-based book data structure, characterized in that, The method includes the following steps: S101, Obtain the full text of the book, perform data preprocessing on the full text of the book, and obtain the preprocessed text. , ； in, For preprocessing text, For book text, For dictionary parameters, It is a positive maximum matching function. This is a word segmentation and cleaning function used to remove stop words, then perform word segmentation on the preprocessed text to obtain a word segmentation set, and assign book type tags based on the word segmentation set. , ； in, For book type tags, For the total number of books, For the quantity of each type of book, The word at the corresponding position in the word set. For each book type, This is a statistical function used to count the number of times each word appears in books of this type. The length of the word segmentation set, The number of book types is specified. The data preprocessing is used to perform maximum word segmentation on the full-text dictionary of books and remove invalid words. The word segmentation process is used to segment the preprocessed text to improve the word segmentation accuracy. S102, Based on the assigned book type tags, filter the word segmentation set and extract the first-level entry keywords. , ； This is a list of keywords for the first-level entries. For word segmentation set, A dictionary of first-level keywords for each book type. This is a keyword matching function used to iterate through the word segmentation set. Find the corresponding first-level entry keywords; S103, Based on the keywords of the first-level entries, determine the corresponding keywords of the second-level entries, specifically including determining the keywords of the first-level entries. The search scope sequence of secondary keyword entries corresponding to each primary keyword entry. , ； in, For the first One primary entry keyword, For word segmentation set, The length of the word segmentation set, The number of keywords for the first-level entries. This is a lookup function used to find the position of the first-level entry keyword in the word segmentation set. The search range for the second-level entry keyword is from the current first-level entry keyword to the appearance of the next first-level entry keyword. Filtering keywords for secondary entries, specifically including: Obtain secondary entry keywords based on the search scope sequence of secondary entry keywords. , ； ； in, For the first Keywords for each secondary item To find the first entry based on the keywords of the first-level entry The search scope when using keywords for second-level entries. This describes the relationship between keywords in first-level entries and keywords in second-level entries. Keywords representing first-level entries Keywords of secondary entries The relationship between them This is a matching function used to extract keywords for second-level entries that are related to the keywords of the first-level entries; S104: Generate a book index directory based on the keywords of the first-level and second-level entries.

2. The method for generating an index catalog based on an artificial intelligence-based book data structure according to claim 1, characterized in that, In step S101, the word segmentation process for the preprocessed text specifically includes: First, obtain the tag sequence corresponding to the preprocessed text. , ； in, For preprocessing text, To preprocess the text, a token sequence is used to represent the word position status of each character, such as word beginning, word middle, word ending, or standing alone as a word. For the sequence position of a word in the preprocessed text, For normalization factor, The number of transition characteristic functions, The parameters corresponding to each transition feature function, For the transition characteristic function, The number of state characteristic functions. The parameters corresponding to each state feature function For state characteristic functions, For the first The tag value corresponding to the position of each character. For the first The tag value corresponding to the position of each character. It is the Viterbi function; Then obtain the word segmentation set , ； in, For word segmentation set, For mapping functions, Given the above-mentioned tag sequence, the mapping relationship between tags and words can be found by traversing the tag sequence, thus obtaining the word segmentation set.

3. The method for generating an index catalog based on an artificial intelligence-based book data structure according to claim 1, characterized in that, This includes multi-level entry keywords, where the multi-level includes three or more levels. The process proceeds progressively by defining the search scope of second-level entry keywords and filtering second-level entry keywords to obtain multi-level entry keywords.

4. A system for generating an index catalog using an artificial intelligence-based book data structure according to any one of claims 1-3, characterized in that, The system includes: The processing module is used to obtain the full text of the book, perform data preprocessing on the full text of the book, obtain the preprocessed text, perform word segmentation on the preprocessed text, obtain the word segmentation set, and assign book type labels according to the word segmentation set. The data preprocessing is used to perform maximum word segmentation on the full text of the book compared with the dictionary and remove invalid words. The word segmentation is used to perform word segmentation on the preprocessed text to improve the word segmentation accuracy. The first-level filtering module is used to filter the word segmentation set based on the assigned book type tags and extract the first-level entry keywords; The secondary filtering module is used to determine the corresponding secondary entry keywords based on the primary entry keywords. Specifically, it includes clarifying the search scope of the secondary entry keywords and then filtering the secondary entry keywords. The generation module is used to generate a book index catalog based on the keywords of the first-level and second-level entries.

5. A computer program product, characterized in that, The computer program product includes a computer program that is executed by a processor to perform an index catalog generation method for a book data structure based on artificial intelligence as described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program, which is executed by a processor to perform an index catalog generation method for a book data structure based on artificial intelligence as described in any one of claims 1-3.

Images