Method for generating index directory of book data structure based on artificial intelligence

Abstract

The invention discloses a book data structure index directory generation method based on artificial intelligence. The method comprises the following steps: acquiring a full text of a book, performing data preprocessing on the full text of the book, acquiring a preprocessed text, performing word segmentation processing on the preprocessed text, acquiring a word segmentation set, and distributing book type labels according to the word segmentation set; screening the word segmentation set according to the distributed book type labels, and extracting first-level entry keywords; according to the first-level entry keyword, determining a second-level entry keyword corresponding to the first-level entry keyword; and generating a book index directory according to the first-level and second-level entry keywords. The method can automatically adapt to different types of books and accurately generate the book index directory.

Description

Technical Field

[0001] The invention belongs to the technical field of book data processing, and in particular relates to an index directory generation method of a book data structure based on artificial intelligence. Background Art

[0002] In the prior art, the book index directory is usually generated by collecting the chapter titles of the predetermined style in the book, but this method has high requirements on the book style. There are also methods in the prior art for generating book index directories through content analysis, but they lack the ability to adapt to different types of books, resulting in inaccurate index directory generation. Therefore, how to adapt to different types of books and accurately generate book index directories has become a problem to be solved in this field. Summary of the invention

[0003] The present invention provides an index directory generation method of a book data structure based on artificial intelligence, the method comprising the following steps: S101, obtaining the full text of the book, performing data preprocessing on the full text of the book, obtaining the preprocessed text, then performing word segmentation processing on the preprocessed text, obtaining a word segmentation set, and assigning a book type label according to the word segmentation set. The data preprocessing is used to perform maximum word segmentation on the full text of the book against the dictionary and remove invalid words. The word segmentation processing is used to perform word segmentation processing on the preprocessed text to improve the word segmentation accuracy; S102, screening the word set according to the assigned book type label to extract the first-level entry keywords; S103, determining the corresponding secondary entry keywords according to the primary entry keywords, specifically including clarifying the search scope of the secondary entry keywords, and then screening the secondary entry keywords; S104, generating a book index catalog based on the primary and secondary entry keywords.

[0004] Further, in step S101, data preprocessing is performed on the full text of the book, specifically including: Get preprocessed text ,

[0005] in, To preprocess the text, For book text, is a dictionary parameter, is the forward maximum matching function, It is a word segmentation cleaning function used to remove invalid words such as stop words.

[0006] Further, in step S101, the word segmentation processing of the preprocessed text specifically includes: First get the token sequence corresponding to the preprocessed text ,

[0007] in, To preprocess the text, The token sequence corresponding to the preprocessed text indicates the word position state of each character, such as the beginning, middle, end, or a single word. is the sequence position of the word in the preprocessed text, is the normalization factor, is the number of transfer characteristic functions, For each parameter corresponding to the transfer characteristic function, is the transfer characteristic function, is the number of state characteristic functions, is the parameter corresponding to each state characteristic function, is the state characteristic function, For the The mark value of the corresponding position of the word, For the The mark value of the corresponding position of the word, is the Viterbi function; Get the word set again , in, is a word segmentation set, is the mapping function, is the above tag sequence. Thus, by traversing the tag sequence to find the mapping relationship between tags and words, a word segmentation set is obtained.

[0008] Further, in step S101, the book type label is assigned according to the word segmentation set, specifically including: Get book type tags ,

[0009] in, is the book type label, is the total number of books, is the number of books of each type, is the word at the corresponding position in the word set, For each book type, is a statistical function used to count the number of times each word appears in books of this type. is the length of the word segmentation set, The number of book types.

[0010] Further, in step S102, the word segmentation set is screened according to the assigned book type label to extract the first-level entry keywords, specifically including: Extract first-level entry keywords ,

[0011] is a list of first-level entry keywords. is a word segmentation set, A first-level keyword dictionary corresponding to each book type, It is a keyword matching function used to traverse the word segmentation set , find the corresponding first-level entry keywords.

[0012] Further, in step S103, the search scope of the secondary entry keywords is clarified, including: Determine The search range sequence of the second-level entry keywords corresponding to the first-level entry keywords ,

[0013] in, For the First-level entry keywords, is a word segmentation set, is the length of the word segmentation set, is the number of first-level entry keywords, is a search function used to find the position of the first-level entry keyword in the word segmentation set. The search range of the second-level entry keyword is from the current first-level entry keyword to the next first-level entry keyword. Filter secondary entry keywords, including: Get the secondary entry keywords according to the search range sequence of the secondary entry keywords ,

[0014] in, For the Second-level entry keywords, To search for the first level entry based on the keywords The search scope when there are two secondary entry keywords, is the association between the first-level entry keywords and the second-level entry keywords, Indicates the first-level entry keyword With secondary entry keywords The relationship between It is a matching function used to extract the secondary entry keywords related to the primary entry keywords.

[0015] Furthermore, it includes multi-level entry keywords, wherein the multi-level includes three levels or more, and the method of clarifying the search scope of the secondary entry keywords and screening the secondary entry keywords is executed progressively to obtain the multi-level entry keywords.

[0016] The present invention also relates to a system using the above-mentioned method for generating an index catalog of a book data structure based on artificial intelligence, the system comprising: A processing module is used to obtain the full text of the book, perform data preprocessing on the full text of the book, obtain the preprocessed text, perform word segmentation processing on the preprocessed text, obtain a word segmentation set, and assign a book type label according to the word segmentation set. The data preprocessing is used to perform maximum word segmentation on the full text of the book against the dictionary and remove invalid words. The word segmentation processing is used to perform word segmentation processing on the preprocessed text to improve the accuracy of word segmentation; The first-level screening module is used to screen the word set according to the assigned book type label and extract the first-level entry keywords; The secondary screening module is used to determine the secondary entry keywords and their corresponding secondary entry keywords according to the primary entry keywords, specifically including determining the secondary entry keyword search scope and then screening the secondary entry keywords; The generation module is used to generate a book index catalog based on the keywords of the first-level and second-level entries.

[0017] The present invention also relates to a computer program product, which includes a computer program. The computer program is executed by a processor to execute the index directory generation method of a book data structure based on artificial intelligence as described above.

[0018] The present invention also relates to a computer-readable storage medium, which is used to store a computer program. The computer program is executed by a processor to execute the index directory generation method of a book data structure based on artificial intelligence as described above.

[0019] The technical solution of the present invention can automatically adapt to different types of books and accurately generate a book index catalog by automatically allocating book type tags and obtaining primary entry keywords and secondary entry keywords in association with book types. DETAILED DESCRIPTION

[0020] The present invention is further described below. The following examples are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the scope of protection of the present invention. It should be pointed out that the following detailed description is exemplary and is intended to provide further description of the present invention.

[0021] Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those generally understood by those of ordinary skill in the art to which the present invention belongs. It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0022] Embodiment 1 of the present invention relates to a method for generating an index directory of a book data structure based on artificial intelligence, the method comprising the following steps: S101, obtaining the full text of the book, performing data preprocessing on the full text of the book to obtain a preprocessed text, then performing word segmentation processing on the preprocessed text to obtain a word segmentation set, and assigning a book type label according to the word segmentation set.

[0023] The data preprocessing is used to perform maximum word segmentation on the book full-text comparison dictionary and remove invalid words, and the word segmentation processing is used to perform word segmentation processing on the preprocessed text to improve the word segmentation accuracy.

[0024] The book types may include, for example, academic research, technical manuals, history and humanities, leisure and entertainment, education and learning, and the like.

[0025] Perform data preprocessing on the full text of the book, including obtaining preprocessed text ,

[0026] in, To preprocess the text, For book text, is a dictionary parameter, is the forward maximum matching function, It is a word segmentation cleaning function used to remove invalid words such as stop words.

[0027] The word segmentation processing of the preprocessed text specifically includes first obtaining the tag sequence corresponding to the preprocessed text ,

[0028] in, To preprocess the text, The token sequence corresponding to the preprocessed text indicates the word position state of each character, such as the beginning, middle, end, or a single word. is the sequence position of the word in the preprocessed text, is the normalization factor, is the number of transfer characteristic functions, For each parameter corresponding to the transfer characteristic function, is the transfer characteristic function, is the number of state characteristic functions, is the parameter corresponding to each state characteristic function, is the state characteristic function, For the The mark value of the corresponding position of the word, For the The mark value of the corresponding position of the word, is the Viterbi function. Thus, the most likely tag sequence is obtained.

[0029] Get the word set again ,

[0030] in, is a word segmentation set, is the mapping function, is the above tag sequence. Thus, by traversing the tag sequence to find the mapping relationship between tags and words, a word segmentation set is obtained.

[0031] According to the word segmentation set, the book type label is assigned, specifically including: obtaining the book type label ,

[0032] in, is the book type label, is the total number of books, is the number of books of each type, is the word at the corresponding position in the word set, For each book type, is a statistical function used to count the number of times each word appears in books of this type. is the length of the word segmentation set, The number of book types.

[0033] S102, based on the assigned book type label, filter the word set to extract the first-level entry keywords .

[0034] Taking academic research books as an example, the following first-level entry keywords are extracted: introduction, literature review, methodology, conclusion, and references. Specifically:

[0035] is a list of first-level entry keywords. is a word segmentation set, is the first-level entry keyword dictionary corresponding to the book type, It is a keyword matching function used to traverse the word segmentation set , find the corresponding first-level entry keywords.

[0036] S103, according to the first-level entry keywords, determine the corresponding second-level entry keywords. Specifically, first clarify the search scope of the second-level entry keywords, and then filter the second-level entry keywords. Taking academic research books as an example, the second-level entry keywords corresponding to the first-level entry keyword "introduction" include the specific content of the research background, the specific content of the research purpose and significance, the specific content of the research method, etc.

[0037] The above clarification of the keyword search scope of the secondary entry includes determining the The search range sequence of the second-level entry keywords corresponding to the first-level entry keywords ,

[0038] in, For the First-level entry keywords, is a word segmentation set, is the length of the word segmentation set, is the number of first-level entry keywords, It is a search function used to find the position of the first-level entry keyword in the word segmentation set. The second-level entry keyword search range is from the current first-level entry keyword to the next first-level entry keyword.

[0039] The above-mentioned screening of the secondary entry keywords specifically includes obtaining the secondary entry keywords according to the search range sequence of the secondary entry keywords ,

[0040] in, For the Second-level entry keywords, To search for the first level entry based on the keywords The search scope when there are two secondary entry keywords, is the association between the first-level entry keywords and the second-level entry keywords, Indicates the first-level entry keyword With secondary entry keywords The relationship between It is a matching function used to extract the secondary entry keywords related to the primary entry keywords.

[0041] S104, generating a book index catalog based on the primary and secondary entry keywords.

[0042] Taking academic research books as an example again, the generated book index directory is as follows: Introduction (keywords for first-level entries) • Research background (secondary entry keywords) • Research purpose and significance (secondary entry keywords) • Research methods (secondary entry keywords) Literature review (keywords for first-level entries) • Theoretical framework (keywords for secondary entries) • Findings from previous studies (secondary entry keywords) • Research gaps (secondary entry keywords) Methodology (keywords for first-level entries) • Data collection (secondary entry keywords) • Data analysis methods (secondary entry keywords) • Research Design (Secondary Entry Keyword) Results (first level entry keywords) • Data analysis results (secondary entry keywords) • Charts and statistics (secondary entry keywords) • Discussion (Secondary Entry Keywords) Conclusion (keywords for first-level entries) • Research summary (secondary entry keywords) • Research contribution (secondary entry keywords) • Future research directions (secondary entry keywords) References (keywords for first-level entries) • Books (secondary entry keywords) • Journal article (secondary entry keywords) • Online resources (secondary entry keywords) Appendix (first level entry keywords) • Questionnaire (Secondary Entry Keyword) • Data table (secondary entry keywords) • Additional Figures (Secondary Entry Keywords) The index directory may also include multi-level entry keywords, wherein the multi-level includes three levels or more. According to the method of clarifying the search scope of the secondary entry keywords and filtering the secondary entry keywords, the multi-level entry keywords are obtained by progressive execution. Embodiment 2 of the present invention relates to a system using the method for generating an index directory of a book data structure based on artificial intelligence according to embodiment 1, comprising: A processing module is used to obtain the full text of the book, perform data preprocessing on the full text of the book, obtain the preprocessed text, perform word segmentation processing on the preprocessed text, obtain a word segmentation set, and assign a book type label according to the word segmentation set. The data preprocessing is used to perform maximum word segmentation on the full text of the book against the dictionary and remove invalid words. The word segmentation processing is used to perform word segmentation processing on the preprocessed text to improve the accuracy of word segmentation; The first-level screening module is used to screen the word set according to the assigned book type label and extract the first-level entry keywords; The secondary screening module is used to determine the secondary entry keywords and their corresponding secondary entry keywords according to the primary entry keywords, specifically including determining the secondary entry keyword search scope and then screening the secondary entry keywords; The generation module is used to generate a book index catalog based on the keywords of the first-level and second-level entries.

[0043] Embodiment 3 of the present invention relates to a computer program product, which includes a computer program. The computer program is executed by a processor to execute the index directory generation method of the book data structure based on artificial intelligence in Embodiment 1.

[0044] Embodiment 4 of the present invention relates to a computer-readable storage medium, which is used to store a computer program. The computer program is executed by a processor to execute the method for generating an index catalog of a book data structure based on artificial intelligence in Embodiment 1.

[0045] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention.

Claims

1. A method for generating an index directory of a book data structure based on artificial intelligence, characterized in that: The method comprises the following steps: S101, obtaining the full text of the book, performing data preprocessing on the full text of the book, obtaining the preprocessed text, then performing word segmentation processing on the preprocessed text, obtaining a word segmentation set, and assigning a book type label according to the word segmentation set. The data preprocessing is used to perform maximum word segmentation on the full text of the book against the dictionary and remove invalid words. The word segmentation processing is used to perform word segmentation processing on the preprocessed text to improve the word segmentation accuracy; S102, screening the word set according to the assigned book type label to extract the first-level entry keywords; S103, determining the corresponding secondary entry keywords according to the primary entry keywords, specifically including clarifying the search scope of the secondary entry keywords, and then screening the secondary entry keywords; S104, generating a book index catalog based on the primary and secondary entry keywords.

2. The method for generating an index directory of a book data structure based on artificial intelligence according to claim 1, characterized in that: In step S101, data preprocessing is performed on the full text of the book, specifically including: Get preprocessed text , ； in, To preprocess the text, For book text, is a dictionary parameter, is the forward maximum matching function, It is a word segmentation cleaning function used to remove invalid words such as stop words.

3. The method for generating an index directory of a book data structure based on artificial intelligence according to claim 2, characterized in that: In step S101, the word segmentation processing of the preprocessed text specifically includes: First get the token sequence corresponding to the preprocessed text , ； in, To preprocess the text, The token sequence corresponding to the preprocessed text indicates the word position state of each character, such as the beginning, middle, end, or a single word. is the sequence position of the word in the preprocessed text, is the normalization factor, is the number of transfer characteristic functions, For each parameter corresponding to the transfer characteristic function, is the transfer characteristic function, is the number of state characteristic functions, is the parameter corresponding to each state characteristic function, is the state characteristic function, For the The mark value of the corresponding position of the word, For the The mark value of the corresponding position of the word, is the Viterbi function; Get the word set again , ； in, is a word segmentation set, is the mapping function, is the above tag sequence; thus, by traversing the tag sequence to find the mapping relationship between tags and words, a word segmentation set is obtained.

4. The method for generating an index directory of a book data structure based on artificial intelligence according to claim 2, characterized in that: In step S101, the book type label is assigned according to the word segmentation set, which specifically includes: Get book type tags , ； in, is the book type label, is the total number of books, is the number of books of each type, is the word at the corresponding position in the word set, For each book type, is a statistical function used to count the number of times each word appears in books of this type. is the length of the word segmentation set, The number of book types.

5. The method for generating an index directory of a book data structure based on artificial intelligence according to claim 1, characterized in that: In step S102, the word segmentation set is screened according to the assigned book type label to extract the first-level entry keywords, which specifically includes: Extract first-level entry keywords , ； is a list of first-level entry keywords. is a word segmentation set, A first-level keyword dictionary corresponding to each book type, It is a keyword matching function used to traverse the word segmentation set , find the corresponding first-level entry keywords.

6. The method for generating an index directory of a book data structure based on artificial intelligence according to claim 1, characterized in that: In step S103, the search scope of the secondary entry keywords is clarified, including: Determine The search range sequence of the second-level entry keywords corresponding to the first-level entry keywords , ； in, For the First-level entry keywords, is a word segmentation set, is the length of the word segmentation set, is the number of first-level entry keywords, is a search function used to find the position of the first-level entry keyword in the word segmentation set. The search range of the second-level entry keyword is from the current first-level entry keyword to the next first-level entry keyword. Filter secondary entry keywords, including: Get the secondary entry keywords according to the search range sequence of the secondary entry keywords , ； in, For the Second-level entry keywords, To search for the first level entry based on the keywords The search scope when there are two secondary entry keywords, is the association between the first-level entry keywords and the second-level entry keywords, Indicates the first-level entry keyword With secondary entry keywords The relationship between It is a matching function used to extract the secondary entry keywords related to the primary entry keywords.

7. The method for generating an index directory of a book data structure based on artificial intelligence according to claim 6, characterized in that: Including multi-level entry keywords, the multi-level includes three levels or more, according to the method of clarifying the search scope of the secondary entry keywords and screening the secondary entry keywords, progressive execution is performed to obtain the multi-level entry keywords.

8. A system using the method for generating an index directory of a book data structure based on artificial intelligence according to any one of claims 1 to 7, characterized in that: The system comprises: A processing module is used to obtain the full text of the book, perform data preprocessing on the full text of the book, obtain the preprocessed text, perform word segmentation processing on the preprocessed text, obtain a word segmentation set, and assign a book type label according to the word segmentation set. The data preprocessing is used to perform maximum word segmentation on the full text of the book against the dictionary and remove invalid words. The word segmentation processing is used to perform word segmentation processing on the preprocessed text to improve the accuracy of word segmentation; The first-level screening module is used to screen the word set according to the assigned book type label and extract the first-level entry keywords; The secondary screening module is used to determine the secondary entry keywords and their corresponding secondary entry keywords according to the primary entry keywords, specifically including clarifying the secondary entry keyword search scope and then screening the secondary entry keywords; The generation module is used to generate a book index catalog based on the keywords of the first-level and second-level entries.

9. A computer program product, characterized in that The computer program product includes a computer program, which is executed by a processor and is used to execute the index directory generation method of a book data structure based on artificial intelligence as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and the computer program is executed by a processor to execute an index directory generation method for a book data structure based on artificial intelligence as described in any one of claims 1 to 7.

Images