High-adaptability literature subject term indexing method, system and equipment and storage medium

Abstract

The invention belongs to the technical field of literature processing, particularly relates to a high-adaptability literature subject term indexing method, system and device and a storage medium, and aims at solving the problem that subject term indexing is prone to being missing in the related technology. The method comprises the steps of obtaining a to-be-indexed literature and a preset subject word table, and performing adaptive expansion on an entry word of each subject word in the subject word table to obtain an expanded entry word set; integrating the extended entry word set, and constructing an indexing word bank which comprises a plurality of lexical elements; on the basis of the indexing word bank, a word segmentation device is constructed, and the word segmentation device is used for segmenting the content of the to-be-indexed literature into lexical elements existing in the indexing word bank; and indexing the to-be-indexed literature based on the word segmentation device and the indexing word bank to obtain a subject term indexing result. According to the method, the related subject terms in the literature can be more comprehensively indexed.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of document processing, and particularly relates to a high-adaptability document subject heading indexing method, system, device and storage medium. BACKGROUND

[0002] Document indexing is a key link in information retrieval and knowledge management, and its purpose is to extract and mark the core subject headings of document content to provide accurate indexing basis for subsequent storage, classification and retrieval. In the practice of automatic indexing, keyword matching based on a preset subject heading table or a controlled vocabulary table is a mainstream technical path.

[0003] However, the automatic indexing method in the prior art usually relies on direct string matching of the content of the document to be indexed and the entries in the subject heading table. This approach has obvious limitations: when dealing with languages such as English with rich morphology, words with the same root often appear in different forms due to case, singular and plural forms, etc. For example, the standard entry in the subject heading table may be "Algorithm", while the word in the document may be "algorithm" or "algorithms". The traditional direct matching method cannot recognize these entries as the same subject concept, which easily leads to missing subject heading indexing, seriously affecting the recall rate and accuracy of indexing, and thus reducing the efficiency of the entire document retrieval system. SUMMARY

[0004] In order to solve the above-mentioned problems in the prior art, i.e., the related art is prone to missing subject heading indexing, the present application provides, in a first aspect, a high-adaptability document subject heading indexing method, comprising:

[0005] obtaining a document to be indexed and a preset subject heading table, and adaptively expanding the entry word of each subject heading in the subject heading table to obtain an expanded entry word set;

[0006] integrating the expanded entry word set to construct an indexing word library, the indexing word library comprising a plurality of word units;

[0007] constructing a word segmenter based on the indexing word library, the word segmenter being used to segment the content of the document to be indexed into word units present in the indexing word library;

[0008] indexing the document to be indexed based on the word segmenter and the indexing word library to obtain a subject heading indexing result.

[0009] In some preferred embodiments, the adaptive expansion of the entry word of each subject heading in the subject heading table comprises:

[0010] initializing a temporary word set corresponding to each entry word in the subject thesaurus respectively;

[0011] adding the entry word itself to the corresponding temporary word set;

[0012] transforming the entry word according to preset morphological extension rules to generate one or more derived words;

[0013] adding all generated derived words to the temporary word set corresponding to the entry word.

[0014] In some preferred embodiments, the method further comprises:

[0015] performing a set union of the extended entry words of all subject words as the indexing word library.

[0016] In some preferred embodiments, the transforming the entry word according to preset morphological extension rules to generate one or more derived words comprises:

[0017] creating an empty array as a temporary array for saving the extended entry words and storing the current subject word in the array;

[0018] determining whether the current subject word is all uppercase, and if not, storing the lowercase form of the subject word in the temporary array;

[0019] determining whether the current subject word is in singular form, and if so, storing the plural form of the subject word in the temporary array;

[0020] determining whether the current subject word is in plural form, and if so, storing the singular form of the subject word in the temporary array;

[0021] returning the temporary array as the derived words of the current subject word.

[0022] In some preferred embodiments, the indexing the document to be indexed based on the word segmenter and the indexing word library comprises:

[0023] performing word segmentation processing on the content of the document to be indexed using the word segmenter to obtain a word segmentation result;

[0024] determining the subject word corresponding to the content of the document to be indexed according to the correspondence between the word segmentation result and the indexing word library.

[0025] In some preferred embodiments, the determining the subject word corresponding to the content of the document to be indexed according to the correspondence between the word segmentation result and the indexing word library comprises:

[0026] creating an empty array as a temporary array for saving the subject word indexing result;

[0027] An intersection set is obtained by intersecting the segmentation result with all the words in the index word library;

[0028] For each element in the intersection set, a subject word to which the element belongs is found by referring to the index word library, and is saved to the temporary array;

[0029] The temporary array is returned as the subject word indexing result of the current sentence.

[0030] In some preferred embodiments, the finding of the subject word to which each element in the intersection set belongs by referring to the index word library comprises:

[0031] Each intersection word is mapped to a corresponding subject word by querying the mapping relationship of the index word library, wherein the index word library stores the association relationship between the extended entry word and the subject word.

[0032] The present application proposes, in a second aspect, a high-adaptability document subject word indexing system for performing the high-adaptability document subject word indexing method as described in the first aspect, comprising:

[0033] A data acquisition module is configured to acquire a document to be indexed and a preset subject word table, and to perform adaptive expansion on the entry word of each subject word in the subject word table to obtain a set of expanded entry words;

[0034] A word library construction module is configured to integrate the set of expanded entry words to construct an index word library, wherein the index word library comprises a plurality of word units;

[0035] A segmentation module is configured to construct a segmenter based on the index word library, wherein the segmenter is configured to segment the content of the document to be indexed into word units present in the index word library;

[0036] A document indexing module is configured to index the document to be indexed based on the segmenter and the index word library to obtain a subject word indexing result.

[0037] The present application proposes, in a third aspect, an electronic device comprising:

[0038] At least one processor;

[0039] and a memory in communication connection with the at least one processor;

[0040] The memory stores instructions executable by the processor, and the instructions are configured to be executed by the processor to implement the high-adaptability document subject word indexing method as described in the first aspect.

[0041] The application provides a computer readable storage medium in a fourth aspect. The computer readable storage medium stores computer instructions for being executed by a computer to implement the high adaptability document subject heading indexing method in the first aspect.

[0042] The application has the following beneficial effects:

[0043] Compared with the prior art, the application constructs a more comprehensive indexing word library by adaptively expanding the entry words in the subject heading table in terms of case, singular and plural forms, etc., so that various writing variants of the keywords in the document can be recognized and captured, indexing omission caused by incomplete string matching is effectively avoided, and relevant subject headings can be fully indexed.

[0044] Meanwhile, the expansion process of the application is based on the preset subject heading table, and all the generated derived words are associated with the explicit subject headings, so that ambiguities and errors (for example, different semantic words are incorrectly reduced to the same word root) caused by general word stem extraction or morphological reduction algorithms are avoided, and each matched word element can be accurately mapped back to the correct and unique original subject heading. BRIEF DESCRIPTION OF DRAWINGS

[0045] Other features, objects and advantages of the application will become more apparent from the following detailed description of non-limiting embodiments with reference to the attached drawings:

[0046] Figure 1 is a flowchart of a high adaptability document subject heading indexing method provided by an embodiment of the application;

[0047] Figure 2 is a framework diagram of a high adaptability document subject heading indexing system provided by an embodiment of the application;

[0048] Figure 3 is a structural diagram of a computer system of a server for implementing the method, system and electronic device embodiments of the application. DETAILED DESCRIPTION

[0049] The application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. In addition, it should be noted that only the parts related to the application are shown in the drawings for convenience of description.

[0050] It should be noted that the embodiments in the application and the features in the embodiments can be combined with each other without conflict. The application will be described in detail below with reference to the drawings and embodiments.

[0051] In order to more clearly illustrate the high adaptability document subject heading indexing method of the present application, the following will be combined with Figure 1 The steps in the embodiment of the present application will be described in detail.

[0052] The first embodiment of the present application provides a high adaptability document subject heading indexing method, comprising steps S10-S40, and each step is described in detail as follows:

[0053] Step S10, obtaining the to-be-indexed document and the preset subject heading table, and adaptively expanding the entry word of each subject heading in the subject heading table to obtain an expanded entry word set;

[0054] Step S20, integrating the expanded entry word set to construct an indexing word library, wherein the indexing word library comprises a plurality of word units;

[0055] Step S30, constructing a word segmenter based on the indexing word library, wherein the word segmenter is used to segment the content of the to-be-indexed document into word units existing in the indexing word library;

[0056] Step S40, indexing the to-be-indexed document based on the word segmenter and the indexing word library to obtain a subject heading indexing result.

[0057] In this embodiment, the word table is first expanded, specifically by obtaining a preset subject heading table, which defines the core concepts of each technical field and the corresponding entry words. For each entry word under each subject heading in the subject heading table, an adaptive expansion operation is performed to generate an expanded entry word set containing multiple common writing forms of the entry word.

[0058] For example, for the entry word "Algorithm", the expanded entry word set can include "algorithm", "algorithms", etc.

[0059] Then, after the expansion of all entry words is completed, the expanded entry word sets generated by all subject headings are merged and duplicate items are removed (i.e. the union set is obtained), thereby constructing an indexing word library with comprehensive content and high morphological adaptability.

[0060] Then, based on the constructed indexing word library, a special word segmenter is constructed. The word segmenter is configured to strictly segment the text according to the word entries in the indexing word library. The special word segmenter can ensure that the segmented word units (tokens) are all valid word entries existing in the indexing word library.

[0061] Finally, the content of a document to be indexed (e.g., sentence by sentence or section by section) is processed using the aforementioned specialized tokenizer and thesaurus. The tokenizer cuts the document text into a series of tokens, and then maps these tokens to the thesaurus to find the actual appearing terms in the text that exist in the thesaurus. According to the correspondence between these terms and the original subject headings, the subject heading identifiers that should be assigned to the document are finally determined.

[0062] In some embodiments, after obtaining the document to be indexed, it is first cleaned. The cleaning process includes: removing all URL links, email addresses and non-text characters (such as special symbols, formula marks) in the document; stripping all HTML or XML tags in the document content; removing common stop words that do not contribute to the subject indexing (e.g., "the", "a", "is" in English, "of", "in" in Chinese, etc.), and the stop word list can be customized according to the specific application field.

[0063] In some embodiments, after obtaining the subject heading indexing result (a list of subject headings), the frequency (Term Frequency, TF) of each subject heading appearing in the document to be indexed (through its corresponding extended entry word) is counted. At the same time, the importance weight of each subject heading (e.g., core subject heading weight high, edge subject heading weight low) can be combined. Finally, according to the product of the frequency and the weight, the indexing result is sorted in descending order, and optionally only the top-K subject headings with the highest scores are output as the final indexing result.

[0064] In some embodiments, for each entry word, a lemmatization algorithm based on a dictionary (such as WordNetLemmatizer) is used to obtain its lemma. For example, for the entry words "computers", "computing", "computed", their lemmas are all "computer". The entry word itself and its lemma (if not the same) are both added to the corresponding temporary word set. At the same time, all entry words in the subject heading table are lemmatized to construct a reverse mapping table from "lemma" to "subject heading". In the subsequent indexing step, the lemmatization is also performed after tokenizing the document to be indexed, and the matching is performed through the lemma, thereby covering a wider range of word form variants.

[0065] Specifically, in the above embodiments, the entry words of each subject heading in the subject heading table are adaptively expanded, including:

[0066] For each entry word in the subject thesaurus, initialize a temporary word set corresponding to the entry word respectively; add the entry word itself to the corresponding temporary word set; convert the entry word according to a preset morphological expansion rule to generate one or more derived words; add all generated derived words to the temporary word set corresponding to the entry word.

[0067] More specifically, the method further comprises:

[0068] Take the union of all expanded entry words of the subject words as the index word library.

[0069] More specifically, the converting the entry word according to a preset morphological expansion rule to generate one or more derived words comprises:

[0070] Create an empty array as a temporary array for saving the expanded entry words, and store the current subject word in the array; determine whether the current subject word is all uppercase, and if not, store the lowercase form of the current subject word in the temporary array; determine whether the current subject word is in singular form, and if so, store the plural form of the current subject word in the temporary array; determine whether the current subject word is in plural form, and if so, store the singular form of the current subject word in the temporary array; return the temporary array as the derived words of the current subject word.

[0071] More specifically, the indexing the to-be-indexed document based on the word segmenter and the index word library comprises:

[0072] Segment the content of the to-be-indexed document using the word segmenter to obtain a segmentation result;

[0073] Determine the subject word corresponding to the content of the to-be-indexed document according to the correspondence between the segmentation result and the index word library.

[0074] More specifically, the determining the subject word corresponding to the content of the to-be-indexed document according to the correspondence between the segmentation result and the index word library comprises:

[0075] Create an empty array as a temporary array for saving the subject word indexing result;

[0076] Take the intersection of the segmentation result and all words in the index word library to obtain an intersection word set;

[0077] For each element in the intersection word set, refer to the index word library to find the subject word to which it belongs and save it to the temporary array;

[0078] Return the temporary array as the subject word indexing result of the current sentence.

[0079] More specifically, the reference to the index library for each element in the intersection word set to find the subject heading to which it belongs includes:

[0080] Each intersection word is mapped to the corresponding subject heading by querying the mapping relationship of the index library, wherein the index library stores the association relationship between the extended portal word and the subject heading.

[0081] Further, the second embodiment of the present application proposes a high adaptability literature subject heading indexing system for executing the above-mentioned high adaptability literature subject heading indexing method, comprising:

[0082] The data acquisition module 210 is configured to acquire the literature to be indexed and a preset subject heading table, and adaptively expand the portal word of each subject heading in the subject heading table to obtain an extended portal word set.

[0083] The word library construction module 220 is configured to integrate the extended portal word set to construct an index library, wherein the index library comprises a plurality of word units.

[0084] The word segmentation module 230 is configured to construct a word segmenter based on the index library, wherein the word segmenter is configured to segment the content of the literature to be indexed into the word units existing in the index library.

[0085] The literature indexing module 240 is configured to index the literature to be indexed based on the word segmenter and the index library to obtain a subject heading indexing result.

[0086] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process and related description of the system described above can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0087] It should be noted that the high adaptability literature subject heading indexing method and system provided in the above embodiments are only exemplified by the division of the above functional modules, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the modules or steps in the embodiments of the present application can be further divided or combined, for example, the modules in the above embodiments can be combined into one module, or can be further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present application are only for distinguishing the modules and steps, and should not be considered as an improper limitation of the present application.

[0088] The third embodiment of the present application is an electronic device, comprising:

[0089] at least one processor;

[0090] and a memory in communication connection with the at least one processor.

[0091] The memory stores instructions executable by the processor, and the instructions are executed by the processor to implement the high adaptability document subject word indexing method.

[0092] The computer readable storage medium of the fourth embodiment of the application stores computer instructions, and the computer instructions are executed by the computer to implement the high adaptability document subject word indexing method.

[0093] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the storage device and the processing device described above and the related descriptions can refer to the corresponding processes in the foregoing method embodiments, and will not be described here.

[0094] Reference will be made to the accompanying drawings Figure 3 which shows a structural schematic diagram of a computer system of a server for implementing the system, method and electronic device embodiments of the present application. Figure 3 The server shown is merely an example, and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0095] As shown in Figure 3 , the computer system includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 302 or programs loaded from a storage portion 308 to a random access memory (RAM) 303. In the RAM 303, various programs and data required for system operation are also stored. The CPU 301, the ROM 302 and the RAM 303 are connected to each other through a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0096] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, etc.; an output section 307 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as necessary. A removable recording medium 311 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 310 as necessary, so that a computer program read out therefrom is installed in the storage section 308 as necessary.

[0097] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 309, and / or installed from the removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, the above-described functions defined in the methods of the present application are performed. It should be noted that the computer readable medium of the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device. In the present application, the computer readable signal medium can include a data signal carried in a baseband or as part of a carrier wave, in which the computer readable program code is carried. Such a propagated data signal can take any of a variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer readable signal medium can also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate or transport a program for use by or in connection with an instruction execution system, apparatus or device. Program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the above.

[0098] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0099] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0100] The terms "first", "second", etc. are used to distinguish between similar objects, and are not used to describe or indicate a particular order or sequence.

[0101] The terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0102] The technical scheme of the present application has been described in combination with the preferred embodiments shown in the drawings, but it is easy for those skilled in the art to understand that the protection scope of the present application is obviously not limited to these specific embodiments. Those skilled in the art can make equivalent changes or replacements to the related technical features without departing from the principles of the present application, and the technical schemes after the changes or replacements will all fall within the protection scope of the present application.

Claims

1. A highly adaptable document subject heading indexing method, characterized in that, include: Obtain the documents to be indexed and a preset thesaurus, and adaptively expand the entry terms of each thesaurus to obtain an expanded set of entry terms; The extended entry word set is integrated to construct an indexing lexicon, which includes multiple word elements; Based on the indexing lexicon, a word segmenter is constructed, which is used to segment the content of the document to be indexed into word units existing in the indexing lexicon; Based on the word segmenter and the indexing lexicon, the document to be indexed is indexed to obtain subject term indexing results.

2. The highly adaptable document subject heading indexing method according to claim 1, characterized in that, The adaptive expansion of the entry term for each subject term in the thesaurus includes: For each entry word in the thesaurus, a corresponding temporary word set is initialized. Add at least one entry word to the corresponding temporary word set; The entry word is transformed according to a preset morphological expansion rule to generate one or more derived words; Add all the generated derived words to the temporary word set corresponding to the entry word.

3. The highly adaptable document subject heading indexing method according to claim 2, characterized in that, The method further includes: The union of all extended entry terms of the subject terms is used as the indexing thesaurus.

4. The highly adaptable document subject heading indexing method according to claim 2, characterized in that, The process of transforming the entry word according to preset morphological expansion rules to generate one or more derived words includes: Create an empty array as a temporary array to store the expanded entry words, and store the current topic words into this array; Determine whether the current keyword is all uppercase; if not, store its lowercase form in the temporary array. Determine whether the current keyword is singular; if so, store its plural form in the temporary array. Determine whether the current keyword is in plural form; if so, store its singular form in the temporary array. The temporary array is returned as the derived words of the current topic word.

5. The highly adaptable document subject heading indexing method according to claim 1, characterized in that, The indexing of the documents to be indexed, based on a word segmenter and an indexing thesaurus, includes: The word segmenter is used to segment the content of the document to be indexed to obtain the segmentation result. Based on the correspondence between the word segmentation results and the indexing thesaurus, the subject terms corresponding to the content of the document to be indexed are determined.

6. The highly adaptable document subject heading indexing method according to claim 5, characterized in that, The step of determining the subject terms corresponding to the content of the document to be indexed based on the correspondence between the word segmentation results and the indexing thesaurus includes: Create an empty array as a temporary array to store the subject heading indexing results; The intersection of the word segmentation result and all words in the indexing lexicon is obtained to obtain the intersection word set; For each element in the intersection word set, refer to the indexing lexicon to find its corresponding subject term and save it to the temporary array; The temporary array is returned as the subject heading indexing result for the current sentence.

7. The highly adaptable document subject heading indexing method according to claim 6, characterized in that, For each element in the intersection term set, the process of searching upwards for its corresponding subject term with reference to the indexing thesaurus includes: By querying the mapping relationship in the indexing thesaurus, each intersection term is mapped to the corresponding subject term. The indexing thesaurus stores the association relationship between extended entry terms and subject terms.

8. A highly adaptable document subject heading indexing system, used to execute the highly adaptable document subject heading indexing method according to any one of claims 1-7, characterized in that, The system includes: The data acquisition module is used to acquire the documents to be indexed and the preset thesaurus, and to adaptively expand the entry words of each subject word in the thesaurus to obtain an expanded entry word set; The lexicon building module is used to integrate the extended entry word set to build an indexing lexicon, which includes multiple word elements; The word segmentation module is used to construct a word segmenter based on the indexing lexicon. The word segmenter is used to segment the content of the document to be indexed into word units that exist in the indexing lexicon. The document indexing module is used to index the document to be indexed based on the word segmenter and the indexing lexicon to obtain subject term indexing results.

9. An electronic device, characterized in that, include: At least one processor; and a memory communicatively connected to at least one of the processors; The memory stores instructions that can be executed by the processor to implement the highly adaptable document subject heading indexing method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are executed by the computer to implement the highly adaptable document subject heading indexing method according to any one of claims 1-7.

Images