Document label extraction method and device, computer equipment and storage medium

Through word segmentation processing and weight vector calculation of document text and framework, document labels are automatically generated, which solves the problems of timeliness and maintenance costs of tag systems in the existing technology, and realizes efficient and accurate document label generation and update.

CN119026600APending Publication Date: 2024-11-26FAW JIEFANG AUTOMOTIVE CO
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411222544.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-02
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing automated document label generation methods are difficult to maintain the timeliness of the label system, and the update and maintenance cost is high, so it is unable to effectively deal with the challenges brought about by the expansion of knowledge system and the emergence of new fields.

Method used

By obtaining the document text and framework, word segmentation processing and word frequency statistics, combined with weight vector calculation, we automatically identify keywords and generate tags, ensuring the timeliness and dynamics of tags, and reducing dependence on manual annotations.

Benefits of technology

It improves the speed and efficiency of document tag generation, improves the relevance and accuracy of keywords and tags, reduces update and maintenance costs, and ensures the timeliness and user experience of tags.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119026600A_ABST
    Figure CN119026600A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of data mining, in particular to a document tag extraction method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining a document text and a document frame, wherein the document frame comprises at least one hierarchical title; performing word segmentation processing on the document body and the at least one hierarchical title to obtain at least one independent word; obtaining the word frequency and the weight vector of each independent vocabulary; determining keywords of a document framework and keywords of a document body according to the word frequency and the weight vector of the at least one independent vocabulary; and determining a document label according to the keyword of the document framework and the keyword of the document body. By adopting the method, the updating and maintaining cost can be reduced while the timeliness of a label system is ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data mining technology, and in particular to a document tag extraction method, device, computer equipment, computer-readable storage medium and computer program product. Background Art

[0002] In the era of information explosion, document management and classification have become key links to improve work efficiency and promote knowledge sharing and retrieval. As a highly summarized and indexed identifier of the document body, the automatic generation technology of document tags is of great significance for the rapid classification, retrieval and personalized recommendation of documents. Traditionally, the generation of document tags mainly relies on manual annotation, which is not only inefficient but also difficult to cope with the challenges of large-scale data processing.

[0003] At present, the mainstream method of automatic document tag generation usually adopts the framework of "keyword extraction-domain mapping-focus domain determination-label generation". First, keywords are extracted from documents through natural language processing technology (NLP). Then, the keywords are mapped to specific knowledge domains using domain knowledge bases or pre-trained models. The focus domain to which the document belongs is determined by analyzing the relevance between the domain and the document body. Finally, based on the information of the focus domain, combined with preset label generation rules or machine learning models, corresponding labels are generated for the document. Although this method has achieved the automatic generation of document labels to a certain extent, with the continuous expansion of the knowledge system and the continuous emergence of new fields, the original domain knowledge base and label system will often become outdated and difficult to accurately reflect the current knowledge status and characteristics of the times. However, periodic maintenance will bring about a large cost investment.

[0004] Therefore, there is an urgent need for a document tag extraction method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can reduce update and maintenance costs while ensuring the timeliness of the tag system. Summary of the invention

[0005] Based on this, it is necessary to provide a document label extraction method, device, computer equipment, computer-readable storage medium and computer program product that can reduce the update and maintenance costs while ensuring the timeliness of the label system in response to the above technical problems.

[0006] In a first aspect, the present application provides a document tag extraction method, comprising:

[0007] Obtain a document body and a document frame, wherein the document frame includes at least one hierarchical title;

[0008] Performing word segmentation processing on the document body and the at least one hierarchical title to obtain at least one separate word;

[0009] Get the frequency and weight vector of each individual word;

[0010] Determining keywords of the document framework and keywords of the document body based on the word frequency and weight vector of the at least one separate word;

[0011] A document tag is determined according to the keywords of the document frame and the keywords of the document body.

[0012] In one embodiment, obtaining the weight vector of each individual word includes:

[0013] Respectively obtaining the importance of the document body and the at least one hierarchical title;

[0014] Constructing a judgment matrix according to the importance of the document body and the at least one hierarchical title;

[0015] Obtaining the maximum eigenvalue and corresponding eigenvector of the judgment matrix;

[0016] A weight vector for each individual word is obtained according to the maximum eigenvalue and the corresponding eigenvector.

[0017] In one embodiment, before obtaining the maximum eigenvalue and the corresponding eigenvector of the judgment matrix, the method further includes:

[0018] Calculating the consistency index and random consistency index of the judgment matrix;

[0019] Obtaining a consistency ratio of the judgment matrix according to the consistency index and the random consistency index;

[0020] When the consistency ratio is lower than a preset value, it is determined that the judgment matrix meets the consistency requirement, and a step of obtaining a maximum eigenvalue and a corresponding eigenvector of the judgment matrix is ​​performed.

[0021] In one embodiment, determining the keywords of the document framework according to the word frequency and weight vector of the at least one individual word includes:

[0022] Performing deduplication processing and stop word removal processing on the at least one level title to obtain at least one first keyword of the document framework;

[0023] Matching at least one first keyword with at least one separate word to obtain a word frequency and a weight vector corresponding to the at least one first keyword;

[0024] A weight of at least one first keyword is determined according to the word frequency and the weight vector of at least one first keyword.

[0025] In one embodiment, determining keywords of the document body according to the word frequency and weight vector of the at least one individual word includes:

[0026] Using a large language model, obtaining a summary of the document body;

[0027] Performing word segmentation and stop word removal on the abstract to obtain at least one second keyword of the document body;

[0028] Matching at least one second keyword with at least one separate word to obtain a word frequency and a weight vector corresponding to the at least one second keyword;

[0029] The weight of at least one second keyword is determined according to the word frequency and the weight vector of at least one second keyword.

[0030] In one embodiment, determining the document tag according to the keywords of the document frame and the keywords of the document body includes:

[0031] According to the weight of the at least one first keyword and the weight of the second keyword, the at least one first keyword and the second keyword are sorted to obtain a sorting result;

[0032] According to the sorting result, a document label is determined.

[0033] In a second aspect, the present application also provides a document tag extraction device, comprising:

[0034] An acquisition module, used for acquiring a document body and a document frame, wherein the document frame includes at least one hierarchical title;

[0035] A processing module, configured to perform word segmentation processing on the document body and the at least one hierarchical title to obtain at least one separate word;

[0036] The processing module is also used to obtain the word frequency and weight vector of each individual word;

[0037] A keyword extraction module, used to determine keywords of the document framework and keywords of the document body according to the word frequency and weight vector of the at least one individual word;

[0038] The tag extraction module is used to determine the document tag according to the keywords of the document frame and the keywords of the document body.

[0039] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0040] Obtain a document body and a document frame, wherein the document frame includes at least one hierarchical title;

[0041] Performing word segmentation processing on the document body and the at least one hierarchical title to obtain at least one separate word;

[0042] Get the frequency and weight vector of each individual word;

[0043] Determining keywords of the document framework and keywords of the document body based on the word frequency and weight vector of the at least one separate word;

[0044] A document tag is determined according to the keywords of the document frame and the keywords of the document body.

[0045] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0046] Obtain a document body and a document frame, wherein the document frame includes at least one hierarchical title;

[0047] Performing word segmentation processing on the document body and the at least one hierarchical title to obtain at least one separate word;

[0048] Get the frequency and weight vector of each individual word;

[0049] Determining keywords of the document framework and keywords of the document body based on the word frequency and weight vector of the at least one separate word;

[0050] A document tag is determined according to the keywords of the document frame and the keywords of the document body.

[0051] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0052] Obtain a document body and a document frame, wherein the document frame includes at least one hierarchical title;

[0053] Performing word segmentation processing on the document body and the at least one hierarchical title to obtain at least one separate word;

[0054] Get the frequency and weight vector of each individual word;

[0055] Determining keywords of the document framework and keywords of the document body based on the word frequency and weight vector of the at least one separate word;

[0056] A document tag is determined according to the keywords of the document frame and the keywords of the document body.

[0057] The above-mentioned document tag extraction method, device, computer equipment, computer-readable storage medium and computer program product, by automatically processing the document body and frame, do not need to read and annotate one by one manually, which greatly improves the speed and efficiency of document tag generation. Through word segmentation and word frequency statistics, combined with the calculation of weight vectors, key words in the document can be more accurately identified, thereby improving the relevance and accuracy of keywords and final tags. As the document body is updated, the system can re-segment and extract keywords to ensure the timeliness and dynamism of the tags. It reduces the reliance on professional manual annotation, thereby reducing long-term operation and maintenance costs. Users can use automatically generated tags to quickly locate documents of interest, which improves the convenience of retrieval and user experience. Compared with traditional manual methods, automation technology can significantly reduce the cost of update and maintenance. The document classification management model uses classification, labeling, version control and other means through a systematic approach to improve the reliability and efficiency of documents. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 An application environment diagram of a document tag extraction method in an embodiment;

[0060] Figure 2 A schematic diagram of a flow chart of a document tag extraction method in an embodiment;

[0061] Figure 3 A schematic diagram of a flow chart of a document tag extraction method in another embodiment;

[0062] Figure 4 is a structural block diagram of a document tag extraction device in one embodiment;

[0063] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0065] The document tag extraction method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0066] Server 104 obtains a document body and a document frame, wherein the document frame includes at least one hierarchical title; performs word segmentation processing on the document body and at least one hierarchical title to obtain at least one individual word; obtains the word frequency and weight vector of each individual word; determines the keywords of the document frame and the keywords of the document body based on the word frequency and weight vector of at least one individual word; determines the document tag based on the keywords of the document frame and the keywords of the document body.

[0067] Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0068] In an exemplary embodiment, Figure 2 As shown, a document tag extraction method is provided, which is applied to Figure 1 Taking the server in as an example, the method includes the following steps S202 to S210.

[0069] in:

[0070] Step S202, obtaining the document body and the document frame, wherein the document frame includes at least one hierarchical title.

[0071] Specifically, the document skeleton refers to the structured part of the document, which usually includes hierarchical information such as titles and subtitles. These elements help readers quickly grasp the organization and main content of the document. The hierarchical titles provide a high-level overview of the document content. For example, the level 0 title may be the main title, and the level 1 and level 2 titles represent different levels of subtopics.

[0072] The document body refers to the text content of the document excluding the title. It is the main part of the document, containing detailed information and arguments. The document framework refers to the organization of these contents.

[0073] Step S204: perform word segmentation processing on the document body and at least one level title to obtain at least one individual word.

[0074] Specifically, word segmentation is a key step in text preprocessing, which involves segmenting continuous text strings into meaningful units, which can be words, phrases, or other meaningful text fragments. Through word segmentation, the text is broken down into separate words, which helps further text analysis and processing, such as keyword extraction, word frequency statistics, etc. Word segmentation takes into account the characteristics of natural language. For example, words in languages ​​such as Chinese and Japanese are not separated by obvious spaces, so word segmentation algorithms are needed to identify word boundaries. Word segmentation makes it more feasible to extract useful information from text, providing a basis for subsequent tasks such as information retrieval, text mining, and natural language understanding. Individual words are the basis for keyword identification. By analyzing the frequency and context of word usage, important concepts and topics in the document can be identified.

[0075] Step S206, obtaining the word frequency and weight vector of each individual word.

[0076] Specifically, word frequency (TF) refers to the number of times a word appears in a document. This is a basic indicator for evaluating the importance of a word in a document. In some cases, word frequency statistics can be performed at the sentence level, paragraph level, or chapter level to meet different analysis needs.

[0077] The weight vector is usually associated with specific properties of the vocabulary, such as the position of the vocabulary in the document, part of speech, relevance to the topic, etc. The weight vector can enhance the importance of certain words to more accurately reflect their semantic contribution in the document.

[0078] TF-IDF is a common weight calculation method, where TF stands for term frequency and IDF is the inverse document frequency index, which is used to evaluate the general importance of vocabulary. TF-IDF can highlight words that are important in a specific document but uncommon in the corpus.

[0079] Step S208: determining keywords of the document framework and keywords of the document body according to the word frequency and weight vector of at least one individual word.

[0080] Specifically, the document framework usually includes structural elements such as titles and subtitles, which summarize the main content and structure of the document. By analyzing the word frequency and weight vectors of the words in these elements, the key words of the document framework can be identified. The body of the document is the main part of the document, which contains detailed information and the elaboration of the argument. After the body content is segmented, the key words in the body can be identified by analyzing the word frequency and weight vector of each word. Combining the word frequency and weight vector of the word, a comprehensive score can be calculated to evaluate the suitability of the word as a keyword. The word with the highest score is usually selected as the keyword of the document. The weight vector may contain weights of multiple dimensions, such as position weights that may increase the importance of words in the title, part-of-speech weights that may increase the importance of nouns and verbs, and topic relevance weights that may increase the importance of words closely related to the topic of the document.

[0081] Step S210, determining the document tag according to the keywords of the document frame and the keywords of the document body.

[0082] Specifically, first, keywords extracted from the document framework (such as title, subtitle) and the document body are integrated. The keywords in these two parts together form the semantic basis of the document. Keywords in the document framework usually have higher weights because they summarize the main content and structure of the document. Keywords in the body provide more in-depth subject information. Each keyword may be assigned different weights according to its importance in the document. For example, keywords appearing in the title may have higher weights than common words in the body. In the process of integrating keywords, it is necessary to remove duplicate words to ensure that each keyword is unique and avoid redundancy of tags. The most representative keywords are selected based on factors such as weight, frequency of occurrence and relevance. This can be achieved by setting thresholds or selecting top-ranked keywords. In some cases, further semantic analysis may be required to ensure that the selected keywords can accurately reflect the core concepts of the document and avoid misleading surface vocabulary. Based on the selected keywords, generate tags for the document. These tags should be able to summarize the main content and theme of the document and be easy for users to understand and retrieve. Determine the appropriate number of tags, neither too many to cause information overload nor too few to cause information loss. Usually, the number of tags is limited to a certain range, such as 3-5.

[0083] In the above document tag extraction method, by automatically processing the document body and frame, there is no need for manual reading and annotation one by one, which greatly improves the speed and efficiency of document tag generation. Through word segmentation and word frequency statistics, combined with the calculation of weight vectors, key words in the document can be more accurately identified, thereby improving the relevance and accuracy of keywords and final tags. As the document body is updated, the system can re-segment and extract keywords to ensure the timeliness and dynamics of the tags. It reduces the reliance on professional manual annotation, thereby reducing long-term operation and maintenance costs. Users can use automatically generated tags to quickly locate documents of interest, improving the convenience of retrieval and user experience. Compared with traditional manual methods, automation technology can significantly reduce the cost of update and maintenance. The document classification management model uses a systematic approach, classification, labeling, version control and other means to improve the reliability and efficiency of documents.

[0084] In an exemplary embodiment, Figure 3 As shown, get the weight vector for each individual word, including:

[0085] Step S302, respectively obtaining the importance of the document body and at least one level title;

[0086] Step S304, constructing a judgment matrix according to the importance of the document body and at least one level title;

[0087] Step S306, obtaining the maximum eigenvalue and corresponding eigenvector of the judgment matrix;

[0088] Step S308: Obtain a weight vector for each individual word according to the maximum eigenvalue and the corresponding eigenvector.

[0089] Specifically, the words in the text are analyzed, and the frequency of the words in the document is usually measured by word frequency (TF) statistics to reflect their importance in the document. Hierarchical titles (such as main titles and subtitles) summarize the structure and theme of the document, so the words in the title are usually considered to have a higher weight. Using AHP (Analytical Hierarchy Process) or other pairwise comparison methods, the words in the document are compared pairwise to determine their relative importance and construct a judgment matrix. The judgment matrix is ​​a square matrix in which the elements represent the importance of one word relative to another. The maximum eigenvalue of the judgment matrix reflects the comprehensive score of the importance of the word, and the corresponding eigenvector contains the weight coefficient of each word. The maximum eigenvalue and corresponding eigenvector of the judgment matrix are solved by mathematical methods (such as the power method or the Jacobi method). After the eigenvector is normalized (so that the sum of the vector elements is 1 or the length is 1), the weight vector of each word is obtained. This weight vector represents the relative importance of the word in the document. In the process of constructing the judgment matrix, a consistency check is required to ensure the logical consistency of the judgment matrix. If the consistency ratio (CR) is less than 0.1, the consistency of the judgment matrix is ​​considered acceptable. The obtained weight vector can be used for a variety of text analysis tasks, such as keyword extraction, document classification, information retrieval, etc., to improve the accuracy and efficiency of these tasks. Document content may be updated over time, so the weight vector may need to be recalculated periodically to ensure that it reflects the latest document content and structure. In some cases, in addition to structure and content, other dimensions (such as word position, part of speech, semantic relevance, etc.) can also be considered to enrich the calculation of the weight vector.

[0090] In this embodiment, through this process, a weight can be automatically assigned to each word in the document. These weights reflect the relative importance of the words in the document, and provide a quantitative method for text mining and information retrieval.

[0091] In an exemplary embodiment, before obtaining the maximum eigenvalue and the corresponding eigenvector of the judgment matrix, the method further includes:

[0092] Calculate the consistency index and random consistency index of the judgment matrix;

[0093] According to the consistency index and the random consistency index, the consistency ratio of the judgment matrix is ​​obtained;

[0094] When the consistency ratio is lower than the preset value, it is determined that the judgment matrix meets the consistency requirement, and the step of obtaining the maximum eigenvalue and the corresponding eigenvector of the judgment matrix is ​​performed.

[0095] Specifically, the consistency index (CI) of the judgment matrix is ​​calculated: the consistency index is calculated based on the maximum eigenvalue of the judgment matrix and is used to measure the logical consistency of the judgment matrix. If the CI value is 0, it means that the judgment matrix is ​​completely consistent; the larger the CI value, the more serious the inconsistency of the judgment matrix.

[0096] Find the random consistency index (RI): The random consistency index is an average value obtained by looking up the table according to the order n of the judgment matrix, which is used to compare with the CI value. The size of the RI value depends on the order of the matrix, usually between 1 and 15. Calculate the consistency ratio (CR): CR is the ratio of CI to RI, which is used to determine whether the consistency of the judgment matrix is ​​acceptable. If the CR value is less than 0.1, the consistency of the judgment matrix is ​​considered acceptable; if the CR value is greater than or equal to 0.1, the judgment matrix needs to be modified to improve consistency. The calculation formula for CR is CR = CI / RI.

[0097] When the consistency ratio CR is lower than a preset value (usually 0.1), the judgment matrix can be considered to meet the consistency requirement. At this time, the step of obtaining the maximum eigenvalue and corresponding eigenvector of the judgment matrix can be performed. After normalization, these eigenvectors can be used as weights of various factors for subsequent decision analysis.

[0098] If the CR value does not meet the consistency requirements, it is necessary to reconsider the elements in the judgment matrix and adjust them until the CR value is less than 0.1. The correction method is to adjust it to the consistency matrix. The characteristic of the consistency matrix is ​​that the rows and columns are in multiples. After adjustment, the consistency test is performed again until it passes.

[0099] In this embodiment, by calculating the consistency index and random consistency index of the judgment matrix, the consistency of the judgment matrix can be evaluated, thereby improving the calculation accuracy of the judgment matrix.

[0100] In an exemplary embodiment, determining keywords of a document framework based on a word frequency and a weight vector of at least one individual word includes:

[0101] Performing deduplication processing and stop word removal processing on at least one level title to obtain at least one first keyword of the document framework;

[0102] Matching at least one first keyword with at least one separate word to obtain a word frequency and a weight vector corresponding to the at least one first keyword;

[0103] A weight of at least one first keyword is determined according to the word frequency and the weight vector of at least one first keyword.

[0104] Specifically, first, preprocess each hierarchical title of the document (such as main title, subtitle, etc.), including removing duplicate words and deleting stop words (such as "of", "is", etc.), which usually do not carry important information in semantic analysis. After deduplication and stop word removal, the remaining words are regarded as the important keywords of the document framework, that is, "first keywords". These keywords can usually better summarize the core content of the title. Match these first keywords with the individual words in the document to determine their occurrence frequency and importance in the document. For each successfully matched first keyword, calculate its word frequency (i.e., the number of occurrences) in the document and the corresponding weight vector. The weight vector may include weights determined according to factors such as word position, part of speech, and relevance to the theme. According to the word frequency and weight vector of each first keyword, calculate a comprehensive weight, which reflects the importance of the keyword in the document. The higher the word frequency, the greater the weight may be; the higher the value in the weight vector, it also indicates the greater importance of the word in a specific dimension. Sort all the first keywords according to the calculated weights, and several keywords with the highest weights can be selected as candidate tags for the document. Finally, these keywords and their weights can be used for document tag generation, helping to determine the main content and theme of the document, and then for document classification, retrieval, and recommendation.

[0105] In this embodiment, through a structured and quantitative method, the keywords that best represent the document content are extracted from the title information of the document framework, and reasonable weights are assigned to these keywords for subsequent information processing and decision support.

[0106] In an exemplary embodiment, determining the keywords of the document body according to the word frequency and weight vector of at least one individual word includes:

[0107] Using a large language model, obtain the abstract of the document body;

[0108] Perform word segmentation processing and stop word removal processing on the abstract to obtain at least one second keyword of the document body;

[0109] Match at least one second keyword with at least one individual word to obtain the word frequency and weight vector corresponding to at least one second keyword;

[0110] Determine the weight of at least one second keyword according to the word frequency and weight vector of at least one second keyword.

[0111] Specifically, a large language model (such as BERT, GPT, etc.) is used to analyze the body of the document and extract or generate a summary. The summary should contain the core content and main concepts of the document. The summary text is segmented, that is, the continuous text string is divided into separate words. At the same time, stop words are removed, which are words that appear frequently in the text but usually do not carry important information. After word segmentation and stop word removal, the words that can represent the core content of the summary are selected from the remaining words, which are called "second keywords". The second keywords are matched with the individual words in the document to determine the frequency and distribution of these keywords in the document. For each second keyword, its word frequency (number of occurrences) and weight vector in the document are calculated. The weight vector may be based on different factors, such as the position of the word, part of speech, and relevance to the topic. Based on the word frequency and weight vector of each second keyword, a comprehensive weight is calculated, which reflects the importance of the keyword in the body of the document. Based on the calculated weights, all second keywords are sorted, and the keywords with the highest weights can be selected as candidate tags for the body of the document. These keywords and their weights can be used to generate document tags, help determine the main content and theme of the document, and then be used for document classification, retrieval and recommendation.

[0112] Multi-dimensional analysis: By combining word frequency and multi-dimensional weight vectors, this method can more comprehensively evaluate the importance of words, rather than relying solely on word frequency. The entire process is automated and can effectively extract key words from the body of the document and assign reasonable weights to them.

[0113] In an exemplary embodiment, determining a document tag according to keywords of a document frame and keywords of a document body includes:

[0114] According to the weight of the at least one first keyword and the weight of the second keyword, the at least one first keyword and the second keyword are sorted to obtain a sorting result;

[0115] Determine the document label based on the sorting processing results.

[0116] Specifically, integrate the weights of the keywords (first keywords) in the document framework and the keywords (second keywords) in the document body. These weights reflect the importance of each keyword in the document. Sort according to the weights of each keyword. The weights may be calculated based on factors such as word frequency, position, part of speech, and relevance to the theme. Sorting can help identify the most important keywords. The sorting result provides an intuitive list of keyword importance. Usually, the keywords with the highest weights will be at the front of the list. According to the sorting result, select the keywords ranked at the front as the tags of the document. These tags should be able to accurately summarize the main content and theme of the document. Determine the number of tags, which can be determined according to actual needs. For example, the top 3 or top 5 keywords with the highest weights can be selected as the tags of the document. The selected tags should be highly representative and can quickly convey the core content of the document to readers or users.

[0117] In this embodiment, by comprehensively considering the keywords from different sources (document framework and document body), this method can evaluate the document content from multiple dimensions and improve the accuracy and comprehensiveness of the tags.

[0118] One of the most detailed embodiments of this application is:

[0119] 1. Obtain the main title and sub-titles of the document framework: First, use methods such as OCR layout recognition or rule extraction to extract the 0-level title (main title), 1-level sub-titles, and 2-level sub-titles from the document. These titles will be important sources for subsequent keyword extraction.

[0120] 2. Segment words and remove stop words: Use word segmentation tools (such as Jieba, HanLP, SnowNLP) to break down the document body and the document framework into individual words. This step is to refine the document body and the document framework for further word frequency analysis and keyword extraction; remove common meaningless words in the text, such as "of", "is", etc., which usually do not carry important information.

[0121] 3. Calculate the term frequency (TF): Count the frequency of each word in the document. The term frequency TF(t, d) is defined as the number of times the word t appears in the document d divided by the total number of words in the document d, and store it in the database T.

[0122] 4. Weight setting, set the weight of the 0-level title keyword as a1, the weight of the 1-level title keyword as a2, the weight of the 2-level title keyword as a3, and the weight of the document body keyword as a4. Use the pairwise comparison method to determine the keyword weights.

[0123] 4.1 In this case, the weights to be determined include four factors: the importance of the 0-level title keyword, the importance of the 1-level title keyword, the importance of the 2-level title keyword, and the importance of the document body keyword.

[0124] 4.2 Construct the judgment matrix: Compare each factor in the criterion layer two by two to determine their relative importance. Use a 1-9 scale to score, where 1 means that the two factors are equally important and 9 means that one factor is 9 times more important than the other.

[0125] 4.3 Consistency test: Calculate the consistency index (CI) and random consistency index (RI). The consistency ratio (CR) of the judgment matrix = CI / RI. If CR < 0.1, the consistency of the judgment matrix is ​​considered acceptable.

[0126] The calculation formula of CI is: where λ max is the maximum eigenvalue of the judgment matrix, and n is the order of the judgment matrix.

[0127] The values ​​of random consistency index (RI) are shown in the following table:

[0128] n 1 2 3 4 5 6 7 8 9 10 RI 0 0 0.58 0.90 1.12 1.24 1.32 1.41 1.45 1.49

[0129] 5. Calculate weights: Obtain the maximum eigenvalue and corresponding eigenvector of the judgment matrix. After normalizing the eigenvector, the weight vector of each factor is w = (a1, a2, a3, a4).

[0130] 6. Document framework keyword acquisition: In this stage, keywords will be extracted according to the title level of the document and assigned different weights: Level 0 title keyword extraction: Level 0 titles are segmented and stop words are removed, the extracted keywords are deduplicated, and the weight is assigned to a1*keyword frequency, which is stored in the vocabulary A1; Level 1 title keyword extraction: The same process is performed on level 1 titles, and the weight of keywords is set to a2*keyword frequency, which is stored in the vocabulary A2; Level 2 title keyword extraction: Level 2 titles are processed, and the weight of keywords is set to a3*keyword frequency, which is stored in the vocabulary A3.

[0131] Keyword acquisition in the document body: call the large language model to generate a summary, segment the generated summary information and remove stop words, extract keywords, deduplicate keywords, assign a weight of a4*keyword frequency, and form the vocabulary A4.

[0132] 7. Obtain document tags based on keywords: The final stage is to merge the word libraries A1, A2, A3, and A4 and sort them by weight:

[0133]

[0134] For any keyword: if the word is in the vocabulary Ai, ai is the weight vector corresponding to the keyword in the vocabulary; then TF iis the frequency of the keyword in the vocabulary. If there is no word frequency in the vocabulary, then TF i =0; Ci is the number of times the keyword appears in the vocabulary A1, A2, A3 and A4.

[0135] 8. Label generation: Sort by weight from high to low, and select the top k (3-5) keywords with the highest weight as the document label.

[0136] In summary, through the above steps, the most representative and important keywords can be systematically extracted from the document to label the document, which is of great significance for the rapid retrieval and classification of the document.

[0137] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0138] Based on the same inventive concept, the embodiment of the present application also provides a document tag extraction device for implementing the document tag extraction method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more document tag extraction device embodiments provided below can refer to the limitations of the document tag extraction method above, and will not be repeated here.

[0139] In an exemplary embodiment, Figure 4 As shown, a document tag extraction device is provided, comprising:

[0140] The acquisition module 402 is used to acquire the document body and the document frame, wherein the document frame includes at least one hierarchical title;

[0141] A processing module 404 is used to perform word segmentation processing on the document body and at least one level title to obtain at least one individual word;

[0142] The processing module 404 is further used to obtain the word frequency and weight vector of each individual word;

[0143] A keyword extraction module 406, for determining keywords of the document frame and keywords of the document body based on the word frequency and weight vector of at least one individual word;

[0144] The tag extraction module 408 is used to determine the document tag according to the keywords of the document frame and the keywords of the document body.

[0145] In an exemplary embodiment, the processing module 404 is further configured to obtain the importance of the document body and at least one level title respectively;

[0146] Construct a judgment matrix based on the importance of the document body and at least one level title;

[0147] Get the maximum eigenvalue and corresponding eigenvector of the judgment matrix;

[0148] Based on the maximum eigenvalue and the corresponding eigenvector, get the weight vector for each individual word.

[0149] In an exemplary embodiment, the processing module 404 is further used to calculate the consistency index and random consistency index of the judgment matrix;

[0150] According to the consistency index and the random consistency index, the consistency ratio of the judgment matrix is ​​obtained;

[0151] When the consistency ratio is lower than the preset value, it is determined that the judgment matrix meets the consistency requirement, and the step of obtaining the maximum eigenvalue and the corresponding eigenvector of the judgment matrix is ​​performed.

[0152] In an exemplary embodiment, the keyword extraction module 406 is further used to perform a deduplication process and a stop word removal process on at least one level title to obtain at least one first keyword of the document framework;

[0153] Matching at least one first keyword with at least one separate word to obtain a word frequency and a weight vector corresponding to the at least one first keyword;

[0154] A weight of at least one first keyword is determined according to the word frequency and the weight vector of at least one first keyword.

[0155] In an exemplary embodiment, the keyword extraction module 406 is further used to obtain a summary of the document body using the large language model;

[0156] Perform word segmentation and stop word removal on the abstract to obtain at least one second keyword of the document body;

[0157] Matching at least one second keyword with at least one separate word to obtain a word frequency and a weight vector corresponding to the at least one second keyword;

[0158] The weight of at least one second keyword is determined according to the word frequency and the weight vector of at least one second keyword.

[0159] In an exemplary embodiment, the tag extraction module 408 is further configured to perform sorting processing on at least one first keyword and the second keyword according to the weight of at least one first keyword and the weight of the second keyword to obtain a sorting processing result;

[0160] Determine the document label based on the sorting processing results.

[0161] Each module in the document tag extraction device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0162] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store document data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a document tag extraction method is implemented.

[0163] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0164] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0165] Performing word segmentation processing on the document body and at least one level title to obtain at least one single word;

[0166] Get the frequency and weight vector of each individual word;

[0167] Determining keywords of the document frame and keywords of the document body based on the word frequency and weight vector of at least one individual word;

[0168] Determine the document tag based on the keywords of the document frame and the keywords of the document body.

[0169] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0170] Obtain the importance of the document body and at least one level title respectively;

[0171] Construct a judgment matrix based on the importance of the document body and at least one level title;

[0172] Get the maximum eigenvalue and corresponding eigenvector of the judgment matrix;

[0173] Based on the maximum eigenvalue and the corresponding eigenvector, get the weight vector for each individual word.

[0174] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0175] Calculate the consistency index and random consistency index of the judgment matrix;

[0176] According to the consistency index and the random consistency index, the consistency ratio of the judgment matrix is ​​obtained;

[0177] When the consistency ratio is lower than the preset value, it is determined that the judgment matrix meets the consistency requirement, and the step of obtaining the maximum eigenvalue and the corresponding eigenvector of the judgment matrix is ​​performed.

[0178] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0179] Performing deduplication processing and stop word removal processing on at least one level title to obtain at least one first keyword of the document framework;

[0180] Matching at least one first keyword with at least one separate word to obtain a word frequency and a weight vector corresponding to the at least one first keyword;

[0181] A weight of at least one first keyword is determined according to the word frequency and the weight vector of at least one first keyword.

[0182] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0183] Use the large language model to obtain a summary of the document body;

[0184] Perform word segmentation and stop word removal on the abstract to obtain at least one second keyword of the document body;

[0185] Matching at least one second keyword with at least one separate word to obtain a word frequency and a weight vector corresponding to the at least one second keyword;

[0186] The weight of at least one second keyword is determined according to the word frequency and the weight vector of at least one second keyword.

[0187] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0188] According to the weight of the at least one first keyword and the weight of the second keyword, the at least one first keyword and the second keyword are sorted to obtain a sorting result;

[0189] Determine the document label based on the sorting processing results.

[0190] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0191] Performing word segmentation processing on the document body and at least one level title to obtain at least one single word;

[0192] Get the frequency and weight vector of each individual word;

[0193] Determining keywords of the document frame and keywords of the document body based on the word frequency and weight vector of at least one individual word;

[0194] Determine the document tag based on the keywords of the document frame and the keywords of the document body.

[0195] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0196] Obtain the importance of the document body and at least one level title respectively;

[0197] Construct a judgment matrix based on the importance of the document body and at least one level title;

[0198] Get the maximum eigenvalue and corresponding eigenvector of the judgment matrix;

[0199] Based on the maximum eigenvalue and the corresponding eigenvector, get the weight vector for each individual word.

[0200] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0201] Calculate the consistency index and random consistency index of the judgment matrix;

[0202] According to the consistency index and the random consistency index, the consistency ratio of the judgment matrix is ​​obtained;

[0203] When the consistency ratio is lower than the preset value, it is determined that the judgment matrix meets the consistency requirement, and the step of obtaining the maximum eigenvalue and the corresponding eigenvector of the judgment matrix is ​​performed.

[0204] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0205] Performing deduplication processing and stop word removal processing on at least one level title to obtain at least one first keyword of the document framework;

[0206] Matching at least one first keyword with at least one separate word to obtain a word frequency and a weight vector corresponding to the at least one first keyword;

[0207] A weight of at least one first keyword is determined according to the word frequency and the weight vector of at least one first keyword.

[0208] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0209] Use the large language model to obtain a summary of the document body;

[0210] Perform word segmentation and stop word removal on the abstract to obtain at least one second keyword of the document body;

[0211] Matching at least one second keyword with at least one separate word to obtain a word frequency and a weight vector corresponding to the at least one second keyword;

[0212] The weight of at least one second keyword is determined according to the word frequency and the weight vector of at least one second keyword.

[0213] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0214] According to the weight of the at least one first keyword and the weight of the second keyword, the at least one first keyword and the second keyword are sorted to obtain a sorting result;

[0215] Determine the document label based on the sorting processing results.

[0216] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0217] Performing word segmentation processing on the document body and at least one level title to obtain at least one single word;

[0218] Get the frequency and weight vector of each individual word;

[0219] Determining keywords of the document frame and keywords of the document body based on the word frequency and weight vector of at least one individual word;

[0220] Determine the document tag based on the keywords of the document frame and the keywords of the document body.

[0221] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0222] Obtain the importance of the document body and at least one level title respectively;

[0223] Construct a judgment matrix based on the importance of the document body and at least one level title;

[0224] Get the maximum eigenvalue and corresponding eigenvector of the judgment matrix;

[0225] Based on the maximum eigenvalue and the corresponding eigenvector, get the weight vector for each individual word.

[0226] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0227] Calculate the consistency index and random consistency index of the judgment matrix;

[0228] According to the consistency index and the random consistency index, the consistency ratio of the judgment matrix is ​​obtained;

[0229] When the consistency ratio is lower than the preset value, it is determined that the judgment matrix meets the consistency requirement, and the step of obtaining the maximum eigenvalue and the corresponding eigenvector of the judgment matrix is ​​performed.

[0230] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0231] Performing deduplication processing and stop word removal processing on at least one level title to obtain at least one first keyword of the document framework;

[0232] Matching at least one first keyword with at least one separate word to obtain a word frequency and a weight vector corresponding to the at least one first keyword;

[0233] A weight of at least one first keyword is determined according to the word frequency and the weight vector of at least one first keyword.

[0234] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0235] Use the large language model to obtain a summary of the document body;

[0236] Perform word segmentation and stop word removal on the abstract to obtain at least one second keyword of the document body;

[0237] Matching at least one second keyword with at least one separate word to obtain a word frequency and a weight vector corresponding to the at least one second keyword;

[0238] The weight of at least one second keyword is determined according to the word frequency and the weight vector of at least one second keyword.

[0239] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0240] According to the weight of the at least one first keyword and the weight of the second keyword, the at least one first keyword and the second keyword are sorted to obtain a sorting result;

[0241] Determine the document label based on the sorting processing results.

[0242] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0243] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0244] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0245] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A document tag extraction method, characterized in that: The method comprises: Obtain a document body and a document frame, wherein the document frame includes at least one hierarchical title; Performing word segmentation processing on the document body and the at least one hierarchical title to obtain at least one separate word; Get the frequency and weight vector of each individual word; Determining keywords of the document framework and keywords of the document body based on the word frequency and weight vector of the at least one separate word; A document tag is determined according to the keywords of the document frame and the keywords of the document body.

2. The method according to claim 1, characterized in that The step of obtaining the weight vector of each individual word includes: Respectively obtaining the importance of the document body and the at least one hierarchical title; Constructing a judgment matrix according to the importance of the document body and the at least one hierarchical title; Obtaining the maximum eigenvalue and corresponding eigenvector of the judgment matrix; A weight vector for each individual word is obtained according to the maximum eigenvalue and the corresponding eigenvector.

3. The method according to claim 2, characterized in that Before obtaining the maximum eigenvalue and the corresponding eigenvector of the judgment matrix, the method further includes: Calculating the consistency index and random consistency index of the judgment matrix; Obtaining a consistency ratio of the judgment matrix according to the consistency index and the random consistency index; When the consistency ratio is lower than a preset value, it is determined that the judgment matrix meets the consistency requirement, and a step of obtaining a maximum eigenvalue and a corresponding eigenvector of the judgment matrix is ​​performed.

4. The method according to claim 1, characterized in that: The step of determining the keywords of the document framework according to the word frequency and weight vector of the at least one separate word comprises: Performing deduplication processing and stop word removal processing on the at least one level title to obtain at least one first keyword of the document framework; Matching at least one first keyword with at least one separate word to obtain a word frequency and a weight vector corresponding to the at least one first keyword; A weight of at least one first keyword is determined according to the word frequency and the weight vector of at least one first keyword.

5. The method according to claim 4, characterized in that The step of determining the keywords of the document body according to the word frequency and weight vector of the at least one separate word comprises: Using a large language model, obtaining a summary of the document body; Performing word segmentation and stop word removal on the abstract to obtain at least one second keyword of the document body; Matching at least one second keyword with at least one separate word to obtain a word frequency and a weight vector corresponding to the at least one second keyword; The weight of at least one second keyword is determined according to the word frequency and the weight vector of at least one second keyword.

6. The method according to claim 5, characterized in that Determining the document tag according to the keywords of the document frame and the keywords of the document body includes: According to the weight of the at least one first keyword and the weight of the second keyword, the at least one first keyword and the second keyword are sorted to obtain a sorting result; According to the sorting result, a document label is determined.

7. A document tag extraction device, characterized in that: The device comprises: An acquisition module, used for acquiring a document body and a document frame, wherein the document frame includes at least one hierarchical title; A processing module, configured to perform word segmentation processing on the document body and the at least one hierarchical title to obtain at least one separate word; The processing module is also used to obtain the word frequency and weight vector of each individual word; A keyword extraction module, used to determine keywords of the document framework and keywords of the document body according to the word frequency and weight vector of the at least one individual word; The tag extraction module is used to determine the document tag according to the keywords of the document frame and the keywords of the document body.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.