Text data information mining method, device and equipment

Abstract

The present disclosure relates to the technical field of semantic mining, and specifically discloses a text data information mining method and device and a computer device. The method comprises: obtaining text data, performing rule analysis on the text data, and extracting hierarchical titles; establishing parent-child relationships of the hierarchical titles and corresponding relationships between the hierarchical titles and paragraph texts and picture paths in the text data; obtaining a central abstract of the paragraph texts according to the paragraph texts; obtaining keywords according to the central abstract and the hierarchical titles; performing entity disassembly and recognition according to the text data and the keywords, and constructing a text context graph. The present disclosure can construct a text context graph according to keywords and text data, and realize deep semantic mining of text data while ensuring mining accuracy.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of semantic mining, and in particular, to a text data information mining method and device. BACKGROUND

[0002] With the rapid development of the information age, the rise of emerging technologies such as mobile Internet, Internet of Things, and artificial intelligence has brought a large amount of data, including structured and unstructured data. How to effectively obtain these data to assist in decision-making is an important link. For structured data, there are very mature databases for storage and retrieval, while for unstructured data, it is completely different from structured data. Because there is no uniform rule and format, there is no general solution for unstructured data analysis and retrieval.

[0003] In addition, most existing unstructured text knowledge bases obtain a large amount of text resources on the Internet, and the formats and topics of the massive resources are chaotic. The knowledge base usually combines a general word library to disassemble and segment the original text and establish a search index. The keyword index established in this way can only superficially reflect the surface text information and cannot reflect the hidden semantic information. In addition, the returned text during retrieval is only simple text and cannot determine whether it is the information needed, which wastes a lot of time during retrieval. SUMMARY

[0004] Therefore, it is necessary to provide a text data information mining method, a text data semantic retrieval knowledge base construction method, a text data semantic retrieval method, device, computer equipment, storage medium, and computer program product to solve the above technical problems.

[0005] In a first aspect, the present disclosure provides a text data information mining method. The method comprises:

[0006] obtaining text data, performing rule analysis on the text data, and extracting hierarchical titles;

[0007] establishing parent-child relationships of the hierarchical titles and corresponding relationships between the hierarchical titles and paragraph texts and picture paths in the text data;

[0008] obtaining a central abstract of the paragraph text according to the paragraph text;

[0009] obtaining a keyword according to the central abstract and the hierarchical titles;

[0010] performing entity disassembly and recognition according to the text data and the keyword, and constructing a text context map.

[0011] In one of the embodiments, the obtaining the central summary according to the paragraph text comprises:

[0012] parsing the paragraph text, extracting candidate sentences from the paragraph text to form a candidate sentence pool;

[0013] generating a first candidate summary based on the candidate sentence attention and the word attention in the candidate sentence pool;

[0014] sorting the pre-set summary templates to determine a candidate summary template;

[0015] generating the central summary according to the combination of the first candidate summary and the candidate summary template.

[0016] In one of the embodiments, the sorting the pre-set summary templates to determine a candidate summary template comprises:

[0017] inputting the summary template into a summary sorting model;

[0018] obtaining the sorting of the summary template by the summary sorting model to determine the output candidate summary template;

[0019] wherein, the summary sorting model is obtained by the following steps:

[0020] obtaining a summary template training set, wherein the summary template training set includes paragraph text and a target summary corresponding to the paragraph text;

[0021] calculating the evaluation index score of the summary template and the target summary;

[0022] sorting the summary template according to the evaluation index score to output a candidate summary template with a specified order.

[0023] In one of the embodiments, the obtaining the key words according to the central summary and the hierarchical titles comprises:

[0024] performing word segmentation and sentence segmentation processing on the central summary and the hierarchical titles to obtain predicted words;

[0025] establishing edges based on the semantic dependency relationship of the predicted words and performing entity relationship extraction to generate a semantic network;

[0026] extracting the key words from the semantic network according to the weight distribution of the central summary and the hierarchical titles.

[0027] In a second aspect, the disclosure also provides a semantic retrieval knowledge base construction method for text data. The method comprises:

[0028] Collect text data in batches, and use the text data information mining method described above to obtain the titles, paragraph texts, central summaries, keywords, and text context maps of the text data, and construct the basic data for the semantic retrieval knowledge base.

[0029] The headings, paragraph texts, central summaries, keywords, and text context maps at each level are classified based on the keywords and written into the distributed retrieval cluster according to the categories.

[0030] Thirdly, this disclosure also provides a semantic retrieval method for text data. The method includes:

[0031] Obtain search terms and extract search terms by segmenting the search terms.

[0032] The search terms are matched against a semantic retrieval knowledge base;

[0033] Obtain the matching results of the search term segmentation;

[0034] The matching results are sorted by search word segmentation and matching frequency, and the text context map is associated with the matching results.

[0035] The semantic retrieval knowledge base is constructed using the semantic retrieval knowledge base construction method for the text data described above.

[0036] In one embodiment, matching the search terms within the retrieval cluster includes:

[0037] The search terms and keywords are matched based on the first matching rule;

[0038] The search terms and the central summary are matched based on the second matching rule;

[0039] The search terms are matched with the paragraph text based on a third matching rule;

[0040] The matching result is obtained if the search word satisfies at least one of the first matching rule, the second matching rule, and the third matching rule.

[0041] Fourthly, this disclosure also provides a text data information mining apparatus. The apparatus includes:

[0042] The title extraction module is used to acquire text data, perform rule parsing on the text data, and extract titles at each level.

[0043] The title relationship module is used to establish the parent-child relationship between the titles at each level and to establish the correspondence between the titles at each level and the paragraph text and image paths in the text data;

[0044] The summary module is used to obtain the central summary of the paragraph text based on the paragraph text.

[0045] The keyword module is used to obtain keywords based on the central summary and the headings at each level;

[0046] The text context module is used to decompose and identify entities based on the text data and keywords, and construct a text context map.

[0047] In one embodiment, the summarization module includes:

[0048] An extraction unit is used to parse the paragraph text, extract candidate sentences from the paragraph text, and form a candidate sentence pool;

[0049] Candidate unit, used to extract a first candidate summary based on the candidate statement pool;

[0050] The template unit is used to sort the pre-set abstract templates and determine the candidate abstract templates;

[0051] A synthesis unit is used to generate the central summary by combining the first candidate summary and the candidate summary template.

[0052] In one embodiment, the template unit includes:

[0053] The model input subunit is used to input the summary template into the summary sorting model;

[0054] The model output subunit is used to obtain the sorting of the summary template by the summary sorting model and determine the output candidate summary template;

[0055] The device further includes a model building module, through which the summary ranking model is obtained. The model building module includes:

[0056] The training set unit is used to obtain a summary template training set, which includes paragraph text and the target summary corresponding to the paragraph text.

[0057] The evaluation index score unit is used to calculate the evaluation index score of the summary template and the target summary;

[0058] The sorting unit is used to sort the summary templates according to the evaluation index scores and output candidate summary templates in a specified order.

[0059] In one embodiment, the keyword module includes:

[0060] The prediction word unit is used to perform word and sentence segmentation on the central summary and the headings at each level to obtain prediction words;

[0061] A semantic network unit is used to establish edges based on the semantic dependencies of the predicted words and to extract entity relationships to generate a semantic network.

[0062] The network extraction unit is used to extract the keywords from the semantic network according to the weight allocation of the central summary and the headings at each level.

[0063] Fifthly, this disclosure also provides an apparatus for constructing a semantic retrieval knowledge base for text data. The apparatus includes:

[0064] The basic data module is used to collect text data in batches. Using the text data information mining method described above, it obtains the titles, paragraph texts, central summaries, keywords, and text context maps of the text data, and constructs the basic data for the semantic retrieval knowledge base.

[0065] The writing module is used to classify the titles, paragraph texts, central summaries, keywords, and text context maps of each level based on the keywords, and write them into the distributed retrieval cluster according to the categories.

[0066] Sixthly, this disclosure also provides a semantic retrieval device for text data. The device includes:

[0067] The search word segmentation module is used to obtain search terms and perform word segmentation on the search terms to extract search words.

[0068] The search word segmentation and matching module is used to match the search words in the semantic retrieval knowledge base.

[0069] The matching results module is used to obtain the matching results of the search word segmentation;

[0070] The result sorting module is used to sort the matching results by search word segmentation matching frequency and associate the text context map with the matching results.

[0071] The semantic retrieval knowledge base is constructed using the semantic retrieval knowledge base construction method for the text data described above.

[0072] In one embodiment, the search word segmentation and matching module includes:

[0073] The first matching unit is used to match the search terms with keywords based on a first matching rule;

[0074] The second matching unit is used to match the search words with the central summary based on the second matching rule;

[0075] The third matching unit is used to match the search word segment with the paragraph text based on the third matching rule;

[0076] The matching judgment unit is used to obtain the matching result of the search word satisfying at least one of the first matching rule, the second matching rule, and the third matching rule.

[0077] Seventhly, this disclosure also provides a computer device. The computer device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described text data information mining method, or the steps of the above-described text data semantic retrieval knowledge base construction method, or the steps of the above-described text data semantic retrieval method.

[0078] Eighthly, this disclosure also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the above-described text data information mining method, or the steps of the above-described text data semantic retrieval knowledge base construction method, or the steps of the above-described text data semantic retrieval method.

[0079] Ninthly, this disclosure also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the above-described text data information mining method, or the steps of the above-described text data semantic retrieval knowledge base construction method, or the steps of the above-described text data semantic retrieval method.

[0080] The aforementioned methods for mining text data, constructing semantic retrieval knowledge bases for text data, semantic retrieval methods, devices, computer equipment, storage media, and computer program products shall at least include the following beneficial effects:

[0081] This disclosure can structurally organize texts with diverse formats and disorganized themes that lack unified rules and formats. It extracts headings at each level recursively and establishes parent-child relationships between headings and relationships between headings and paragraph text. It obtains a central summary based on semantic analysis of paragraph text and combines the central summary with headings at each level to extract keywords. Compared to extracting keywords directly from text data, keyword coverage is improved and interference is reduced. At the same time, a text context map is constructed based on keywords and text data, achieving deep semantic mining of text data while ensuring mining accuracy. Attached Figure Description

[0082] To more clearly illustrate the technical solutions in the embodiments or conventional technologies of this disclosure, the accompanying drawings used in the description of the embodiments or conventional technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0083] Figure 1 This is an application environment diagram of a text data information mining method in one embodiment;

[0084] Figure 2 This is a flowchart illustrating a text data information mining method in one embodiment;

[0085] Figure 3 This is a flowchart illustrating the process of extracting headings at each level in one embodiment;

[0086] Figure 4 This is a flowchart illustrating a text data information mining method in one embodiment;

[0087] Figure 5 This is a flowchart illustrating a text data information mining method in one embodiment;

[0088] Figure 6 This is a flowchart illustrating a text data information mining method in one embodiment;

[0089] Figure 7 This is a flowchart illustrating a text data information mining method in one embodiment;

[0090] Figure 8 This is a flowchart illustrating a method for constructing a semantic retrieval knowledge base for text data in one embodiment.

[0091] Figure 9 This is a flowchart illustrating a method for constructing a semantic retrieval knowledge base for text data in one embodiment.

[0092] Figure 10 This is a flowchart illustrating a semantic retrieval method for text data in one embodiment;

[0093] Figure 11 This is a flowchart illustrating a semantic retrieval method for text data in one embodiment;

[0094] Figure 12 This is a flowchart illustrating a semantic retrieval method for text data in one embodiment;

[0095] Figure 13 This is a structural block diagram of a text data information mining device in one embodiment;

[0096] Figure 14This is a structural block diagram of a text data information mining device in one embodiment;

[0097] Figure 15 This is a structural block diagram of a text data information mining device in one embodiment;

[0098] Figure 16 This is a structural block diagram of a text data information mining device in one embodiment;

[0099] Figure 17 This is a structural block diagram of a text data information mining device in one embodiment;

[0100] Figure 18 This is a structural block diagram of a semantic retrieval knowledge base construction device for text data in one embodiment;

[0101] Figure 19 This is a structural block diagram of a semantic retrieval device for text data in one embodiment;

[0102] Figure 20 This is a structural block diagram of a semantic retrieval device for text data in one embodiment;

[0103] Figure 21 This is a block diagram of the internal structure of a computer device in one embodiment. Detailed Implementation

[0104] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0105] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of this disclosure.

[0106] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims. The terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of other identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded. For example, the use of terms such as "first," "second," etc., is to denote names and does not indicate any specific order.

[0107] It should be noted that when one element is considered to be "connected" to another element, it can be directly connected to the other element or connected to the other element through an intermediary element. Furthermore, in the following embodiments, "connection" should be understood as "electrical connection," "communication connection," etc., if there is transmission of electrical signals or data between the connected objects.

[0108] When used herein, the singular forms of “a,” “an,” and “the” may also include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising / including” or “having,” etc., specify the presence of the stated features, wholes, steps, operations, components, parts, or combinations thereof, but do not preclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof. Meanwhile, in this specification, the term “and / or” includes any and all combinations of the associated listed items.

[0109] The text data information mining method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Server 104 can acquire text data, perform information mining on the text data, and obtain keywords and text context maps. The text data can be unstructured text data. Server 104 can also perform information mining on large batches of text data, and establish a semantic retrieval knowledge base based on the mined keywords and text context maps. Server 104 can also receive query requests sent by terminal 102, perform searches in the semantic retrieval knowledge base, and return the search results to terminal 102. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster composed of multiple servers.

[0110] In some embodiments of this disclosure, such as Figure 2 As shown, a text data information mining method is provided, which can be applied to... Figure 1 Taking the server in the example, the following steps are included:

[0111] Step S10: Obtain text data, perform rule parsing on the text data, and extract the titles of each level.

[0112] Specifically, the text data obtained in this disclosure can be unstructured text data. Unstructured text data can be diverse in resource format and disorganized in subject matter, lacking a unified rule or format. The obtained text data is parsed according to rules; it can be broken down according to preset rules, starting with the first-level heading and recursively extracting headings at each level. For example, a rule file for recursively extracting headings at each level can be created, and the rule file can be run in a language environment to generate a target file containing headings at each level of the text data.

[0113] Step S20: Establish the parent-child relationship of each level of heading, and establish the correspondence between each level of heading and the paragraph text and image path in the text data.

[0114] Specifically, based on the extracted headings at each level, a parent-child relationship is established. The image paths of the paragraph text contained under each heading are mapped one-to-one with the headings at each level, thus establishing this correspondence. For example... Figure 3As shown, in some embodiments, starting from the first-level heading, the system first identifies whether the current heading contains a text paragraph. If so, the text paragraph and the image paths within it are extracted and entered into the database. If not, the system recursively queries the next level heading. If a sub-level heading is found, the text paragraph and the image paths within it are extracted and entered into the database. If not, the system recursively queries the next level heading. When it is determined that no next level heading is found, the system recursively returns the current level heading and inserts the returned heading into the sub-heading field. Finally, the headings and corresponding paragraph texts parsed from the text data are stored in the database, and each paragraph text can be linked to its respective level heading.

[0115] Step S30: Obtain the central summary of the paragraph text based on the paragraph text.

[0116] Specifically, text mining is performed on paragraphs to obtain their semantic information, thereby deriving a central summary of the paragraph text. This text mining can employ techniques such as Natural Language Processing (NLP), deep semantic networks, and knowledge graphs for intelligent text extraction. The central summary of the paragraph text is then extracted based on the semantic information obtained through mining.

[0117] Step S40: Obtain keywords based on the central summary and the headings at each level.

[0118] Specifically, when extracting keywords from text data, the headings at each level and the central summary are used as input to comprehensively obtain keywords. In other words, keyword extraction is done based on the central summary.

[0119] Step S50: Entity decomposition and recognition are performed based on the text data and keywords to construct a text context map.

[0120] Specifically, the extracted keywords and text data are decomposed and identified to create a text context map. This text context map typically includes contextual information about the text, such as the subject, object, and event sequence, and can cover the outline of events within the text.

[0121] The aforementioned text data information mining method structurally organizes texts with diverse formats and disorganized themes that lack unified rules and formats. It recursively extracts headings at each level and establishes parent-child relationships between headings and the relationships between headings and paragraph text. Furthermore, it obtains a central summary based on paragraph text semantic analysis and combines this summary with headings at each level to extract keywords. Compared to directly extracting keywords from text data, this method improves keyword coverage and reduces interference. Simultaneously, it constructs a text context map based on keywords and text data, achieving deep semantic mining of text data while ensuring mining accuracy.

[0122] In some embodiments of this disclosure, such as Figure 4 As shown, step S30 includes:

[0123] Step S32: Parse the paragraph text, extract candidate sentences from the paragraph text, and form a candidate sentence pool.

[0124] Specifically, the paragraph text under each level of heading is parsed, and candidate sentences are extracted from the paragraph text to form a candidate sentence pool. Sentences with higher information content can be extracted based on the information content of the sentences in the paragraph text to form the candidate sentence pool. In some embodiments, extraction can be performed using an extractor. For example, the extractor employs a hierarchical bidirectional GRU (Gate Recurrent Unit) network, taking the parsed text paragraphs as input and extracting sentences with higher information content from the paragraph text based on information content ranking, resulting in a fine-grained probability distribution of sentences.

[0125] Step S34: Generate a first candidate summary based on the candidate sentence attention and word attention in the candidate sentence pool.

[0126] Specifically, a generative summary is output based on candidate sentences in the candidate sentence pool. This can be achieved by using a joint attention mechanism of words and sentences within the candidate sentences to calculate the highest-scoring summary, which is then used as the first candidate summary. In some embodiments, a generator can be used to generate the first candidate summary. For example, the generator can employ a Pointer-generator network, a general encoder-decoder (pre-trained language model) structure. During prediction, a joint attention mechanism of words and sentences is used to calculate the highest-scoring summary. In this case, the sentences in the first candidate summary are not necessarily all original sentences from the paragraph text.

[0127] Step S36: Sort the pre-set abstract templates to determine candidate abstract templates.

[0128] Specifically, pre-set summary templates are introduced. These templates can be sentence-based or logical templates based on summary elements (such as purpose, methods, results, conclusions, events, etc.), templates formed based on expert experience, or templates automatically generated after batch summarization using deep learning. Several summary templates are sorted according to preset rules, and the templates with a high degree of relevance to the paragraph text are selected as candidate summary templates.

[0129] Step S38: Generate the central summary by combining the first candidate summary and the candidate summary template.

[0130] Specifically, the generated first candidate summary and the introduced candidate summary template are combined to finally generate the central summary. In some embodiments, this can be implemented using a Retrieve submodule, a Rerank submodule, and a Rewrite submodule. The Retrieve submodule is used to use the extraction results of the extractor as candidate statements; the Rerank submodule is used to sort the summary templates; and the Rewrite submodule is used to combine the candidate summary templates and the first candidate summary generated by the generator to generate the central summary.

[0131] This embodiment obtains the first candidate summary by directly extracting and processing it, which is more flexible and accurate than directly extracting sentences from paragraph text to form a summary. Furthermore, the first candidate summary is combined with the candidate template summary to further improve the logical summarization ability and clear readability of the central summary.

[0132] In some embodiments of this disclosure, such as Figure 5 As shown, step S36 includes:

[0133] Step S362: Input the summary template into the summary sorting model.

[0134] Specifically, this embodiment uses a candidate abstract ranking model to obtain candidate abstract templates. This abstract ranking model can output a ranking of the abstract templates based on the input abstract templates.

[0135] Step S364: Obtain the sorting of the summary template by the summary sorting model, and determine the output candidate summary template.

[0136] Specifically, the sorting of the summary templates output by the summary ranking model is obtained, and the candidate summary templates are determined.

[0137] Among them, such as Figure 6 As shown, the summary ranking model is trained through the following steps:

[0138] Step A10: Obtain the summary template training set, which includes paragraph text and the target summary corresponding to the paragraph text.

[0139] Specifically, the summary ranking model is trained using a summary template training set. The summary template training set includes paragraph text and the corresponding target summary. The target summary can refer to a known true summary of the paragraph text.

[0140] Step A20: Calculate the evaluation index scores of the summary template and the target summary.

[0141] Step A30: Sort the summary templates according to the evaluation index scores and output candidate summary templates in the specified order.

[0142] Specifically, the ROUGE (Recall-Oriented Understudy for Gisting Evaluation) score between each abstract template and the target abstract is calculated as the ranking criterion. ROUGE is a commonly used evaluation metric in machine translation, automatic summarization, and question-answering generation. By calculating the ROUGE score between each abstract template and the target abstract, the abstract templates are ranked according to their ROUGE scores, and candidate abstract templates in a specified order are output. The specified order can be the order of the highest scores.

[0143] This embodiment uses a pre-trained summary ranking model to quickly output candidate summary templates that match paragraph text based on preset evaluation indicators, thereby improving the accuracy of the summary templates. Then, the candidate summary templates are combined with the first candidate summary to further improve the logical summarization ability and clear readability of the central summary.

[0144] In some embodiments of this disclosure, such as Figure 7 As shown, step S40 includes:

[0145] Step S42: Perform word segmentation and sentence segmentation on the central summary and the headings at each level to obtain predicted words.

[0146] Specifically, the generated central summary and headings at each level are segmented into words and sentences to remove conjunctions, pause words, etc., and predictive words are extracted. It's important to note that predictive words are not extracted directly from the text data, but rather from the headings at each level and the central summary generated based on semantic understanding. Predictive words can refer to phrases that are predicted to be keywords, extracted according to certain rules; for example, predictive words can be extracted by ranking by frequency.

[0147] Step S44: Establish edges based on the semantic dependencies of the predicted words, and extract entity relationships to generate a semantic network.

[0148] Specifically, the semantic relationships between predicted words are analyzed, and edges are established based on the semantic dependencies of the predicted words. Here, edges can refer to relationships between predicted words represented in a directed graph. For example, rules can be constructed starting with verbs to constrain the part-of-speech tags of nodes and the dependencies on edges. Further entity relationship extraction generates a semantic network. A semantic network typically refers to a form of expressing human knowledge construction in a network format, used to describe object concepts and states and the relationships between them. Mathematically, a semantic network is a directed graph, corresponding to logical representation. Semantic networks can be used in the natural language understanding scenario of this embodiment. Semantic dependencies typically refer to the semantic relationships between content words based on grammatical analysis. Entity relationship extraction typically refers to identifying named entities and extracting the relationships between them.

[0149] Step S46: Extract the keywords from the semantic network according to the weight allocation of the central summary and the headings at each level.

[0150] Specifically, the weight allocation of the central summary and the headings at each level is determined. For example, in some embodiments, the weight of the central summary is greater than the weight of the headings at each level. In the established semantic network, predicted words are used as nodes, and the edges connecting each node to other nodes have corresponding weights. By calculating the comprehensive weight of each node (the comprehensive weight can depend on the weight allocation of the central summary and the headings at each level, as well as the weights of the edges connecting the node to other nodes), all nodes are ranked based on the comprehensive weight, and the predicted word with the highest ranking can be extracted as the keyword.

[0151] This embodiment improves the semantic coverage of keywords by extracting predicted words from the generated central summary and the headings at each level, constructing a semantic network based on the predicted words, representing the semantic relationships between the predicted words in the form of a directed graph, and calculating the extracted keywords by combining weights.

[0152] In some embodiments of this disclosure, such as Figure 8 As shown, a method for constructing a semantic retrieval knowledge base for text data is provided, and this method is applied to... Figure 1 Taking the server in the example, the following steps are included:

[0153] Step B10: Collect text data in batches, and use the text data information mining method described above to obtain the titles, paragraph texts, central summaries, keywords, and text context maps of the text data at each level, and construct the basic data for the semantic retrieval knowledge base.

[0154] Specifically, in combination Figure 9As shown, batches of text data with heading levels are collected. The text data format can be HTML, and it may also include relevant images. Using the aforementioned text data information mining methods, semantic mining is performed on the text data to obtain headings, paragraph text, central abstracts, keywords, and text context maps at each level of the text data, thus constructing the basic data for a semantic retrieval knowledge base.

[0155] Step B20: Classify the headings, paragraph texts, central summaries, keywords, and text context maps at each level based on the keywords, and write them into the distributed retrieval cluster according to the categories.

[0156] Specifically, the basic data of the semantic retrieval knowledge base, including headings, paragraph texts, central summaries, keywords, and text context maps at each level, are classified based on keywords and written into a distributed retrieval cluster according to categories. This enables rapid matching of headings, paragraph texts, central summaries, keywords, and text context maps at each level based on the search fields.

[0157] This embodiment employs the text data information mining method described in the previous embodiment to perform deep semantic mining on batch text data, improving the semantic coverage of the central summary, keywords, and text context map of the text data. It can quickly match the search fields with the headings, paragraph texts, central summaries, keywords, and text context maps at each level, and optimize the retrieval and matching capabilities of the semantic retrieval knowledge base.

[0158] In some embodiments of this disclosure, such as Figure 10 As shown, a semantic retrieval method for text data is provided, which can be applied to... Figure 1 Taking the server in the example, the following steps are included:

[0159] Step C10: Obtain the search terms and perform word segmentation on the search terms to extract the search terms.

[0160] Specifically, the server can receive search requests sent by terminals and obtain search terms based on the search requests. The search terms are then segmented to extract the search terms.

[0161] Step C20: Match the search terms in the semantic retrieval knowledge base.

[0162] Specifically, the extracted search terms are matched against the semantic retrieval knowledge base, that is, the search terms are matched against keywords, headings at each level, central summaries, and paragraph texts in the semantic retrieval knowledge base.

[0163] Step C30: Obtain the matching results of the search word segmentation.

[0164] Specifically, by matching the search terms in a semantic retrieval knowledge base, matching results for the search terms can be obtained. These matching results can include a collection of text data matched by the search terms.

[0165] Step C40: Sort the matching results by search word segmentation and matching frequency, and associate the text context map with the matching results.

[0166] Specifically, in combination Figure 11 As shown, the set of matching text data in the matching results is sorted, which can be based on the matching frequency of the search terms; for example, the higher the matching frequency, the higher the ranking of the text data. Simultaneously, the text context map corresponding to the text data in the matching results is associated with the matching results, enabling rapid semantic analysis of the text data based on the text context map. The semantic retrieval knowledge base is constructed using the semantic retrieval knowledge base construction method for text data described in the aforementioned embodiments.

[0167] This embodiment performs matching based on search word segmentation in the semantic retrieval knowledge base constructed in the aforementioned embodiments. It can quickly match based on keywords, headings at each level, central abstracts, and paragraph text, and can also associate text context maps with the matching results. This not only improves retrieval efficiency but also helps users quickly grasp text information based on text context maps.

[0168] In some embodiments of this disclosure, such as Figure 12 As shown, step C20 includes:

[0169] Step C22: Match the search terms with keywords based on the first matching rule.

[0170] Specifically, firstly, the search terms are matched with keywords based on the first matching rule, which is used to match both search terms and keywords. If a keyword is matched based on the first matching rule, the text data matching the keyword is obtained. After obtaining the matched text data, the process can proceed directly to step C28 or step C24. If no keyword is matched based on the first matching rule, the process proceeds to step C24.

[0171] Step C24: Match the search terms with the central summary based on the second matching rule.

[0172] Specifically, the search terms are matched with the central summary based on the second matching rule, which is used to match both the search terms and the central summary. If the central summary is matched based on the second matching rule, the text data matching the central summary is obtained. After obtaining the matched text data, the process can proceed directly to step C28 or step C26. If the central summary is not matched based on the second matching rule, the process proceeds to step C26.

[0173] Step C26: Match the search terms with the paragraph text based on the third matching rule.

[0174] Specifically, the search terms are matched against the paragraph text based on the third matching rule. This third matching rule is used to match both the search terms and the paragraph text. If the paragraph text is matched based on the third matching rule, the matched text data is obtained, and the process can proceed directly to step C28. If no paragraph text is matched based on the third matching rule, an empty result can be returned.

[0175] Step C28: Obtain the matching result of the search word satisfying at least one of the first matching rule, the second matching rule, and the third matching rule.

[0176] Specifically, the matching results obtained by the search word segmentation satisfying at least one of the first matching rule, the second matching rule, and the third matching rule can be summarized, sorted, and finally returned for display on the terminal.

[0177] This embodiment performs matching based on search word segmentation in the priority order of keywords, central summary, and paragraph text. It can stop the search and return the corresponding matching results after any keyword, central summary, or paragraph text is matched, depending on the fine-grained matching requirements. Alternatively, it can return the matching results after completing all matches of keywords, central summary, and paragraph text. This ensures that the search results are sorted by semantic relevance and improves search efficiency.

[0178] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0179] Based on the same inventive concept, this disclosure also provides a text data mining apparatus for implementing the text data mining method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more text data mining apparatus embodiments provided below can be found in the limitations of the text data mining method described above, and will not be repeated here.

[0180] The apparatus may include a system (including a distributed system), software (application), module, component, server, client, etc., that uses the methods described in the embodiments of this specification, combined with necessary hardware implementation. Based on the same innovative concept, the apparatuses in one or more embodiments provided in this disclosure are as described in the following embodiments. Since the implementation schemes and methods for solving the problem by the apparatus are similar, the implementation of the specific apparatus in the embodiments of this specification can refer to the implementation of the foregoing methods, and repeated details will not be repeated. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatuses described in the following embodiments are preferably implemented in software, hardware implementations, or a combination of software and hardware, are also possible and contemplated.

[0181] In some embodiments of this disclosure, such as Figure 13 As shown, a text data information mining device is provided. The device can be the aforementioned terminal, a server, or a module, component, device, or unit integrated into the terminal. The device Z00 may include:

[0182] The title extraction module Z10 is used to acquire text data, perform rule parsing on the text data, and extract titles at each level.

[0183] The title relationship module Z20 is used to establish the parent-child relationship between the titles at each level and to establish the correspondence between the titles at each level and the paragraph text and image paths in the text data.

[0184] The summary module Z30 is used to obtain the central summary of the paragraph text based on the paragraph text;

[0185] Keyword module Z40 is used to obtain keywords based on the central summary and the headings at each level;

[0186] The text context module Z50 is used to decompose and identify entities based on the text data and keywords, and construct a text context map.

[0187] In some embodiments of this disclosure, such as Figure 14 As shown, the summary module Z30 includes:

[0188] Extraction unit Z32 is used to parse the paragraph text, extract candidate sentences from the paragraph text, and form a candidate sentence pool;

[0189] Candidate unit Z34 is used to extract a first candidate summary based on the candidate statement pool;

[0190] Template unit Z36 is used to sort the pre-set abstract templates and determine the candidate abstract templates;

[0191] Synthesis unit Z38 is used to generate the central summary by combining the first candidate summary and the candidate summary template.

[0192] In some embodiments of this disclosure, such as Figure 15 As shown, the template unit Z36 includes:

[0193] The model input subunit Z362 is used to input the summary template into the summary sorting model;

[0194] The model output subunit Z364 is used to obtain the sorting of the summary template by the summary sorting model and determine the output candidate summary template;

[0195] The device further includes a model building module Z60, through which the summary ranking model is obtained, such as... Figure 16 As shown, the model building module Z60 includes:

[0196] Training set unit Z62 is used to obtain a summary template training set, which includes paragraph text and target summary corresponding to the paragraph text.

[0197] Evaluation index score unit Z64 is used to calculate the evaluation index score of the summary template and the target summary;

[0198] The sorting unit Z66 is used to sort the summary templates according to the evaluation index scores and output candidate summary templates in a specified order.

[0199] In some embodiments of this disclosure, such as Figure 17 As shown, the keyword module Z40 includes:

[0200] The prediction word unit Z42 is used to perform word and sentence segmentation on the central summary and the headings at each level to obtain prediction words;

[0201] Semantic network unit Z44 is used to establish edges based on the semantic dependencies of the predicted words and extract entity relations to generate a semantic network;

[0202] The network extraction unit Z46 is used to extract the keywords from the semantic network according to the weight allocation of the central summary and the headings at each level.

[0203] In some embodiments of this disclosure, such as Figure 18 As shown, a semantic retrieval knowledge base construction device for text data is provided. The device can be the aforementioned terminal, a server, or a module, component, device, or unit integrated into the terminal. The device X00 may include:

[0204] The basic data module X10 is used to collect text data in batches. It uses the text data information mining method described above to obtain the titles, paragraph texts, central summaries, keywords, and text context maps of the text data at each level, and to construct the basic data for the semantic retrieval knowledge base.

[0205] The writing module X20 is used to classify the titles, paragraph texts, central summaries, keywords, and text context maps of each level based on the keywords, and write them into the distributed retrieval cluster according to the categories.

[0206] In some embodiments of this disclosure, such as Figure 19 As shown, a semantic retrieval device for text data is provided. The device can be the aforementioned terminal, a server, or a module, component, device, or unit integrated into the terminal. The device M00 may include:

[0207] The search word segmentation module M10 is used to acquire search terms and perform word segmentation on the search terms to extract search words;

[0208] The search word segmentation and matching module M20 is used to match the search words in the semantic retrieval knowledge base;

[0209] The matching result module M30 is used to obtain the matching results of the search word segmentation;

[0210] The result sorting module M40 is used to sort the matching results by search word segmentation matching frequency and associate the text context map with the matching results.

[0211] The semantic retrieval knowledge base is constructed using the semantic retrieval knowledge base construction method for text data as described in the foregoing embodiments.

[0212] In some embodiments of this disclosure, such as Figure 20 As shown, the search word segmentation and matching module M20 includes:

[0213] The first matching unit M22 is used to match the search segment with the keywords based on the first matching rule;

[0214] The second matching unit M24 is used to match the search word segment with the central summary based on the second matching rule;

[0215] The third matching unit M26 is used to match the search word segment with the paragraph text based on the third matching rule;

[0216] The matching judgment unit M28 is used to obtain the matching result of the search word satisfying at least one of the first matching rule, the second matching rule, and the third matching rule.

[0217] Each module in the aforementioned text data mining device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module. It should be noted that the module division in this embodiment is illustrative and only represents a logical functional division; in actual implementation, there may be other division methods.

[0218] Based on the aforementioned embodiments of the text data information mining method, in another embodiment provided in this disclosure, a computer device is provided. This computer device may be a server, and its internal structure diagram may be as follows: Figure 21As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the above-described method embodiments.

[0219] Those skilled in the art will understand that the structure shown in the figure is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0220] Based on the aforementioned embodiments of the text data information mining method, in another embodiment provided in this disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps in the above-described method embodiments.

[0221] Based on the aforementioned embodiments of the text data information mining method, in another embodiment provided in this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0222] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0223] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0224] In the description of this specification, references to terms such as "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative descriptions of the above terms do not necessarily refer to the same embodiments or examples.

[0225] It is understood that the various embodiments of the methods described in this specification are presented in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. Relevant details can be found in the descriptions of other method embodiments.

[0226] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features of the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0227] The embodiments described above are merely illustrative of several implementations of this disclosure, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this disclosure, and these all fall within the scope of protection of this disclosure. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method for mining information from text data, characterized in that, The method includes: Obtain text data, perform rule parsing on the text data, and extract headings at each level; Establish the parent-child relationship between the headings at each level, and establish the correspondence between the headings at each level and the paragraph text and image paths in the text data; Obtain the central summary of the paragraph text based on the paragraph text; Keywords are obtained based on the central abstract and the headings at each level; Based on the text data and keywords, entity decomposition and recognition are performed to construct a text context map; The step of obtaining the central summary of the paragraph text based on the paragraph text includes: The paragraph text is parsed, and candidate sentences are extracted from the paragraph text to form a candidate sentence pool; The first candidate summary is generated by jointly generating candidate sentence attention and word attention based on the candidate sentence attention and word attention in the candidate sentence pool; Sort the pre-set abstract templates to determine candidate abstract templates; The central summary is generated by combining the first candidate summary and the candidate summary template. The process of obtaining keywords based on the central abstract and the headings at each level includes: The central summary and the headings at each level are segmented into words and sentences to obtain predicted words; Edges are established based on the semantic dependencies of the predicted words, and entity relations are extracted to generate a semantic network; Based on the weight allocation of the central abstract and the headings at each level, the keywords are extracted from the semantic network; The process of parsing the paragraph text and extracting candidate sentences from the paragraph text to form a candidate sentence pool includes: Based on the information content of the sentences in the paragraph text, sentences with an information content greater than a preset threshold are extracted to form the candidate sentence pool; The process of jointly generating the first candidate summary based on candidate sentence attention and word attention in the candidate sentence pool includes: Generative summaries are output based on the candidate statements in the candidate statement pool. The highest-scoring summary is calculated using a joint attention mechanism of words in the candidate statements and attention of the candidate statements, and is used as the first candidate summary.

2. The method according to claim 1, characterized in that, The step of sorting the pre-set abstract templates to determine candidate abstract templates includes: Input the summary template into the summary sorting model; Obtain the sorting of the summary templates by the summary sorting model, and determine the output candidate summary templates; The summary ranking model is trained through the following steps: Obtain a summary template training set, which includes paragraph text and the target summary corresponding to the paragraph text; Calculate the evaluation index scores of the summary template and the target summary; The summary templates are sorted according to the scores of the evaluation indicators, and candidate summary templates in a specified order are output.

3. A method for constructing a semantic retrieval knowledge base for text data, characterized in that, The method includes: Collect text data in batches, and use the method described in any one of claims 1-2 to obtain the titles, paragraph texts, central summaries, keywords, and text context maps of the text data, and construct the basic data for a semantic retrieval knowledge base; The headings, paragraph texts, central summaries, keywords, and text context maps at each level are classified based on the keywords and written into the distributed retrieval cluster according to the categories.

4. A semantic retrieval method for text data, characterized in that, include: Obtain search terms and extract search terms by segmenting the search terms. The search terms are matched against a semantic retrieval knowledge base; Obtain the matching results of the search word segmentation; The matching results are sorted by search word segmentation and matching frequency, and the text context map is associated with the matching results. The semantic retrieval knowledge base is constructed using the method described in claim 3.

5. The method according to claim 4, characterized in that, The step of matching the search terms in the semantic retrieval knowledge base includes: The search terms and keywords are matched based on the first matching rule; The search terms and the central summary are matched based on the second matching rule; The search terms are matched with the paragraph text based on a third matching rule; The matching result is obtained if the search word satisfies at least one of the first matching rule, the second matching rule, and the third matching rule.

6. A text data information mining device, characterized in that, The device includes: The title extraction module is used to acquire text data, perform rule parsing on the text data, and extract titles at each level. The title relationship module is used to establish the parent-child relationship between the titles at each level and to establish the correspondence between the titles at each level and the paragraph text and image paths in the text data; The summary module is used to obtain the central summary of the paragraph text based on the paragraph text. The keyword module is used to obtain keywords based on the central summary and the headings at each level; The text context module is used to decompose and identify entities based on the text data and keywords, and construct a text context map; The summary module includes: An extraction unit is used to parse the paragraph text, extract candidate sentences from the paragraph text, and form a candidate sentence pool; Candidate units are used to jointly generate a first candidate summary based on candidate sentence attention and word attention in the candidate sentence pool; The template unit is used to sort the pre-set abstract templates and determine the candidate abstract templates; A synthesis unit is configured to generate the central summary by combining the first candidate summary and the candidate summary template. The keyword module includes: The prediction word unit is used to perform word and sentence segmentation on the central summary and the headings at each level to obtain prediction words; A semantic network unit is used to establish edges based on the semantic dependencies of the predicted words and to extract entity relationships to generate a semantic network. A network extraction unit is used to extract the keywords from the semantic network according to the weight allocation of the central summary and the headings at each level; The extraction unit is further configured to extract sentences whose information content is greater than a preset threshold based on the information content of the sentences in the paragraph text, and form the candidate sentence pool. The candidate unit is further configured to output a generative summary based on the candidate statements in the candidate statement pool, and to calculate the summary with the highest score by adopting a joint mechanism of attention to words in the candidate statements and attention to the candidate statements, as the first candidate summary.

7. A semantic retrieval knowledge base construction device for text data, characterized in that, The device includes: The basic data module is used to collect text data in batches. It uses the method described in any one of claims 1-2 to obtain the titles, paragraph texts, central summaries, keywords, and text context maps of the text data, and to construct the basic data for the semantic retrieval knowledge base. The writing module is used to classify the titles, paragraph texts, central summaries, keywords, and text context maps of each level based on the keywords, and write them into the distributed retrieval cluster according to the categories.

8. A semantic retrieval device for text data, characterized in that, The device includes: The search word segmentation module is used to obtain search terms and perform word segmentation on the search terms to extract search words. The search word segmentation and matching module is used to match the search words in the semantic retrieval knowledge base. The matching results module is used to obtain the matching results of the search word segmentation; The result sorting module is used to sort the matching results by search word segmentation matching frequency and associate the text context map with the matching results. The semantic retrieval knowledge base is constructed using the method described in claim 3.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images