Text data information mining method, device, electronic device and storage medium

Abstract

The present application relates to a method, device, electronic device and storage medium for information mining of text data, the method comprising: obtaining target text data to be mined; performing deep information mining on the target text data to obtain text mining results S1 of the i-th level, wherein i is a positive integer; obtaining text data to be mined of the i+1-th level from the target text data based on the text mining results Si; performing information mining on the text data to be mined of the i+1-th level based on the text mining results Si to obtain text mining results Si+1 of the i+1-th level. In this way, it is possible to mine more comprehensive, accurate and hierarchically structured information for the target text data, thereby assisting users to understand and read the target text data from the shallow to the deep, from the surface to the inside, and layer by layer.

Description

Technical Field

[0001] The present application relates to the field of text processing, and in particular to a method, device, electronic device and storage medium for information mining of text data. Background Art

[0002] Information mining of text data is the process of using computer systems to read and understand text written by humans in order to gain business insights.

[0003] Existing information mining methods for text data are based on text summarization, including extractive summarization and generative summarization. Extractive summarization calculates the keywords of the text and the relationship between them to extract important sentences as summaries. Since only the keywords in the sentence are extracted, and the context and semantic information are ignored, inaccurate or missing information may occur. Generative summarization encodes and decodes the document content to generate a new summary text, but generative summaries are also prone to duplicate information and inaccurate information. The deep learning model of generative summaries needs to be carefully adjusted and optimized to achieve better results.

[0004] It can be seen that there is an urgent need for a new type of information mining method for text data. Summary of the invention

[0005] The present application provides a method, device, electronic device and storage medium for information mining of text data to solve the problem that the current information mining method of text data is prone to inaccurate or missing information.

[0006] In a first aspect, the present application provides a method for information mining of text data, the method comprising:

[0007] Obtain target text data to be mined;

[0008] Perform deep information mining on the target text data to obtain the text mining result S of the i-th level 1 , wherein i is a positive integer;

[0009] Based on the text mining result S i Acquire the text data to be mined at the i+1th level from the target text data;

[0010] Based on the text mining result S i Perform information mining on the text data to be mined at the i+1th level to obtain the text mining result S at the i+1th level i+1 .

[0011] In a possible implementation, the text mining result S iAcquiring the text data to be mined at the i+1th level from the target text data, including:

[0012] Obtain the text mining result S from the target text data i The corresponding source text data;

[0013] The source text data is used as the text data to be mined at the i+1th level.

[0014] In a possible implementation, the text mining result S is obtained from the target text data. i The corresponding source text data includes:

[0015] The text mining result S i Vectorized representation is performed to obtain the text mining result S i The corresponding first text vector;

[0016] Performing vectorization on each text block in the target text data to obtain a second text vector corresponding to each text block;

[0017] respectively determining a vector similarity between each of the second text vectors and the first text vector;

[0018] The second text vector whose corresponding vector similarity satisfies the set condition is determined as the target second text vector, and the text block corresponding to the target second text vector is determined as the text mining result S i The corresponding source text data.

[0019] In a possible implementation, the target text data is subjected to deep information mining to obtain the text mining result S of the i-th level. i ,include:

[0020] Decomposing the target text data to obtain multiple text blocks;

[0021] Each of the text blocks is input into the trained information mining model, so that the information mining model performs deep information mining on each of the text blocks to obtain the text mining result S of the i-th level. i .

[0022] In a possible implementation, the text mining result S i Perform information mining on the text data to be mined at the i+1th level to obtain the text mining result S at the i+1th level i+1 ,include:

[0023] Based on the text mining result S iand constructing prompt words with the text data to be mined at the i+1th level;

[0024] The text mining result S i The i+1th level of text data to be mined is input into the trained information mining model, so that the information mining model performs information mining on the i+1th level of text data to be mined based on the prompt words, and obtains the i+1th level of text mining result S i+1 .

[0025] In a possible implementation manner, in the step of obtaining the text mining result S at the i+1th level i+1 After that, it also includes:

[0026] The text mining result S of the i+1th level i+1 Vectorized representation is performed to obtain the text mining result S of the i+1th level i+1 The corresponding third text vector;

[0027] Performing vectorization on each text block in the target text data to obtain a fourth text vector corresponding to each text block;

[0028] respectively determining the vector similarity between each of the fourth text vectors and the third text vector;

[0029] When it is determined that the vector similarity corresponding to each of the fourth text vectors is less than a set similarity threshold, it is determined that the mining of the target text data is completed.

[0030] In a possible implementation, the method further includes:

[0031] Based on the text mining result S i Perform information mining on the specified source knowledge and obtain the text mining result S at the i+1th level i+1 .

[0032] In a possible implementation, the method further includes:

[0033] After obtaining the text mining result S at level i i Afterwards, the text mining result S is presented i ;

[0034] The text mining result S i Acquiring the text data to be mined at the i+1th level from the target text data, including:

[0035] In response to the text mining result S i The trigger operation is based on the text mining result S iThe text data to be mined at the i+1th level is obtained from the target text data.

[0036] In a possible implementation, the method further includes:

[0037] After obtaining the text mining result S at level i i Afterwards, the text mining result S is presented i ;

[0038] In response to the text mining result S of the i-th level i The trigger operation obtains the text mining result S from the target text data i The corresponding source text data is then presented.

[0039] In a second aspect, the present application provides an information mining device for text data, the device comprising:

[0040] A target text acquisition module is used to acquire target text data to be mined;

[0041] The first information mining module is used to perform deep information mining on the target text data to obtain the text mining result S of the i-th level. i , wherein i is a positive integer;

[0042] The second information mining module is used to mine the text based on the text mining result S i Acquire the text data to be mined at level i+1 from the target text data, and based on the text mining result S i Perform information mining on the text data to be mined at the i+1th level to obtain the text mining result S at the i+1th level i+1 .

[0043] In a possible implementation, the second information mining module includes:

[0044] A source text acquisition unit, used to acquire the text mining result S from the target text data i Corresponding source text data; taking the source text data as the text data to be mined at the i+1th level.

[0045] In a possible implementation, the source text acquisition unit includes:

[0046] The first vectorization subunit is used to vectorize the text mining result S i Vectorized representation is performed to obtain the text mining result S i The corresponding first text vector;

[0047] A second vectorization subunit is used to vectorize each text block in the target text data to obtain a second text vector corresponding to each text block;

[0048] a vector similarity calculation subunit, used to respectively determine the vector similarity between each of the second text vectors and the first text vector;

[0049] A target determination subunit is used to determine the second text vector whose vector similarity satisfies the set condition as the target second text vector, and determine the text block corresponding to the target second text vector as the text mining result S i The corresponding source text data.

[0050] In a possible implementation, the first information mining module includes:

[0051] A text decomposition unit, used for decomposing the target text data to obtain a plurality of text blocks;

[0052] The information mining unit is used to input each of the text blocks into the trained information mining model, so that the information mining model performs deep information mining on each of the text blocks to obtain the text mining result S of the i-th level. i .

[0053] In a possible implementation, the second information mining module includes:

[0054] A prompt word construction unit is used to construct a prompt word based on the text mining result S i and constructing prompt words with the text data to be mined at the i+1th level;

[0055] Model mining unit, used to convert the text mining result S i The i+1th level of text data to be mined is input into the trained information mining model, so that the information mining model performs information mining on the i+1th level of text data to be mined based on the prompt words, and obtains the i+1th level of text mining result S i+1 .

[0056] In a possible implementation, the device further includes:

[0057] The mining state determination module is used to obtain the text mining result S of the i+1th level. i+1 Then, the text mining result S of the i+1th level is i+1 Vectorized representation is performed to obtain the text mining result S of the i+1th level i+1A corresponding third text vector; vectorizing each text block in the target text data to obtain a fourth text vector corresponding to each text block; determining the vector similarity between each of the fourth text vectors and the third text vector; when it is determined that the vector similarity corresponding to each of the fourth text vectors is less than a set similarity threshold, determining that the mining of the target text data is completed.

[0058] In a possible implementation, the device further includes:

[0059] An extended mining module is used to expand the mining result S based on the text mining result S i Perform information mining on the specified source knowledge and obtain the text mining result S at the i+1th level i+1 .

[0060] In a possible implementation, the device further includes:

[0061] The presentation module is used to obtain the text mining result S at the i-th level. i Afterwards, the text mining result S is presented i ;

[0062] The second information mining module is specifically used for:

[0063] In response to the text mining result S i The trigger operation is based on the text mining result S i The text data to be mined at the i+1th level is obtained from the target text data.

[0064] In a possible implementation, the device further includes:

[0065] The presentation module is used to obtain the text mining result S at the i-th level. i Afterwards, the text mining result S is presented i ;

[0066] The tracing module is used to respond to the text mining result S of the i-th level i The trigger operation obtains the text mining result S from the target text data i The corresponding source text data is then presented.

[0067] In a third aspect, the present application provides an electronic device comprising: a processor and a memory, wherein the processor is used to execute an information mining program for text data stored in the memory to implement the information mining method for text data described in any one of the first aspects.

[0068] In a fourth aspect, the present application provides a storage medium storing one or more programs, wherein the one or more programs can be executed by one or more processors to implement the information mining method for text data described in any one of the first aspects.

[0069] The above-mentioned technical scheme provided by the embodiment of the present application has the following advantages over the prior art: the method provided by the embodiment of the present application, by acquiring the target text data to be mined, performs in-depth information mining on the target text data, and obtains multiple levels of text mining results, which can achieve multiple levels of in-depth text mining results for the target text data; and for the text mining results of any level, obtains the next level of text data to be mined from the target text data based on the text mining results, and performs information mining on the text data to be mined based on the text mining results to obtain the text mining results of the next level, because starting from the second level of text mining, information mining is continued in the target text data based on the text mining results of the previous level, so that the text mining results of different levels are progressive, which realizes the continuous acquisition of more detailed information, so that the mined information is more comprehensive and accurate, and has a hierarchical structure, thereby assisting users to understand and read the target text data from shallow to deep, from the surface to the inside, and progressively. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0071] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0072] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0073] Figure 1 A flowchart of an embodiment of a method for information mining of text data provided in an embodiment of the present application;

[0074] Figure 2 A flowchart of another method for information mining of text data provided in an embodiment of the present application;

[0075] Figure 3A flowchart of another method for information mining of text data provided in an embodiment of the present application;

[0076] Figure 4 A schematic diagram of an application scenario of a text data information mining method provided in an embodiment of the present application;

[0077] Figure 5 A block diagram of an embodiment of an information mining device for text data provided in an embodiment of the present application;

[0078] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0080] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0081] In order to solve the technical problem that the information mining method of text data in the prior art is prone to inaccurate or missing information, the present application provides an information mining method for text data, which can mine more comprehensive, accurate and hierarchical information from text data, thereby assisting users to understand and read text data from shallow to deep, from the surface to the inside, and layer by layer.

[0082] Figure 1A flowchart of an embodiment of a method for information mining of text data provided in an embodiment of the present application. In one embodiment, the execution subject of the embodiment of the present application may be a program that provides users with information mining functions for text data, which may be an application running in an independent process, a sub-application (applet) independent of the main page embedded in the client, a plug-in independent of the main page embedded in the client, a function running on a web browser (also known as WebApp), etc. In another embodiment, the execution subject of the embodiment of the present application may be a service that provides users with information mining functions for text data, which may run in the cloud. Figure 1 As shown, the process may include the following steps:

[0083] Step 101: Obtain target text data to be mined.

[0084] The above-mentioned target text data can be a document in a format such as .doc, .docx, .ppt, .txt, .pdf, etc., or it can be text data recognized from a document in an image format such as .jpg, .jpeg, etc. through OCR (Optical Character Recognition) technology, or text data extracted from a document in a format such as .ppt, .pdf, etc. through layout restoration technology.

[0085] In one embodiment, the execution subject of the embodiment of the present application is a service running on the cloud to provide users with information mining functions for text data. Then in step 101, the execution subject of the embodiment of the present application can receive a document uploaded by the client and obtain the target text data to be mined from the document. Here, as an optional implementation method, the client provides a document upload portal, and the client can call the interface of the above service provided by the cloud, so that the user can upload the document to be mined through the above document upload portal on the client side, and then the client uploads the document uploaded by the user to the cloud. As another optional implementation method, the client side is loaded with a corresponding function plug-in. When the function plug-in is triggered, the function plug-in pulls the document currently opened on the client side and uploads the document to the cloud as a document to be mined.

[0086] In one embodiment, the execution subject of the embodiment of the present application is a program that provides users with information mining functions for text data. Taking an application running in an independent process as an example, it provides a document upload portal, and the user can upload the document to be mined through the document upload portal on the client side, so that the client can obtain the document to be mined. When the document is a document in the format of .doc, .docx, .txt, etc., the execution subject of the embodiment of the present application can use the document as the target text data to be mined. When the document is a picture in the format of jpg, .jpeg, etc., the execution subject of the embodiment of the present application can use the OCR technology to identify the text data therefrom, and use the identified text data as the target text data to be mined. When the document is a document in the format of .ppt, .pdf, etc., the execution subject of the embodiment of the present application can extract text data from it through layout restoration, and use the extracted text data as the target text data to be mined.

[0087] Step 102: Perform deep information mining on the target text data to obtain the text mining result S of the i-th level. i .

[0088] The above i is a positive integer, that is, the value of i can be 1, 2, 3... That is to say, in the embodiment of the present application, deep information mining is performed on the target text data to obtain text mining results of multiple levels, such as the text mining result S of the first level. 1 , the second level text mining results S 2 , the text mining results of the third level S 3 , and so on. Among them, the second-level text mining result S 2 Take the first level text mining result S 1 Based on the information mining, the third level text mining result S 3 The second level text mining results S 2 As can be seen, the deep information mining mentioned in step 102 refers to mining the information contained in the target text data to a deeper degree, which includes multiple information mining processes from shallow to deep, so as to touch the more essential information in the target text data.

[0089] In one embodiment, deep information mining is performed on the target text data to obtain the text mining result S of the i-th level. i The specific implementation includes: disassembling the target text data to obtain multiple text blocks, inputting each text block into the trained information mining model, so that the information mining model performs deep information mining on each text block, and obtaining the text mining result S of the i-th level iHere, the text blocks obtained by decomposing in the text mining process at different levels are not exactly the same. For example, in the text mining process at the first level, the target text data is decomposed into 5 text blocks, and in the text mining process at the second level, the target text data is decomposed into 8 text blocks. It can be understood that the different numbers of text blocks have different contents, and thus the text mining results obtained by the text mining processes at different levels are also different.

[0090] It can be seen that the text mining result S of level i i It may include multiple text mining results S i These are obtained by mining information from the above multiple text blocks. Through this processing, the target text data can be summarized in blocks, so that the final mining results can more comprehensively and accurately express the original content, especially for long text data, this advantage is more obvious.

[0091] Among them, as an optional implementation method, the above-mentioned decomposition of the target text data to obtain multiple text blocks includes: parsing the target text data, obtaining the paragraph structured information of the target text data, decomposing the target text data into different text paragraphs according to the paragraph structured information of the target text data, and determining each text paragraph as a text block, thereby obtaining multiple text blocks.

[0092] As another optional implementation method, the above-mentioned decomposing the target text data to obtain multiple text blocks includes: parsing the target text data, obtaining the paragraph structured information of the target text data, decomposing the target text data into different text paragraphs according to the paragraph structured information of the target text data, and splicing the multiple text paragraphs to obtain multiple text blocks.

[0093] Among them, when splicing multiple text paragraphs, the splicing can be performed according to a fixed number of paragraphs, for example, every two text paragraphs are spliced ​​into a text block, and every three text paragraphs are spliced ​​into a text block. Here, when splicing multiple text paragraphs, overlapping splicing can be performed, that is, the same paragraphs can be included between two adjacent text blocks, for example, paragraph 1 and paragraph 2 are spliced ​​into a text block, paragraph 2 and paragraph 3 are spliced ​​into a text block, and paragraph 3 and paragraph 4 are spliced ​​into a text block. Through this processing, the semantic relevance between adjacent text blocks can be improved, thereby improving the coherence between multiple text mining results obtained by subsequent information mining of text blocks.

[0094] When splicing multiple text paragraphs, the multiple text paragraphs can also be spliced ​​based on the maximum number of tokens (here, the text blocks into which the text data is decomposed) that the information mining model can process at one time to obtain multiple text blocks. Here, the number of tokens in a single text block does not exceed the maximum number of tokens that the information mining model can process at one time.

[0095] The specific implementation of splicing multiple text paragraphs based on the maximum number of tokens that the information mining model can process at one time to obtain multiple text blocks includes: starting from the first text paragraph, traversing the text paragraphs one by one, and determining the number of tokens of all the text paragraphs currently traversed each time a text paragraph is traversed, and determining whether the number of tokens exceeds the maximum number of tokens that the information mining model can process at one time, if not, continue to traverse the next text paragraph, if more, discard the currently traversed text paragraph, and splice the previously traversed text paragraphs into one text block. Afterwards, starting from the currently traversed text paragraph, continue to traverse the text paragraphs one by one, repeat the above process until the last text paragraph is traversed to obtain multiple text blocks.

[0096] Through this processing, the single processing resources of the information mining model can be maximized and the information mining efficiency can be improved.

[0097] In another embodiment, when i is 1, that is, when performing the first level of information mining, deep information mining is performed on the target text data to obtain the first level of text mining results S 1 The specific implementation includes: decomposing the target text data to obtain multiple text blocks, inputting each text block into the trained information mining model, so that the information mining model performs information mining on each text block, and obtaining the first-level text mining result S i .

[0098] When i is not 1, that is, when conducting information mining at the second level, third level, or other level other than the first level, deep information mining of the target text data means: continuing information mining based on the text mining results of the previous level. For example, when conducting information mining at the second level, the text mining results S at the first level are used as the basis for information mining. 1 Continue to mine information as the basis for mining, and obtain the second-level text mining results S 2 When conducting information mining at the third level, the text mining results S at the second level are used. 2 Continue to mine information as a basis for mining, and obtain the third level text mining result S 3 , and so on, until the information mining is completed. The specific implementation here is described in the following steps, which will not be described in detail here.

[0099] The above-mentioned information mining model can be a large language model (Large Language Model, LLM model for short). The large language model is a model based on machine learning and natural language processing technology. It learns the ability to serve human language understanding and generation by training a large amount of text data. The core idea of ​​the large language model is to learn the patterns and language structures of natural language through large-scale unsupervised training, which can simulate the human language cognition and generation process to a certain extent. Compared with traditional NLP models, LLM can better understand and generate natural text, and can also show certain logical thinking and reasoning ability.

[0100] In one embodiment, the training process of the above information mining model includes the following steps: text collection and cleaning, word segmentation, word embedding, masking, sentence construction and training.

[0101] Among them, text collection and cleaning refers to collecting a large amount of text data, which can come from the Internet, books, news, social media, etc., and then cleaning these data to remove unnecessary HTML tags, special characters, repeated content and other noise. In addition, in order to increase data diversity, technical means can also be used to generate augmented samples, such as random cutting, rearrangement, masking, etc. of text to perform data augmentation, which helps to improve the robustness and generalization ability of the model.

[0102] Word segmentation refers to dividing the above text data into words or subwords. Dividing text data into smaller units helps the model better understand the language structure. For example: "I love natural language processing" can be divided into ["I", "love", "natural", "language", "processing"].

[0103] Word embedding refers to converting the text after word segmentation into a numerical representation that the model can understand. Word embedding technology is usually used to map each word or subword into a real vector space so that the model can process them. Among them, Word2Vec, GloVe and FastText are common word embedding methods.

[0104] Masking refers to randomly masking some words or subwords during pre-training so that the model can predict the masked words during training, which helps the model learn contextual understanding and grammar.

[0105] Sentence construction refers to constructing the segmented text into the input sequence of the model. For the autoregressive model, the input sequence is usually a continuous text paragraph.

[0106] Training means inputting the constructed sentences into the initial model, designing a series of tasks such as predicting the masked words or sentences, etc. to let the initial model return the corresponding answers based on the prompt words. Then adjust the model parameters based on the answers until the model converges to obtain the above information mining model.

[0107] Step 103: Based on the text mining result S i Get the i+1th level of text data to be mined from the target text data.

[0108] Step 104: Based on the text mining result S i Perform information mining on the text data to be mined at the i+1th level to obtain the text mining result S at the i+1th level i+1 .

[0109] As can be seen from the description of step 103, in the embodiment of the present application, after obtaining the text mining result S of the i-th level i After that, we will continue to mine the text based on the i-th level result S i Perform information mining again to obtain the text mining result S of the i+1th level i+1 The information mining mentioned here refers to a one-time information mining process.

[0110] For example, the target text data is the first-level text data to be mined, and information mining is performed on the target text data to obtain the first-level text mining results (which are the key information of the target text data). Afterwards, for the first-level text mining results, the second-level text data to be mined is obtained from the target text data, and information mining is performed on the second-level text data to be mined based on the first-level text mining results to obtain the second-level text mining results (which are the extended key information of the first-level text mining results in the target text data). Afterwards, for the second-level text mining results, the third-level text data to be mined is obtained from the target text data, and information mining is performed on the third-level text data to be mined based on the second-level text mining results to obtain the third-level text mining results (which are the extended key information of the second-level text mining results in the target text data), and so on, until the information mining is completed.

[0111] It can be seen that through this processing, the document can be reconstructed into a hierarchical structure, with the first level being the key information of the document, the second level being the expanded key information of the first level in the original text, the third level being the expanded key information of the second level in the original text, and so on, thereby achieving the continuous acquisition of more detailed information, assisting users to understand and read the document from shallow to deep, from the outside to the inside, layer by layer.

[0112] In one embodiment, based on the text mining result Si The specific implementation of obtaining the i+1th level of text data to be mined from the target text data includes: obtaining the text mining result S from the target text data i The corresponding source text data is used as the text data to be mined at the i+1th level.

[0113] As an optional implementation method, the text mining result S is obtained from the target text data. i The specific implementation of the corresponding source text data includes: i Vectorized representation is performed to obtain the text mining result S i The corresponding text vector (for the convenience of description, hereinafter referred to as the first text vector) is vectorized for each text block in the target text data to obtain the text vector corresponding to each text block (for the convenience of description, hereinafter referred to as the second text vector), and the vector similarity between each second text vector and the first text vector is determined respectively, and the second text vector whose corresponding vector similarity meets the set conditions is determined as the target second text vector, and the text block corresponding to the target second text vector is determined as the text mining result S i The corresponding source text data.

[0114] Here, the above-mentioned setting condition is, for example: the corresponding vector similarity is the largest and is greater than the set similarity threshold. When the vector similarity between the first text vector and the second text vector is greater than the set similarity threshold, it means that the text mining result and the text block corresponding to the second text vector have a relatively large relevance in content or semantics, which means that the text mining result S i It may be the result of information mining for the text block. When the vector similarity is greater than the set similarity threshold, the text block corresponding to the largest vector similarity is selected as the text mining result S. i The corresponding source text data can be combined with the text mining result S i The text block with the largest correlation between them is taken as the text mining result S i The corresponding source text data improves the accuracy of the determined source text data.

[0115] Of course, in practical applications, the above setting condition can also be that the vector similarity is greater than the set similarity threshold. i The corresponding source text data may include one or more text blocks. Through this setting, the subsequent mining scope can be expanded to mine more information.

[0116] The above-mentioned vectorized representation of text can adopt the TF (term frequency) method or the TF-IDF (term frequency-inverse frequency) method, and can also adopt other methods, which are not limited in this application.

[0117] When calculating the similarity between two text vectors, the cosine similarity calculation method, the Pearson correlation coefficient calculation method, the Euclidean distance calculation method, etc. can be used, and this application does not limit this.

[0118] The technical solution provided in the embodiment of the present application can achieve multiple levels of in-depth text mining results for the target text data by acquiring target text data to be mined, performing in-depth information mining on the target text data, and obtaining multiple levels of text mining results; and for the text mining results of any level, obtaining the next level of text data to be mined from the target text data based on the text mining results, and performing information mining on the text data to be mined based on the text mining results to obtain the next level of text mining results. Since the text mining starts from the second level, information mining is continued in the target text data based on the text mining results of the previous level, and the text mining results of different levels are progressive, which realizes the continuous acquisition of more detailed information, so that the mined information is more comprehensive and accurate, and has a hierarchical structure, thereby assisting users to understand and read the target text data from shallow to deep, from the surface to the inside, and progressively.

[0119] In one embodiment, based on the text mining result S i Perform information mining on the text data to be mined at the i+1th level to obtain the text mining result S at the i+1th level i The specific implementation includes: based on the text mining results S i The text data to be mined is used to construct the prompt word prompt, and the text mining result S i The i+1th level text data to be mined is input into the trained information mining model, so that the information mining model performs information mining on the i+1th level text data to be mined based on prompt, and obtains the i+1th level text mining result.

[0120] Prompt word engineering refers to inputting a prompt word before asking questions to the information mining model, so that the information mining model can understand and answer subsequent questions based on the prompt according to a specific pattern. In one embodiment, the following prompt is constructed based on the text mining results and the text data to be mined: You are an editor-in-chief. Your job is to supplement and expand the key information input based on the given article content. The following is the article content: {context}, which is the key information {information} of the article. Please supplement and expand the key information in points.

[0121] The above article content corresponds to the text data to be mined, and the key information corresponds to the text mining result. It can be seen that through the above embodiment, the information mining model can be used to perform information mining on the text data to be mined based on prompt, so as to supplement and expand the key information of the text mining result in points, thereby obtaining the text mining result of the next level.

[0122] It can be seen from the above description that the embodiment of the present application performs information mining on text data based on a large language model and a prompt word project. There is no need to train a large number of deep learning models, nor is there any need to perform detailed model parameter adjustment and optimization. Compared with the generative summary method in the prior art, it can save computing resources and improve the efficiency of information mining.

[0123] Figure 2 The flowchart of another method for information mining of text data provided in the present application is as follows. Figure 2 As shown, the following steps are included:

[0124] Step 201: Obtain target text data to be mined.

[0125] Step 202: Perform information mining on the target text data to obtain the first level text mining result S 1 .

[0126] Step 203: Text mining result S for level 1 1 , based on the text mining results S 1 The second level of text data to be mined is obtained from the target text data, and information mining is performed on the text data to be mined based on the text mining results to obtain the second level text mining results S 2 .

[0127] For detailed description of steps 201 to 203, please refer to the above Figure 1 The description in the illustrated embodiment will not be repeated here.

[0128] Step 204 , determine whether the mining of the target text data is completed, if so, execute step 206 , otherwise, execute step 205 .

[0129] Step 205: Text mining result S for the current level i , based on the text mining results S i Get the i+1th level of text data to be mined from the target text data, and based on the text mining result S i Perform information mining on the text data to be mined at the i+1th level to obtain the text mining result S at the i+1th level i+1 , return to execute step 204.

[0130] It can be seen from the description of step 204 and step 205 that, as an embodiment, when mining of the target text data is not completed, more detailed information can be continuously obtained from the target text data until mining of the target text data is completed.

[0131] In one embodiment, since when constructing the i+1th level of text data to be mined, the source text data corresponding to the text mining results of the i-th level in the target text data can be determined as the text data to be mined of the i+1th level, it can be considered that if the source text data corresponding to the text mining results of the i-th level does not exist in the target text data, it is impossible to continue information mining, that is, to complete the mining of the target text data. Further, under normal circumstances, at least two levels of information mining can be performed on the target text data. Therefore, in the embodiment of the present application, after completing the second level of information mining on the target text data and obtaining the second level of text mining results, it is determined whether the mining of the target text data is completed. Of course, it is not ruled out that there is only one information mining situation, and the embodiment of the present application does not limit this.

[0132] Based on this, in this embodiment, starting from the second level of information mining, each time a level of text mining results is obtained, it can be determined whether the mining of the target text data is completed by determining whether the target text data contains the source text data corresponding to the text mining results. In the case where the mining of the target text data is not completed, the mining is continuously performed according to the description of step 204. As for the detailed description of step 204, please refer to the relevant description in the above embodiment, which will not be repeated here.

[0133] Among them, as an optional implementation method, the specific implementation of determining whether the mining of the target text data is completed includes: vectorizing the text mining results of the current level that has just been completed to obtain the text vector corresponding to the text mining result (for the convenience of description, hereinafter referred to as the third text vector), vectorizing each text block in the target text data, and obtaining the text vector corresponding to each text block (for the convenience of description, hereinafter referred to as the fourth text vector), and then determining the vector similarity between each fourth text vector and the third text vector respectively. When it is determined that the vector similarity corresponding to each fourth text vector is less than the set similarity threshold, it is determined that the mining of the target text data is completed.

[0134] Here, when the vector similarity between the third text vector and the fourth text vector is greater than or equal to the set similarity threshold, it means that the text mining result and the text block corresponding to the fourth text vector have a relatively large correlation in content or semantics, which further means that the text mining result may be the result of information mining on the text block, that is, the text block may be the source text data corresponding to the text mining result. Therefore, when it is determined that there is a fourth text vector greater than or equal to the set similarity threshold, it is determined that the source text data corresponding to the text mining result exists in the target text data. According to the above description, information mining can be continued at this time, which means that mining of the target text data has not been completed at this time.

[0135] On the contrary, when it is determined that the vector similarity corresponding to each fourth text vector is less than the set similarity threshold, it is determined that the source text data corresponding to the text mining result does not exist in the target text data, which means that it can be determined that the mining of the target text data has been completed.

[0136] Step 206: Based on the text mining result S i Perform information mining on the specified source knowledge and obtain the text mining result S at the i+1th level i+1 .

[0137] As can be seen from the description of step 206, when the mining of the target text data is completed, it is also possible to jump out of the scope of the target text data and continue to mine information in other knowledge, so as to obtain more information to better assist the user in reading the target text data. Here is another point to explain that in actual applications, information mining in other knowledge is not necessarily limited to the mining of the target text data, so that more information can be obtained.

[0138] In one embodiment, the specified source knowledge may be the knowledge possessed by the information mining model during training. Currently, when the information mining model is trained using a large language model, tens of millions of corpora are used for training, so that the information mining model itself has a relatively strong knowledge base capability.

[0139] Furthermore, in one embodiment, information mining may be continued in designated source knowledge for each level of text mining results, thereby obtaining more information.

[0140] In another embodiment, information mining may be continued in designated source knowledge for text mining results at a partial level, thereby obtaining more information.

[0141] Here, the text mining results of some levels can be the text mining results of the last level mined from the target text data, that is, after completing the mining of the target text data, the level expansion mining can be performed vertically on the current mining results based on the specified source knowledge. Further, the cutoff condition for information mining based on the specified source knowledge can be limited, for example, it is set to perform N levels of information mining based on the specified source knowledge, where N is an integer greater than 1, such as 3. Such a setting can avoid endless level expansion mining and waste of computing resources.

[0142] Some levels of text mining results can also be user-selected text mining results. That is, after completing the mining of target text data, users can manually select text mining results that need to be further expanded according to actual business needs. This process can improve user experience and avoid mining information that users do not need, thereby wasting computing resources.

[0143] It should be noted here that, as can be seen from the above description, a level of text mining results may also include multiple ones. Therefore, the above-mentioned extended mining of text mining results of each level or part of the levels may also refer to extended mining of part of the text mining results in a level, and this application does not limit this.

[0144] In addition, in one embodiment, the execution of the above step 206 can also be manually triggered by the user. That is to say, when the mining of the target text data is completed, the user can manually select whether to jump out of the own scope of the target text data according to actual business needs, and continue to perform extended mining in other knowledge. For example, when the mining of the target text data is completed, the execution subject of the embodiment of the present application can output an instruction message for indicating that the mining of the target text data has been completed, and the user is asked to indicate whether to perform extended mining. The user responds to the instruction message and makes further instructions so that the execution subject of the embodiment of the present application determines whether to perform extended mining according to the user's instructions. The above instruction message can be presented by a function control (such as a pop-up window), or it can be presented by a voice message. Correspondingly, the user can make a click operation on the corresponding button in the function control to make further instructions, or it can make further instructions by natural language, and the embodiment of the present application does not limit this.

[0145] This processing can improve user experience and avoid mining information that users do not need, thereby wasting computing resources.

[0146] Figure 2The illustrated embodiment continuously obtains more detailed information from the target text data without completing information mining of the target text data, thereby assisting the user to understand and read the target text data from shallow to deep and from the outside to the inside, layer by layer. After completing information mining of the target text data, the text mining results mined from the target text data are continued to be expanded in other knowledge, so as to jump out of the scope of the target text data and obtain more information to better assist the user in reading the target text data.

[0147] Figure 3 The following is a flow chart of another embodiment of a text data information mining method provided in the present application. Figure 3 As shown, the following steps are included:

[0148] Step 301: Obtain target text data to be mined.

[0149] Step 302: Perform deep information mining on the target text data to obtain the text mining result S of the i-th level. i .

[0150] For detailed description of step 301 and step 302, please refer to the above Figure 1 The description in the illustrated embodiment will not be repeated here.

[0151] Step 303: After obtaining the text mining result S i Afterwards, the text mining results S are presented i .

[0152] Step 304: In response to the text mining result S i The trigger operation is based on the text mining result S i Get the i+1th level of text data to be mined from the target text data, and based on the text mining result S i Perform information mining on the text data to be mined at the i+1th level to obtain the text mining result S at the i+1th level i+1 .

[0153] To facilitate understanding of the above steps 303 and 304, the following is shown: Figure 4 Schematic diagram of the example scenario.

[0154] like Figure 4 As shown in Figure 1, after information mining is performed on the target text data, the first level text mining result S is obtained. 1 Afterwards, the text mining results S of the first level can be presented on the interface 1 Among them, Figure 4 As shown, the text mining results S at the first level 1 Including M1, M2, and M3.

[0155] Assuming that the user wants to continue to perform in-depth mining on the first-level text mining result M3, the user can select the text mining result on the interface and further trigger the in-depth mining of the text mining result. At this time, the execution subject of the embodiment of the present application responds to the triggering operation on the text mining result and performs a deeper level of mining on the text mining result to obtain the second-level text mining result S 2 Among them, Figure 4 As shown, the text mining results of the second level S 2 Including M31, M32, M33 and M34.

[0156] As for the specific implementation of the execution subject of the embodiment of the present application to perform a deeper level of mining on the text mining results, please refer to the above related description, which will not be repeated here.

[0157] In addition, if Figure 4 As shown, when presenting the text mining results, the text mining results can be folded, and when the user triggers the presentation of the text mining results, such as clicking Figure 4 When the user clicks “Click to expand” on the interface shown, the complete text mining results are presented on the interface. Figure 4 When you click “Collect” on the interface shown, the text mining results presented on the interface will be collapsed.

[0158] like Figure 4 As shown, the interface also provides an "export mind map" icon. When the "export mind map" icon is triggered, the execution subject of the embodiment of the present application can generate a mind map of the mining results according to the hierarchical structure of the mining results currently obtained, and store the mind map. Here, the execution subject of the embodiment of the present application can store the exported mind map in a default path or in a path specified by the user.

[0159] Figure 3 The process shown presents the text mining results on the interface, responds to the trigger operation for the text mining results of any level, obtains the text data to be mined at the next level from the target text data based on the triggered text mining results, and performs information mining on the text data to be mined based on the triggered text mining results to obtain the text mining results of the next level. A solution is provided for the user to manually select and trigger further mining. In this solution, multi-level information mining can be achieved through simple user operations, thereby improving the user experience.

[0160] In one embodiment, the present application also provides the following technical solutions:

[0161] After obtaining the text mining results S at any level i Afterwards, the text mining results S are presentedi , in response to the text mining result S i The trigger operation obtains the text mining result S from the target text data i The corresponding source text data is presented. Among them, the execution subject of the embodiment of the present application can present the above-mentioned source text data through a separate interface, or can present the above-mentioned source text data on the display interface of the text mining results. For example, the corresponding source text data is presented below the text mining results. Furthermore, in the case where both the text mining results and the corresponding source text data are displayed on the same interface, the source text data can be folded by referring to the above-mentioned method of folding the text mining results, and the source text data can be presented on the interface when the user triggers the display. This makes the user's interactive experience more friendly.

[0162] It can be seen that the technical solution provided in the embodiment of the present application can also trace the text mining results and improve the user's reading experience.

[0163] In addition, those skilled in the art can also design other user interaction functions based on the above embodiments, such as a one-key copy function. The one-key copy function means that the user can copy the text mining results by triggering the corresponding function control provided on the interface, thereby facilitating the user to quote the text mining results and improving the user experience.

[0164] Figure 5 This is a block diagram of an embodiment of a text data information mining device provided in an embodiment of the present application. Figure 5 As shown, the device comprises:

[0165] A target text acquisition module 51 is used to acquire target text data to be mined;

[0166] The first information mining module 52 is used to perform deep information mining on the target text data to obtain the text mining result S of the i-th level. i , wherein i is a positive integer;

[0167] The second information mining module 53 is used to mine the text based on the text mining result S i Acquire the text data to be mined at level i+1 from the target text data, and based on the text mining result S i Perform information mining on the text data to be mined at the i+1th level to obtain the text mining result S at the i+1th level i+1 .

[0168] In a possible implementation, the second information mining module 53 includes:

[0169] A source text acquisition unit, used to acquire the text mining result S from the target text data i Corresponding source text data; taking the source text data as the text data to be mined at the i+1th level.

[0170] In a possible implementation, the source text acquisition unit includes:

[0171] The first vectorization subunit is used to vectorize the text mining result S i Perform vectorization to obtain the text mining result S i The corresponding first text vector;

[0172] A second vectorization subunit is used to vectorize each text block in the target text data to obtain a second text vector corresponding to each text block;

[0173] a vector similarity calculation subunit, used to respectively determine the vector similarity between each of the second text vectors and the first text vector;

[0174] A target determination subunit is used to determine the second text vector whose vector similarity satisfies the set condition as the target second text vector, and determine the text block corresponding to the target second text vector as the text mining result S i The corresponding source text data.

[0175] In a possible implementation, the first information mining module 52 includes:

[0176] A text decomposition unit, used for decomposing the target text data to obtain a plurality of text blocks;

[0177] The information mining unit is used to input each of the text blocks into the trained information mining model, so that the information mining model performs deep information mining on each of the text blocks to obtain the text mining result S of the i-th level. i .

[0178] In a possible implementation, the second information mining module 53 includes:

[0179] A prompt word construction unit is used to construct a prompt word based on the text mining result S i and constructing prompt words with the text data to be mined at the i+1th level;

[0180] Model mining unit, used to convert the text mining result S iThe i+1th level of text data to be mined is input into the trained information mining model, so that the information mining model performs information mining on the i+1th level of text data to be mined based on the prompt words, and obtains the i+1th level of text mining result S i+1 .

[0181] In a possible implementation, the device further includes:

[0182] The mining state determination module is used to obtain the text mining result S of the i+1th level. i+1 Then, the text mining result S of the i+1th level is i+1 Vectorized representation is performed to obtain the text mining result S of the i+1th level i+1 A corresponding third text vector; vectorizing each text block in the target text data to obtain a fourth text vector corresponding to each text block; determining the vector similarity between each of the fourth text vectors and the third text vector; when it is determined that the vector similarity corresponding to each of the fourth text vectors is less than a set similarity threshold, determining that the mining of the target text data is completed.

[0183] In a possible implementation, the device further includes:

[0184] An extended mining module is used to expand the mining result S based on the text mining result S i Perform information mining on the specified source knowledge and obtain the text mining result S at the i+1th level i+1 .

[0185] In a possible implementation, the device further includes:

[0186] The presentation module is used to obtain the text mining result S at the i-th level. i Afterwards, the text mining result S is presented i ;

[0187] The second information mining module 53 is specifically used for:

[0188] In response to the text mining result S i The trigger operation is based on the text mining result S i The text data to be mined at the i+1th level is obtained from the target text data.

[0189] In a possible implementation, the device further includes:

[0190] The presentation module is used to obtain the text mining result S at the i-th level. i Afterwards, the text mining result S is presented i;

[0191] The tracing module is used to respond to the text mining result S of the i-th level i The trigger operation obtains the text mining result S from the target text data i The corresponding source text data is then presented.

[0192] like Figure 6 As shown, an embodiment of the present application provides an electronic device, including a processor 611, a communication interface 612, a memory 613 and a communication bus 614, wherein the processor 611, the communication interface 612, and the memory 613 communicate with each other through the communication bus 614.

[0193] Memory 613, used for storing computer programs;

[0194] In one embodiment of the present application, the processor 611 is used to implement the information mining method of text data provided by any one of the aforementioned method embodiments when executing the program stored in the memory 613, including:

[0195] Obtain target text data to be mined;

[0196] Perform deep information mining on the target text data to obtain the text mining result S of the i-th level 1 , wherein i is a positive integer;

[0197] Based on the text mining result S i Acquire the text data to be mined at the i+1th level from the target text data;

[0198] Based on the text mining result S i Perform information mining on the text data to be mined at the i+1th level to obtain the text mining result S at the i+1th level i+1 .

[0199] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the information mining method for text data provided in any of the aforementioned method embodiments are implemented.

[0200] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0201] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0202] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0203] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for information mining of text data, characterized in that: The method comprises: Obtain target text data to be mined; Perform deep information mining on the target text data to obtain the text mining result S of the i-th level i , wherein i is a positive integer; Based on the text mining result S i Acquire the text data to be mined at the i+1th level from the target text data; Based on the text mining result S i Perform information mining on the text data to be mined at the i+1th level to obtain the text mining result S at the i+1th level i+1 .

2. The method according to claim 1, characterized in that The text mining result S i Acquiring the text data to be mined at the i+1th level from the target text data, including: Obtain the text mining result S from the target text data i The corresponding source text data; The source text data is used as the text data to be mined at the i+1th level.

3. The method according to claim 2, characterized in that The step of obtaining the text mining result S from the target text data i The corresponding source text data includes: The text mining result S i Vectorized representation is performed to obtain the text mining result S i The corresponding first text vector; Performing vectorization on each text block in the target text data to obtain a second text vector corresponding to each text block; respectively determining a vector similarity between each of the second text vectors and the first text vector; The second text vector whose vector similarity satisfies the set condition is determined as the target second text vector, and the text block corresponding to the target second text vector is determined as the text mining result S i The corresponding source text data.

4. The method according to claim 1, characterized in that: The target text data is subjected to deep information mining to obtain the text mining result S of the i-th level. i ,include: Decomposing the target text data to obtain multiple text blocks; Each of the text blocks is input into the trained information mining model, so that the information mining model performs deep information mining on each of the text blocks to obtain the text mining result S of the i-th level. i .

5. The method according to claim 1, characterized in that The text mining result S i Perform information mining on the text data to be mined at the i+1th level to obtain the text mining result S at the i+1th level i+1 ,include: Based on the text mining result S i and constructing prompt words with the text data to be mined at the i+1th level; The text mining result S i The i+1th level of text data to be mined is input into the trained information mining model, so that the information mining model performs information mining on the i+1th level of text data to be mined based on the prompt words, and obtains the i+1th level of text mining result S i+1 .

6. The method according to claim 1, characterized in that In the above, the text mining result S of the i+1th level is obtained i+1 After that, it also includes: The text mining result S of the i+1th level i+1 Vectorized representation is performed to obtain the text mining result S of the i+1th level i+1 The corresponding third text vector; Performing vectorization on each text block in the target text data to obtain a fourth text vector corresponding to each text block; respectively determining the vector similarity between each of the fourth text vectors and the third text vector; When it is determined that the vector similarity corresponding to each of the fourth text vectors is less than a set similarity threshold, it is determined that the mining of the target text data is completed.

7. The method according to claim 1, characterized in that The method further comprises: Based on the text mining result S i Perform information mining on the specified source knowledge and obtain the text mining result S at the i+1th level i+1 .

8. The method according to claim 1, characterized in that: The method further comprises: After obtaining the text mining result S at level i i Afterwards, the text mining result S is presented i ; The text mining result S i Acquiring the text data to be mined at the i+1th level from the target text data, including: In response to the text mining result S i The trigger operation is based on the text mining result S i The text data to be mined at the i+1th level is obtained from the target text data.

9. The method according to claim 1, characterized in that: The method further comprises: After obtaining the text mining result S at level i i Afterwards, the text mining result S is presented i ; In response to the text mining result S of the i-th level i The trigger operation obtains the text mining result S from the target text data i The corresponding source text data is then presented.

10. An information mining device for text data, characterized in that: The device comprises: A target text acquisition module is used to acquire target text data to be mined; The first information mining module is used to perform deep information mining on the target text data to obtain the text mining result S of the i-th level. i , wherein i is a positive integer; The second information mining module is used to mine the text based on the text mining result S i Acquire the text data to be mined at level i+1 from the target text data, and based on the text mining result S i Perform information mining on the text data to be mined at the i+1th level to obtain the text mining result S at the i+1th level i+1 .

11. An electronic device, characterized in that: include: A processor and a memory, wherein the processor is used to execute an information mining program for text data stored in the memory to implement the information mining method for text data according to any one of claims 1 to 9.

12. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the information mining method for text data according to any one of claims 1 to 9.

Images