Text analysis system, method, electronic device and storage medium

Abstract

The present application provides a text analysis system, method, electronic device and storage medium. The system includes a preprocessing module and a text analysis module; the text analysis module includes at least one of a multi-level text classification unit, a text summary extraction unit, a label element extraction unit, an intelligent text recommendation unit and an emotional intention analysis unit; the multi-level text classification unit is used to classify the text to be processed according to the text vector; the text summary extraction unit is used to integrate the text information to be processed according to the text vector to obtain a text summary; the label element extraction unit is used to extract labels from the text to be processed according to the text vector and collect index data, and obtain text index data information according to the labels and the index data; the intelligent text recommendation unit is used to determine a recommended text from the text to be processed according to the text vector; the emotional intention analysis unit is used to determine the emotional tendency of the text to be processed and the intention category of the text according to the text vector. This solution can meet various application scenarios.

Description

Technical Field

[0001] This application relates to the technical field of information processing, and in particular, to a text analysis system, method, electronic device, and storage medium. Background Art

[0002] Text Analysis, as an important research direction in Natural Language Processing (NLP), aims to extract key information from text for quantification, so as to be applied to subsequent text processing.

[0003] In the prior art, basic text processing functions such as Chinese word segmentation, text vectorization, and keyword extraction can be performed on text. On this basis, three types of Chinese text analysis services, namely sentiment analysis, text classification, and text clustering, are provided, which can support the basic Chinese processing functions of most data.

[0004] However, since the text analysis services provided by the prior art are relatively basic and the models are fixed, complex functions cannot be supported, which greatly limits their applicable scenarios and affects user experience. Summary of the Invention

[0005] In view of this, embodiments of this application provide a text analysis system, method, electronic device, and storage medium, which can meet various application scenarios.

[0006] According to the first aspect of the embodiments of this application, a text analysis system is provided, including: a preprocessing module and a text analysis module; the text analysis module includes at least one of a multi-level text classification unit, a text summary extraction unit, a label element extraction unit, an intelligent text recommendation unit, and a sentiment intention analysis unit; the preprocessing module is configured to perform Chinese word segmentation, stop word removal, and text vectorization processing on the text to be processed in sequence to obtain a text vector; the multi-level text classification unit is configured to classify the text to be processed according to the text vector; the text summary extraction unit is configured to integrate information of the text to be processed according to the text vector to obtain a text summary; the label element extraction unit is configured to extract labels from the text to be processed according to the text vector and collect index data, and obtain text index data information according to the labels and the index data; the intelligent text recommendation unit is configured to determine a recommended text from the text to be processed according to the text vector; the sentiment intention analysis unit is configured to determine the sentiment tendency of the text to be processed and the intention category of the text according to the text vector.

[0007] In a possible implementation method, the multi-level text classification unit is used to perform the following processing: input the text vector into a classification model to obtain the first-level category probability of the text to be processed; if the first-level category probability is greater than a preset probability threshold, determine the first-level category of the text to be processed according to the first-level category probability, and perform secondary classification on the text to be processed through a dictionary matching algorithm to determine the second-level category of the text to be processed; if the first-level category probability is less than or equal to the probability threshold, determine the first-level category of the text to be processed through a clustering model, and perform secondary classification on the text to be processed through the dictionary matching algorithm to determine the second-level category of the text to be processed.

[0008] In a possible implementation method, the multi-level text classification unit is used to perform the following processing through the clustering model: calculate the text category of the text to be processed through the elbow method; cluster the text category of the text to be processed through a K-Means model to obtain a text category clustering result; extract the central word of the text to be processed as the clustering center; perform label aggregation on the clustering center to obtain a label aggregation result.

[0009] In a possible implementation method, the text summary extraction unit is used to perform the following processing: determine the sentence vector corresponding to each sentence in the text to be processed according to the text vector, where the text to be processed includes at least two sentences; determine the similarity between adjacent sentences in the text to be processed according to the sentence vectors of each sentence in the text to be processed; determine the importance score corresponding to each sentence in the text to be processed according to the similarity between adjacent sentences in the text to be processed; merge the K1 sentences with relatively large importance scores in the text to be processed according to the syntactic relationship to obtain the text summary, where K1 is a positive integer greater than or equal to 2.

[0010] In a possible implementation method, the label element extraction unit is used to perform the following processing: input the text vector into a multi-classification model to match the text to be processed with predefined labels to obtain the text to be processed after matching; input the text to be processed after matching into an LSTM model to perform probability prediction on the text to be processed after matching to obtain the label; classify the text vector through a dictionary matching algorithm to obtain the index data; perform syntactic dependency analysis on the label and the index data to obtain text index data information.

[0011] In a possible implementation method, the intelligent text recommendation unit is used to perform the following processing: input the text vector into a label generation model to obtain text labels; calculate the similarity of the texts corresponding to each of the text labels through an Item-base collaborative filtering algorithm, and output K2 sentences with relatively large similarity to obtain a first text recommendation list, where K2 is a positive integer greater than or equal to 2; extract features from the text vector to obtain the extracted features of the text vector; perform dimensionality reduction processing on the extracted features through an SVD model to obtain a dimensionality reduction processing result; calculate the cosine similarity of the texts corresponding to the dimensionality reduction processing result, and output K3 sentences with relatively large similarity to obtain a second text recommendation list, where K3 is a positive integer greater than or equal to 2; merge the first text recommendation list and the second text recommendation list to obtain a merged recommendation list; sort the texts corresponding to the merged recommendation list according to time and similarity values, and output K4 sentences with relatively large similarity to obtain the recommended text, where K4 is a positive integer greater than or equal to 2.

[0012] In a possible implementation method, the sentiment intention analysis unit is used to perform the following processing: input the text vector into an LSTM model, and obtain first text features by extracting sentiment information in the text to be processed; calculate the sentiment category of the text through a fully connected network for the first text features to obtain the sentiment tendency of the text; input the text vector into a TextCNN model, and use convolutional calculation to extract the text features of the text to be processed to obtain second text features; input the second text features into the fully connected network to obtain the intention category of the text.

[0013] According to the second aspect of the embodiments of the present application, a text analysis method is provided. The method includes: obtaining a text vector by successively performing Chinese word segmentation, stop word removal, and text vectorization processing on the text to be processed; inputting the text to be processed into at least one text analysis unit according to the text vector for processing to obtain an analysis result, where the text analysis includes a multi-level text classification unit, a text summary extraction unit, a label element extraction unit, an intelligent text recommendation unit, and a sentiment intention analysis unit. The multi-level text classification unit performs: classifying the text to be processed according to the text vector. The text summary extraction unit performs: integrating information of the text to be processed according to the text vector to obtain a text summary. The label element extraction unit extracts labels from the text to be processed and collects index data according to the text vector, and obtains text index data information according to the labels and the index data. The intelligent text recommendation unit performs: determining recommended text from the text to be processed according to the text vector. The sentiment intention analysis unit performs: determining the sentiment tendency of the text to be processed and the intention category of the text according to the text vector.

[0014] According to a third aspect of the embodiments of the present application, an electronic device is provided, including: a processor, a memory, a communication interface, and a communication bus. The processor, the memory, and the communication interface complete communication with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform the operations corresponding to the method described in the second aspect.

[0015] According to a fourth aspect of the embodiments of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, it implements the method described in the second aspect.

[0016] According to a fifth aspect of the embodiments of the present application, a computer program product is provided. The computer program product is tangibly stored on a computer-readable medium and includes computer-executable instructions. When the computer-executable instructions are executed, at least one processor is caused to perform the method described in the second aspect.

[0017] Based on the text analysis system provided by the above solution, first, the text to be processed is preprocessed. After Chinese word segmentation, stop word removal, and text vectorization processing, a text vector is obtained. Secondly, according to the different application scenarios of the text, the obtained preprocessing result, that is, the text vector, can be input into different text analysis modules. The multi-level text classification unit classifies the text to be processed according to the text vector. The text summary extraction unit integrates the information of the text to be processed according to the text vector to obtain a text summary. The label element extraction unit extracts labels from the text to be processed and collects index data according to the text vector, and obtains text index data information according to the labels and index data. The intelligent text recommendation unit determines the recommended text from the text to be processed according to the text vector. The sentiment intention analysis unit can determine the sentiment tendency of the text to be processed and the intention category of the text according to the text vector. This solution preprocesses the text to be processed, and then inputs the text vector obtained after preprocessing into each text processing unit included in the text analysis module, so that the user only needs to input the text to be processed into the system of this solution to realize the automatic processing of the text to be processed and obtain text analysis results that meet different application scenarios. Description of the Drawings

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments recorded in the embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings.

[0019] Figure 1Schematic diagram of a text analysis system provided by an embodiment of the present application;

[0020] Figure 2 Flowchart of the operation of a multi-level text classification unit provided by an embodiment of the present application;

[0021] Figure 3 Flowchart of the operation of a multi-level text classification unit through a clustering model provided by an embodiment of the present application;

[0022] Figure 4 Flowchart of the operation of a text summary extraction unit provided by an embodiment of the present application;

[0023] Figure 5 Flowchart of the operation of a label element extraction unit provided by an embodiment of the present application;

[0024] Figure 6 Flowchart of the operation of an intelligent text recommendation unit provided by an embodiment of the present application;

[0025] Figure 7 Flowchart of the operation of an emotion intention analysis unit provided by an embodiment of the present application;

[0026] Figure 8 Flowchart of a text analysis method provided by an embodiment of the present application;

[0027] Figure 9 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0028] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part rather than all of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application shall fall within the scope of protection of the embodiments of the present application.

[0029] The terms used in the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The singular forms "a", "the" and "said" used in the present application and the appended claims are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0030] It should be understood that although terms such as first, second, and third may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to a determination".

[0031] Text analysis system

[0032] Figure 1 It is a schematic diagram of a text analysis system provided by an embodiment of this application. As Figure 1 shown, the text analysis system 100 includes a preprocessing module 110 and a text analysis module 120:

[0033] The text analysis module 110 includes at least one of a multi-level text classification unit 10, a text summary extraction unit 20, a label element extraction unit 30, an intelligent text recommendation unit 40, and an emotion intention analysis unit 50.

[0034] The preprocessing module 120 is configured to perform Chinese word segmentation, stop word removal, and text vectorization processing on the text to be processed in sequence to obtain a text vector.

[0035] The text to be processed may be structured text or unstructured text. A typical representative of unstructured text data is the documents in a library database. These documents may contain structured fields such as title, author, publication date, length, classification, etc., and may also contain a large amount of unstructured text components such as abstracts and body content.

[0036] Chinese word segmentation is to perform word segmentation processing on the original Chinese text by using the jieba word segmentation algorithm to obtain a phrase corpus that retains the context relationship.

[0037] Stop word removal is to filter and screen the stop words in the text by constructing a stop word dictionary for the phrase corpus obtained by Chinese word segmentation to remove the phrases that do not contain semantic information.

[0038] By using TF-IDF and text vectorization to convert the text into a digital representation method, TF-IDF obtains the data representation of a phrase by calculating the frequency and importance of the phrase appearing in the corpus. As shown in formulas (1), (2), and (3), text vectorization is to use a neural network to calculate the context relationship between phrases to obtain the semantic information contained therein, thereby obtaining a vectorized representation of the text.

[0039]

[0040]

[0041] tf-idf i,j =tf i,j ×idf i #(3)

[0042] Among them, tf i，j is the word frequency, which is the phrase n i，j In the document d j The frequency of occurrence in , idf i represents the inverse document frequency, which is the number of corpus phrases|| and the number of phrases containing the phrase t i The TF-IDF value of the target phrase is obtained by multiplying the term frequency by the inverse document frequency, which is used as the digital representation of the term.

[0043] The multi-level text classification unit 10 is used to classify the text to be processed according to the text vector.

[0044] Multi-level classification can divide text into several major categories, each of which is divided into many subcategories, and each subcategory is further divided into subcategories. For example, the text to be processed can be divided into two major categories: topic classification and question-answering tasks. The topic classification can be divided into three subcategories: finance, sports, and military, while the question-answering tasks can be divided into two subcategories: yes and no.

[0045] The text summary extraction unit 20 is used to integrate information of the text to be processed according to the text vector to obtain a text summary.

[0046] Text summary extraction requires extracting key semantic information from the processed text based on the text vector through an algorithm model, integrating the key semantic information, and then reorganizing the language of the integrated semantic information to form a summary.

[0047] The label element extraction unit 30 is used to extract labels from the text to be processed and collect index data based on the text vector, and obtain text index data information based on the labels and index data.

[0048] The label element extraction unit is used to extract labels from the content of the processed text based on the text vector, and collect corresponding indicator data for the extracted labels, so as to determine the one-to-one correspondence between "labels and data" based on the extracted labels and indicator data and output them, that is, the text indicator data information.

[0049] In an implementable case, the text to be processed can be various public reports, annual reports, yearbooks, etc. Taking the reports and hospital data publicly available from a certain unit as an example currently, these data contain important indicator information such as the total number of health institutions, the number of beds in health institutions, health personnel, and total health expenditure. By extracting tags from the text to be processed and collecting indicator data, text indicator data information can be obtained. For example, in a piece of available text to be processed: At the end of 2021, the total number of medical and health institutions in the country was 1,030,935, the number of beds in medical and health institutions in the country was 9.448 million, the total number of health personnel in the country was 13.983 million, and the total health expenditure in the country was initially estimated to be 7,559.36 billion yuan. In this piece of text to be processed, four tags can be generated: the total number of health institutions, the number of beds in health institutions, health personnel, and total health expenditure, and the corresponding indicator data for each tag are respectively: 1,030,935, 9.448 million, 13.983 million, and 7,559.36 billion yuan.

[0050] The intelligent text recommendation unit 40 is used to determine the recommended text from the text to be processed according to the text vector.

[0051] The intelligent text recommendation unit is used to calculate and determine the recommended text from the text to be processed according to the text vector, and output it to the customer.

[0052] In an implementable case, for example, in a piece of available text to be processed, the text words extracted from the text to be processed are the two words "marriage and love". Through automatic calculation, texts similar to "marriage and love" can be recommended, such as: "pre-marital check-up", "marriage and love website", "marriage and love market", and "marriage and love website", etc.

[0053] The sentiment intention analysis unit 50 is used to determine the sentiment tendency and intention category of the text to be processed according to the text vector.

[0054] The sentiment intention analysis unit is used to screen the text features in the text to be processed from the perspectives of time and space according to the text vector, and then analyze the sentiment tendency and intention category contained in the text to be processed, so as to help the machine better understand the user's needs.

[0055] In the embodiment of the present application, by inputting the text to be processed into the text analysis system designed by the present invention, while providing basic text preprocessing functions such as Chinese word segmentation, stop word removal, and text vectorization for the text to be processed, at least one of multi-level text classification, text summary extraction, label element extraction, intelligent text recommendation, and sentiment intention analysis is performed on the text to be processed according to the text vector, and it can be used in different scenarios such as resource integration systems, file management systems, and information portals.

[0056] Figure 2It is a flowchart of the operation of a multi-level text classification unit provided by an embodiment of the present application. As Figure 2 shown, the multi-level text classification unit 10 can classify the text at multiple levels through the following steps 201 to 204:

[0057] Step 201: Input the text vector into the classification model to obtain the probability of the first-level category of the text to be processed.

[0058] The classification model can be an XgBoost classification model, an NB model, a random forest model (RF), an SVM classification model, a KNN classification model, a neural network classification model, etc. Taking the XgBoost classification model as an example, the XgBoost classification model is trained on the data in the database to obtain the first-level category text. When the text vector is input into the classification model, the first-level category of the text to be processed is predicted, and the files belonging to the same prediction result are placed in the same file library directory, and at the same time, the probability of the first-level category of the text to be processed is obtained.

[0059] Step 202: Determine whether the probability of the first-level category is greater than the preset probability threshold. If it is, execute Step 203; otherwise, execute Step 204.

[0060] Step 203: Determine the first-level category of the text to be processed according to the probability of the first-level category, and perform secondary classification on the text to be processed through the dictionary matching algorithm to determine the second-level category of the text to be processed, and end the current process.

[0061] In an implementable example, assume that the obtained first-level categories are "management", "sales", and "technology", and assume that the obtained probability of the first-level category is P, and the preset probability threshold is 80%. When the probability P of the first-level category of the text to be processed obtained is > 80%, then perform secondary classification on the text to be processed through the dictionary matching algorithm to obtain the second-level category of the text to be processed. For example, for the first-level category "management", the second-level categories of the text to be processed obtained are "decision-making", "planning", and "organization".

[0062] Step 204: Determine the first-level category of the text to be processed through the clustering model, and perform secondary classification on the text to be processed through the dictionary matching algorithm to determine the second-level category of the text to be processed.

[0063] Based on the above example, when P ≤ 80%, it indicates that the classification model cannot classify the text to be processed, so the first-level category is obtained. Therefore, it is necessary to cluster the text to be processed through the clustering model to determine the first-level category of the text to be processed, and then perform secondary classification on the text to be processed with the determined first-level category through the dictionary matching algorithm to determine the second-level category of the text to be processed.

[0064] In the embodiments of the present application, by inputting the text vector into the classification model, the first-level category probability of the text to be processed is obtained, and the magnitude of the first-level category probability and the preset probability threshold is judged. If it is greater, the first-level category of the processed text is determined according to the first-level category probability; if it is less than or equal, the first-level category of the text to be processed is determined through the clustering model. Finally, the text to be processed with the determined first-level category is secondarily classified through the dictionary matching algorithm to determine the second-level category of the text to be processed. In this step, the first-level category of the text to be processed is predicted through different models, so as to realize the category prediction of different texts to be processed, realize the multi-level classification of different texts to be processed, and make the text analysis system more universal.

[0065] Figure 3 The working flowchart of a multi-level text classification unit provided by the embodiments of the present application through the clustering model. As Figure 3 shown, the multi-level text classification unit 10 can perform the following processing steps 301 to 305 through the clustering model:

[0066] Step 301, calculating the text category of the text to be processed through the elbow method.

[0067] Calculate the text category of the text to be processed through the elbow method. For example, the calculated categories are "A", "B" and "C".

[0068] Step 302, clustering the text to be processed through the K-Means model to obtain the text category clustering result.

[0069] In an implementable example, assume that the text to be processed includes different sentences, namely: "a", "b", "c", "d", "e" and "f". The sentences included in the text to be processed are classified according to the above text categories through the K-Means model. For example, "A" includes "a", "B" includes "b", "c", and "C" includes "d", "e" and "f".

[0070] Step 303, extracting the central word of the text to be processed as the clustering center.

[0071] In an implementable example, the text to be processed is preliminarily divided through the TF-IDF keyword extraction technology, so that the machine autonomously learns the category of the text to be processed, and then extracts multiple central words of the text to be processed as the clustering center.

[0072] Step 304, performing label aggregation on the clustering center to obtain the label aggregation result.

[0073] Step 305, determining the first-level category of the text to be processed according to the text category clustering result and the label aggregation result.

[0074] The clustering results and label aggregation results obtained by clustering are respectively calculated for similarity with the first-level text categories obtained by training the XgBoost model. As shown in formula (4), the category with the highest similarity is taken as the first-level category of the current text.

[0075]

[0076] Among them, l t represents the label vector of the current text t, and c1 represents the first-level category vector. The similarity sim(l t , c1) between the two is obtained by calculating the overlap degree in the two vector spaces.

[0077] In the embodiments of the present application, by introducing a clustering model, soft classification is performed on the text to be processed for which the XgBoost model fails to accurately predict the first-level category. The K-Means model and the TF-IDF label extraction method are used to preliminarily divide the text to be processed to obtain the first-level category. On this basis, the text to be processed that has been preliminarily divided into the first-level category by the clustering model is secondarily classified by the dictionary matching method to obtain the second-level category. Using the clustering model can continue to perform the first-level classification on the text to be processed for which the XgBoost model fails to accurately predict the first-level category, realizing the multi-level classification of different texts to be processed, making the text analysis system more universal.

[0078] Figure 4 The flowchart of the text summary extraction unit provided by the embodiments of the present application. As Figure 4 shown, the text summary extraction unit 20 performs the following processing steps 401 to step 404:

[0079] Step 401: Determine the sentence vector corresponding to each sentence in the text to be processed according to the text vector, where the text to be processed includes at least two sentences.

[0080] In this solution, the word vector of the text to be processed is obtained through preprocessing, and the sentence vector corresponding to it in the text to be processed is determined through the multiple word vectors corresponding to each sentence. Specifically, the text to be processed includes at least two sentences.

[0081] Step 402: Determine the similarity between adjacent sentences in the text to be processed according to the sentence vectors of each sentence in the text to be processed.

[0082] According to the sentence vectors of each sentence in the text to be processed, calculate the similarity between adjacent sentences, and perform feature extraction on the text to be processed through the TextRank model, so as to obtain the weights of the corresponding sentences according to formula (5).

[0083]

[0084] Among them, WS(v i ) represents the weight of sentence i, which is calculated from the similarity W ji with the weight WS(v j ) of this sentence, where j represents the sentence adjacent to sentence j; d is the damping coefficient to avoid the situation that the current sentence has no adjacent sentences.

[0085] Step 403: Determine the importance score corresponding to each sentence in the text to be processed according to the similarity of adjacent sentences in the text to be processed.

[0086] According to the similarity values of adjacent sentences in the text to be processed, the importance corresponding to each sentence in the text to be processed is determined through iterative calculation, and the importance score of each sentence is obtained, thereby obtaining a sentence score list.

[0087] Step 404: Merge the K1 sentences with relatively large importance scores in the text to be processed according to the grammatical relationship to obtain a text summary, where K1 is a positive integer greater than or equal to 2.

[0088] For example, when K1 is 3, the first three sentences in the sentence score list are regarded as the sentences with relatively large importance scores, and these three sentences are merged to obtain a text summary.

[0089] In the embodiments of the present application, by introducing the TextRank model into the text summary extraction unit, the text problem is modeled as a graph model. The sentences in the text are used as basic semantic units and represented by nodes in the graph model, and the edges in the graph represent the similarity between sentences. Through iterative calculation, the importance of each sentence is obtained, and finally the important sentences are extracted as the summary output of the current text, so that the important sentences of the text to be processed can be extracted and information integrated, thereby obtaining a text summary.

[0090] Figure 5 The flowchart of the label element extraction unit provided by the embodiments of the present application. As Figure 5 shown, the label element extraction unit 30 performs the following processing steps 501 to 504:

[0091] Step 501: Input the text vector into a multi-classification model to match the text to be processed with predefined labels, and obtain the text to be processed after matching.

[0092] In an implementable example, there can be 5 predefined labels. For example, tax, sports, medicine, education, and others. After inputting the text vector into the multi-classification model, each sentence in the text to be processed is matched with these 5 labels, and the text to be processed that is matched with different labels is obtained respectively, thereby obtaining the text to be processed after matching.

[0093] Step 502: Input the text to be processed after matching into the LSTM model, perform probability prediction on the text to be processed after matching, and obtain labels.

[0094] Based on the above example, input the text to be processed after matching for 5 labels respectively into the LSTM model, perform probability prediction on the text to be processed after matching under each label. For example, the probability predicted for the text to be processed after matching under each label is P i , when P i is greater than the preset probability value, output the label corresponding to P i .

[0095] Step 503: Classify the text vectors through the dictionary matching algorithm to obtain index data.

[0096] Step 504: Perform syntactic dependency analysis on the labels and index data to obtain text index data information.

[0097] Classify the text vectors through the dictionary matching algorithm, and combine the syntactic dependency relationship of the labels and index data to correspond the labels and index data one by one to obtain index data for subsequent text analysis.

[0098] The present invention adopts a search algorithm based on dynamic local optimization, models the syntactic dependency analysis problem as a tree structure for processing, and each phrase containing independent semantic information in the text is used as a leaf node, so as to obtain a dependency tree containing n nodes, where there are n-1 dependency arcs, as shown in formula (6), and each dependency arc is represented as a quadruple, as shown in formula (7). First, input the text to be analyzed, scan the text multiple times, calculate the dependency probability between leaf nodes, as shown in formula (8), and only reduce the leaf nodes with larger dependency probabilities each time until reducing to the root node, i.e., the algorithm converges, and output the dependency analysis tree of the text to be analyzed.

[0099] T = {A1, A2, …, A n-1} #(6)

[0100] A ij = <Node i , Node j , Direction, Relation>, 1 ≤ i, j ≤ n - 1 #(7)

[0101] P(A ij ) = P(R, D\Tag i , Word i , Tag j , Word j , Dist) #(8)

[0102] Among them, Node iand Node j are two nodes of a dependency arc. Direction represents the direction of the dependency arc, which is divided into two types: left and right. Relation is the relationship label on the dependency arc, indicating the relationship type of the dependency arc. Dependency arc A ij 's probability P(A ij ) is uniquely determined by two nodes Node i and Node j .

[0103] In the embodiment of the present application, in the label element extraction unit, by introducing syntactic dependency relationship analysis, based on extracting the labels and index data in the text to be processed, the relationship between the labels and the index data is judged through syntactic dependency relationship analysis, so as to obtain a one-to-one correspondence between the two, and obtain the text index data information. It can help users identify the relationship between the labels and the index data from the text to be processed, and obtain the corresponding text index data information.

[0104] Figure 6 Flowchart of the intelligent text recommendation unit provided by the embodiment of the present application. As Figure 6 shown, the intelligent text recommendation unit 40 is used to execute the following steps 601 to step 607:

[0105] Step 601: Input the text vector into the label generation model to obtain text labels.

[0106] By inputting the text vector into the text summary extraction unit, the text summary is obtained;

[0107] By inputting the text vector into the label element extraction unit, the text index data information is obtained;

[0108] In this solution, either only the text vector can be used as the input of the intelligent text recommendation unit, or the text vector, the text summary and the text index data information can be used as the input of the intelligent text recommendation unit at the same time.

[0109] The label generation model can be a TextRank model, a TF-IDF model or a Word2Vec model. By inputting the text vector into the label generation model, text labels are automatically generated.

[0110] Step 602: Calculate the similarity of the texts corresponding to each text label through the Item-base collaborative filtering algorithm, and output the K2 sentences with larger similarity to obtain the first text recommendation list, where K2 is a positive integer greater than or equal to 2.

[0111] For example, if K2 is equal to 3, the similarity of the text corresponding to each text tag is calculated through the Item-base collaborative filtering algorithm, and the 3 sentences with larger similarity are output as the first text recommendation list, and the first text recommendation list includes 3 similar sentences.

[0112] Step 603: Extract features from the text vector to obtain the extracted features of the text vector.

[0113] When extracting features from the text vector, the number of words to be processed can be minimized without damaging the core information of the text, so as to reduce the dimensionality of the text vector space, thereby simplifying the calculation and improving the speed and efficiency of text processing.

[0114] Step 604: Perform dimensionality reduction processing on the extracted features through the SVD model to obtain the dimensionality reduction processing result.

[0115] Perform dimensionality reduction processing on the extracted features through the SVD model to remove the noise of the extracted features of the text vector, so as to improve the algorithm, and thus obtain the dimensionality reduction processing result.

[0116] Step 605: Calculate the cosine similarity of the text corresponding to the dimensionality reduction processing result, and output the K3 sentences with larger similarity to obtain the second text recommendation list, where K3 is a positive integer greater than or equal to 2.

[0117] For example, if K3 is equal to 4, calculate the cosine similarity of the text corresponding to the dimensionality reduction result, and output the 4 sentences with larger similarity as the second text recommendation list, and the second text recommendation list includes 4 similar sentences obtained.

[0118] Step 606: Merge the first text recommendation list and the second text recommendation list to obtain the merged recommendation list.

[0119] Merge the first text recommendation list and the second text recommendation list according to formula (9), as shown in formula (9):

[0120] rec list =rec1∪rec2#(9)

[0121] Among them, rec list is the final recommended text list, the collaborative filtering recommended text list is rec1, and the similarity calculation recommended text list is rec2.

[0122] Step 607: Sort the texts corresponding to the merged recommendation list according to time and similarity value, and output the K4 sentences with larger similarity to obtain the recommended text, where K4 is a positive integer greater than or equal to 2.

[0123] Based on the above example, the first text recommendation list has 3 similar sentences, assumed to be A1, A2, and A3 respectively, and the second text recommendation list has 4 similar sentences, assumed to be B1, B2, B3, and B4 respectively. The combined recommendation list is A1, A2, A3, B1, B2, B3, and B4. Sort the corresponding texts in the combined recommendation list according to time and similarity, as shown in formula (10). Assume K4 = 5, and the finally obtained recommended texts are assumed to be A1, A2, B2, B4, A3.

[0124]

[0125] where l i is the i-th text in the combined recommendation list. When the text contents are the same, the current recommended text can be directly taken from the text in the combined recommendation list; when the text contents in the combined recommendation list are different, the recommendation score s is used as the main sorting basis. If the recommendation scores are the same, the text updated at time t is taken as the i-th recommended text.

[0126] In the embodiment of the present application, the intelligent text recommendation unit calculates the similarity between the corpus text and the text to be processed by using the Item-base collaborative filtering and text similarity calculation methods for the text to be processed according to the text vector, combines and sorts the texts in the first text recommendation list and the second text recommendation list according to time and similarity values, and takes the K4 sentences with higher similarity texts as the recommended texts, thereby improving the recommendation accuracy of the intelligent text recommendation unit.

[0127] Figure 7 The flowchart of the sentiment intention analysis unit provided by the embodiment of the present application. As Figure 7 shown, the sentiment intention analysis unit is used to perform the following steps 701 to 703:

[0128] Step 701: Input the text vector into the LSTM model, and obtain the first text feature by extracting the sentiment information in the text to be processed.

[0129] Text sentiment intention analysis is a common application in natural language processing methods, which can refine the sentiment and intention content in the text, and then obtain the text with emotional color in the text through analysis, processing, induction, and reasoning. When the text vector is input into the LSTM model, the sentiment information in the text to be processed includes positive information, negative information, or neutral information, etc. By extracting the sentiment information of all texts to be processed, the first text feature is finally obtained.

[0130] Step 702: Perform sentiment category calculation on the first text feature through a fully connected network to obtain the sentiment tendency of the text.

[0131] The sentiment category of the text sentiment information is calculated through a fully connected network. For example, the text sentiment information can be divided into three categories: positive, negative, and neutral, and finally the sentiment tendency of the text is judged.

[0132] Step 703: Input the text vector into the TextCNN model, and use convolution calculation to extract the text features of the text to be processed, obtaining the second text feature.

[0133] Step 704: Input the second text feature into the fully connected network to obtain the intent category of the text.

[0134] In the embodiment of the present application, the sentiment intent analysis unit respectively screens the first text feature and the second text feature from the perspectives of time and space by introducing the LSTM model and convolution processing, and then analyzes the sentiment tendency and intent category of the text contained, helping the text analysis system better understand the user's needs.

[0135] Text analysis method

[0136] Figure 8 is a flowchart of a text analysis method provided by an embodiment of the application. As Figure 8 shown, the text analysis method includes:

[0137] Step 801: Obtain a text vector by successively performing Chinese word segmentation, stop word removal, and text vectorization processing on the text to be processed.

[0138] Step 802: According to the text vector, input the text to be processed into at least one text analysis unit for processing to obtain an analysis result. Among them, the text analysis includes a multi-level text classification unit, a text summary extraction unit, a label element extraction unit, an intelligent text recommendation unit, and a sentiment intent analysis unit. The multi-level text classification unit performs: classifying the text to be processed according to the text vector. The text summary extraction unit performs: integrating the information of the text to be processed according to the text vector to obtain a text summary. The label element extraction unit extracts labels from the text to be processed and collects index data according to the text vector, and obtains text index data information according to the labels and the index data. The intelligent text recommendation unit performs: determining a recommended text from the text to be processed according to the text vector. The sentiment intent analysis unit performs: determining the sentiment tendency and intent category of the text to be processed according to the text vector.

[0139] In the embodiments of the present application, through the text analysis method of the present solution, while providing basic text preprocessing functions such as Chinese word segmentation, stop word removal, and text vectorization for the text to be processed, at least one of multi-level text classification, text summary extraction, label element extraction, intelligent text recommendation, and sentiment intention analysis is performed on the text to be processed according to the text vector, and it can be used in different scenarios such as resource integration systems, file management systems, and information portals.

[0140] It should be noted that for the information interaction, execution process, etc. between the steps in the above text analysis method, since it is based on the same concept as the foregoing text analysis system embodiment, the specific content can be referred to the description in the foregoing text analysis system embodiment and will not be elaborated here.

[0141] Electronic device

[0142] Refer to [[ID=2,8]]Figure 9 , which shows a schematic structural diagram of an electronic device according to an embodiment of the present application. The specific implementation of the electronic device is not limited in the specific embodiments of the present application.

[0143] As Figure 9 shown, the electronic device 900 may include: a processor 902, a communications interface 904, a memory 906, and a communication bus 908.

[0144] Wherein:

[0145] The processor 902, the communications interface 904, and the memory 906 communicate with each other through the communication bus 908.

[0146] The communications interface 904 is used to communicate with other electronic devices or servers.

[0147] The processor 902 is used to execute the program 910, and specifically can execute the relevant steps in the above text analysis method embodiment.

[0148] Specifically, the program 910 may include program code, and the program code includes computer operation instructions.

[0149] The processor 902 may be a central processing unit CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. One or more processors included in the intelligent device may be of the same type of processor, such as one or more CPUs; or may be of different types of processors, such as one or more CPUs and one or more ASICs.

[0150] A memory 906 for storing a program 910. The memory 906 may include a high-speed RAM memory and may also include a non-volatile memory, such as at least one disk memory.

[0151] The program 910 can be specifically used to cause the processor 802 to execute the text analysis method in any of the foregoing embodiments.

[0152] For the specific implementation of each step in the program 910, reference can be made to the corresponding steps and descriptions in the corresponding units in any of the foregoing text analysis method embodiments, which will not be elaborated here. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the devices and modules described above can refer to the corresponding process descriptions in the foregoing method embodiments, which will not be repeated here.

[0153] Through the electronic device of this embodiment, first, the text to be processed is preprocessed. After Chinese word segmentation, stop word removal, and text vectorization processing, a text vector is obtained. Secondly, according to the different application scenarios of the text, the obtained preprocessing result, that is, the text vector, can be input into different text analysis modules. The multi-level text classification unit classifies the text to be processed according to the text vector, the text summary extraction unit integrates the information of the text to be processed according to the text vector to obtain a text summary, the label element extraction unit extracts labels and collects index data for the text to be processed according to the text vector, obtains text index data information according to the labels and index data, the intelligent text recommendation unit determines the recommended text from the text to be processed according to the text vector, and the sentiment intention analysis unit can determine the sentiment tendency and intention category of the text to be processed according to the text vector. This solution preprocesses the text to be processed, and then inputs the text vector obtained after preprocessing into each text processing unit included in the text analysis module, so that the user only needs to input the text to be processed into the system of this solution to realize the automatic processing of the text to be processed and obtain text analysis results that meet different application scenarios.

[0154] Computer storage medium

[0155] An embodiment of the present application also provides a computer-readable storage medium storing instructions for causing a machine to execute the text analysis method as described herein. Specifically, a system or device equipped with a storage medium can be provided. Software program codes for implementing the functions of any of the foregoing embodiments are stored on the storage medium, and the computer (or CPU or MPU) of the system or device is caused to read and execute the program codes stored on the storage medium.

[0156] In this case, the program code read from the storage medium itself can implement the functions of any one of the above embodiments, so the program code and the storage medium storing the program code constitute a part of this application.

[0157] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Optionally, the program code can be downloaded from a server computer via a communication network.

[0158] Furthermore, it should be clear that not only can the functions of any one of the above embodiments be realized by executing the program code read by the computer, but also by causing an operating system or the like operating on the computer to complete some or all of the actual operations based on the instructions of the program code.

[0159] In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion module connected to the computer, and then based on the instructions of the program code, the CPU or the like installed on the expansion board or the expansion module is caused to execute some and all of the actual operations, so as to realize the functions of any one of the above embodiments.

[0160] Computer program product

[0161] An embodiment of this application also provides a computer program product, including computer instructions, where the computer instructions direct a computing device to perform the operations corresponding to any one of the above method embodiments.

[0162] It should be noted that, according to the needs of implementation, each component / step described in the embodiments of this application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0163] The method according to the embodiments of the present application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code originally stored in a remote recording medium or a non-transitory machine-readable medium and downloaded via a network and to be stored in a local recording medium, so that the method described herein can be stored as such software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component (such as RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the image annotation method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the image annotation method shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for executing the check code generation method shown herein.

[0164] Those of ordinary skill in the art can realize that the units and method steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present application.

[0165] The above embodiments are only used to illustrate the embodiments of the present application, rather than to limit the embodiments of the present application. Those of ordinary skill in the relevant technical field can also make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present application, and the patent protection scope of the embodiments of the present application should be defined by the claims.

Claims

1. A text analysis system (100), comprising: A preprocessing module (110) and a text analysis module (120); The text analysis module (110) includes a multi-level text classification unit (10), a text summary extraction unit (20), a label element extraction unit (30), an intelligent text recommendation unit (40), and an emotional intention analysis unit (50); the text analysis module (110) is used to perform text analysis using the multi-level text classification unit (10), the text summary extraction unit (20), the label element extraction unit (30), the intelligent text recommendation unit (40), and the emotional intention analysis unit (50); The preprocessing module (120) is used to perform Chinese word segmentation, stop word removal, and text vectorization processing on the text to be processed in sequence to obtain a text vector; The multi-level text classification unit (10) is used to input the text vector into a classification model to obtain the first-level category probability of the text to be processed; if the first-level category probability is greater than a preset probability threshold, the first-level category of the text to be processed is determined according to the first-level category probability, and the text to be processed is secondarily classified by a dictionary matching algorithm to determine the second-level category of the text to be processed; If the first-level category probability is less than or equal to the probability threshold, the first-level category of the text to be processed is determined by a clustering model, and the text to be processed is secondarily classified by the dictionary matching algorithm to determine the second-level category of the text to be processed; The text summary extraction unit (20) is used to determine the sentence vector corresponding to each sentence in the text to be processed according to the text vector, where the text to be processed includes at least two sentences; determine the similarity between adjacent sentences in the text to be processed according to the sentence vectors of each sentence in the text to be processed; determine the importance score corresponding to each sentence in the text to be processed by iterative calculation according to the similarity between adjacent sentences in the text to be processed to obtain a sentence score list; merge the first K1 sentences sorted in descending order of importance score in the sentence score list according to the syntactic relationship to obtain a text summary, where K1 is a positive integer greater than or equal to 2; The label element extraction unit (30) is used to input the text vector into a multi-classification model to match the text to be processed with predefined labels to obtain the text to be processed after matching; input the text to be processed after matching into an LSTM model to perform probability prediction on the text to be processed after matching to obtain a label; classify the text vector by a dictionary matching algorithm to obtain index data; based on a search algorithm of dynamic local optimization, model the syntactic dependency analysis between the label and the index data as a tree structure for analysis and processing, and judge the corresponding relationship between the label and the index data to obtain text index data information; The intelligent text recommendation unit (40) is configured to input the text vector into a tag generation model to obtain text tags; calculate the similarity of the texts corresponding to the text tags through the Item-base collaborative filtering algorithm, and output the top K2 sentences sorted from large to small based on the similarity to obtain a first text recommendation list, where K2 is a positive integer greater than or equal to 2; extract features from the text vector to obtain the extracted features of the text vector; perform dimensionality reduction processing on the extracted features through an SVD model to obtain a dimensionality reduction result; calculate the cosine similarity of the text corresponding to the dimensionality reduction result, and output the top K3 sentences sorted from large to small based on the similarity to obtain a second text recommendation list, where K3 is a positive integer greater than or equal to 2; merge the first text recommendation list and the second text recommendation list to obtain a merged recommendation list; sort the texts corresponding to the merged recommendation list according to time and similarity values, and output the top K4 sentences sorted from large to small based on the similarity to obtain recommended texts, where K4 is a positive integer greater than or equal to 2; The sentiment intention analysis unit (50) is configured to input the text vector into an LSTM model, and obtain first text features by extracting sentiment information from the text to be processed; calculate the sentiment category of the text through a fully connected network for the first text features to obtain the sentiment tendency of the text; input the text vector into a TextCNN model, and extract the text features of the text to be processed by using convolutional calculation to obtain second text features; input the second text features into the fully connected network to obtain the intention category of the text.

2. The system according to claim 1, wherein, The multi-level text classification unit (10) is configured to perform the following processing through the clustering model: Calculate the text category of the text to be processed through the elbow method; Cluster the text categories of the text to be processed through a K-Means model to obtain a text category clustering result; Extract the central words of the text to be processed as the clustering centers; Perform label aggregation on the clustering centers to obtain a label aggregation result; Determine the first-level category of the text to be processed according to the text category clustering result and the label aggregation result.

3. A text analysis method (800), the method comprising: Obtain a text vector by successively performing Chinese word segmentation, stop word removal, and text vectorization processing on the text to be processed; According to the text vector, input the text to be processed into a text analysis module for processing to obtain an analysis result, where the text analysis module includes a multi-level text classification unit, a text summary extraction unit, a tag element extraction unit, an intelligent text recommendation unit, and a sentiment intention analysis unit; The multi-level text classification unit performs the following operations: input the text vector into a classification model to obtain the probability of the first-level category of the text to be processed; if the probability of the first-level category is greater than a preset probability threshold, determine the first-level category of the text to be processed according to the probability of the first-level category, and perform secondary classification on the text to be processed through a dictionary matching algorithm to determine the second-level category of the text to be processed; if the probability of the first-level category is less than or equal to the probability threshold, determine the first-level category of the text to be processed through a clustering model, and perform secondary classification on the text to be processed through the dictionary matching algorithm to determine the second-level category of the text to be processed; The text summary extraction unit performs the following operations: determine the sentence vector corresponding to each sentence in the text to be processed according to the text vector, where the text to be processed includes at least two sentences; determine the similarity between adjacent sentences in the text to be processed according to the sentence vectors of each sentence in the text to be processed; determine the importance score corresponding to each sentence in the text to be processed through iterative calculation according to the similarity between adjacent sentences in the text to be processed, and obtain a sentence score list; merge the first K1 sentences sorted by importance score from large to small in the sentence score list according to the syntactic relationship to obtain a text summary, where K1 is a positive integer greater than or equal to 2; The label element extraction unit inputs the text vector into a multi-classification model to match the text to be processed with predefined labels to obtain the text to be processed after matching; inputs the text to be processed after matching into an LSTM model to perform probability prediction on the text to be processed after matching to obtain a label; classifies the text vector through a dictionary matching algorithm to obtain index data; based on a search algorithm of dynamic local optimization, models the syntactic dependency analysis between the label and the index data as a tree structure for analysis and processing, determines the corresponding relationship between the label and the index data, and obtains text index data information; The intelligent text recommendation unit performs the following operations: input the text vector into a tag generation model to obtain text tags; calculate the similarity of the texts corresponding to each of the text tags through the Item-base collaborative filtering algorithm, and output the top K2 sentences sorted from largest to smallest based on the similarity to obtain a first text recommendation list, where K2 is a positive integer greater than or equal to 2; extract features from the text vector to obtain the extracted features of the text vector; perform dimensionality reduction processing on the extracted features through an SVD model to obtain a dimensionality reduction processing result; calculate the cosine similarity of the texts corresponding to the dimensionality reduction processing result, and output the top K3 sentences sorted from largest to smallest based on the similarity to obtain a second text recommendation list, where K3 is a positive integer greater than or equal to 2; merge the first text recommendation list and the second text recommendation list to obtain a merged recommendation list; sort the texts corresponding to the merged recommendation list according to time and similarity values, and output the top K4 sentences sorted from largest to smallest based on the similarity to obtain recommended texts, where K4 is a positive integer greater than or equal to 2; The sentiment intention analysis unit performs the following operations: input the text vector into an LSTM model, and obtain first text features by extracting the sentiment information in the text to be processed; calculate the sentiment category of the text through a fully connected network for the first text features to obtain the sentiment tendency of the text; input the text vector into a TextCNN model, and extract the text features of the text to be processed by using convolutional calculation to obtain second text features; input the second text features into the fully connected network to obtain the intention category of the text.

4. An electronic device (900) includes a processor (902), a communication interface (904), a memory (906), and a communication bus (908). The processor (902), the memory (906), and the communication interface (904) complete communication with each other through the communication bus (908); The memory (906) is used to store at least one executable instruction, and the executable instruction causes the processor (902) to perform the operations corresponding to the method described in claim 3.

5. A computer storage medium stores a computer program thereon, and when the program is executed by a processor, it implements the method described in claim 3.

Images