Text data visualization analysis method, device, equipment and storage medium

Abstract

The present application relates to artificial intelligence technology, and discloses a method, device, equipment and storage medium for visual analysis of text data, including obtaining a plurality of text data to be processed; extracting keywords from a plurality of text data to be processed to obtain keywords and weights corresponding to the text data to be processed; using visualization tools to visualize the plurality of text data to be processed and the corresponding keywords and weights to obtain a scatter plot; clustering the scatter plot using a DBscan clustering model to obtain cluster clusters; based on each cluster cluster, extracting the corresponding keywords and text data to be processed to obtain cluster keywords and cluster text data; matching the cluster keywords through a matching model, and finally connecting the cluster text data to construct a knowledge graph. The present application also relates to blockchain technology, and the knowledge graph is stored in the blockchain. The present application can improve processing efficiency while removing noise data.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a text data visualization analysis method, device, equipment and storage medium. Background Art

[0002] In the context of today's information age, there are many channels for obtaining information, such as hearsay, online collection, field investigation, etc., but the amount of information obtained is also very huge. However, it becomes very difficult and time-consuming to summarize and organize useful information from a large amount of information. In the existing data screening and analysis technology, there are manual data entry into Excel documents and then manual analysis; there are also models for extracting keywords from text data for analysis, but for the above solutions, there are problems such as low efficiency of screening and analysis, and the inability to remove noise data in text data. Therefore, how to remove noise data in the text data under the premise of improving the efficiency of screening and analysis has become an urgent problem to be solved. Summary of the invention

[0003] The present application provides a text data visualization analysis method, device, equipment and storage medium to solve the problems in the prior art that the screening analysis efficiency is low and the noise data in the text data cannot be removed.

[0004] To solve the above problems, this application provides a text data visualization analysis method, including:

[0005] Get multiple text data to be processed;

[0006] Perform keyword extraction processing on the plurality of text data to be processed to obtain keywords and their weights corresponding to the text data to be processed;

[0007] Using a visualization tool to perform visualization according to the plurality of text data to be processed and the corresponding keywords and their weights, a scatter plot is obtained;

[0008] According to the scatter plot, cluster analysis is performed using a DBscan clustering model to obtain clusters;

[0009] Based on each of the clusters, extract the corresponding keywords and the text data to be processed to obtain cluster keywords and cluster text data;

[0010] The clustered text data are taken as nodes and the clustered keywords are taken as attributes. The clustered keywords corresponding to different clustered text data are matched through a pre-trained matching model. If the clustered keywords are matched successfully, the corresponding clustered text data are connected to construct a knowledge graph.

[0011] Furthermore, the obtaining of a plurality of text data to be processed includes:

[0012] Send a call request to the database, the call request carrying a signature verification token;

[0013] The signature verification result returned by the database is received, and when the signature verification result is passed, the text data in the database is called, so as to obtain the multiple text data to be processed.

[0014] Furthermore, the keyword extraction process of the plurality of text data to be processed includes:

[0015] Using the Jieba word segmentation to perform word segmentation processing on the text data to be processed and to remove stop words, to obtain a corresponding plurality of words;

[0016] The TF-IDF algorithm is used to extract keywords from the multiple words, and the weights corresponding to the keywords are obtained.

[0017] Furthermore, the using the Jieba word segmentation to perform word segmentation processing on the text data to be processed includes:

[0018] Scan the text data based on a preset Trie tree to identify multiple segmentation combinations of words in the text data;

[0019] Constructing a directed acyclic graph based on all identified segmentation combinations, using the directed acyclic graph to perform dynamic programming to find the maximum probability path, determining the segmentation combination with the maximum probability, and segmenting the sentence to be processed based on the segmentation combination with the maximum probability;

[0020] For unrecognized words, a hidden Markov model is used for segmentation.

[0021] Furthermore, the using of the TF-IDF algorithm to extract keywords from the multiple words and obtaining the corresponding weights includes:

[0022] Obtain the number of times a first word appears in each document in the data to be processed and the total number of words, and obtain the word frequency of the first word by dividing the number of times the first word appears by the total number of words;

[0023] Obtaining the total number of documents in the data to be processed and the number of documents in which the first word appears, and calculating the inverse document frequency of the first word according to the total number of documents and the number of documents in which the first word appears;

[0024] Multiply the word frequency corresponding to the first word by the inverse document frequency to obtain the weight of the first word;

[0025] The first word whose weight is greater than a preset value is extracted as the keyword, and the keyword and its corresponding weight are obtained.

[0026] Furthermore, after extracting keywords from the multiple words using the TF-IDF algorithm and obtaining the corresponding weights, the method further includes:

[0027] Sorting the keywords according to the weights corresponding to the keywords;

[0028] Based on the ranking results, a preset number of keywords are extracted.

[0029] Furthermore, after performing cluster analysis using the DBscan clustering model to obtain clusters, the method further includes:

[0030] Using the KMeans model, each cluster is processed to obtain the center point data;

[0031] Based on the center point data, the corresponding keywords and their weights are output.

[0032] In order to solve the above problems, the present application also provides a text data visualization analysis device, the device comprising:

[0033] An acquisition module, used for acquiring a plurality of text data to be processed;

[0034] A quantification module, used for performing keyword extraction processing on the plurality of text data to be processed, and obtaining keywords and their weights corresponding to the text data to be processed;

[0035] A visualization module, used for performing visualization display using a visualization tool according to the plurality of text data to be processed and the corresponding keywords and their weights to obtain a scatter plot;

[0036] A clustering module, used to perform cluster analysis based on the scatter plot using a DBscan clustering model to obtain clusters;

[0037] An extraction module, used for extracting the corresponding keywords and text data to be processed based on each of the clusters to obtain cluster keywords and cluster text data;

[0038] A construction module is used to use the clustered text data as nodes and the clustered keywords as attributes, and to match the clustered keywords corresponding to different clustered text data through a pre-trained matching model. If the clustered keywords are matched successfully, the corresponding clustered text data are connected to construct a knowledge graph.

[0039] In order to solve the above problem, the present application also provides a computer device, including:

[0040] at least one processor; and,

[0041] a memory communicatively connected to the at least one processor; wherein,

[0042] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the text data visualization analysis method as described above.

[0043] In order to solve the above problems, the present application also provides a non-volatile computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the text data visualization analysis method as described above is implemented.

[0044] The text data visualization analysis method, device, equipment and storage medium provided by the embodiments of the present application have at least the following beneficial effects compared with the prior art:

[0045] By acquiring a plurality of text data to be processed and performing keyword extraction processing on the text data to be processed, keywords corresponding to the text data to be processed and their weights are obtained, that is, the attributes of the text data are obtained; according to the text data to be processed and the corresponding keywords and their weights, visualization tools are used for visualization display to obtain a scatter plot, each point in the scatter plot corresponds to the text data one by one, and visualization is performed, and then cluster analysis is further performed using the DBscan clustering model to obtain cluster clusters, remove text data with large differences, that is, remove noise data, and display in real time, extract the keywords and the text data to be processed in the cluster clusters, obtain cluster keywords and cluster keywords, and implement data screening, and finally, according to the cluster text data as nodes and the cluster keywords as attributes, the cluster keywords corresponding to different cluster text data are matched through a pre-trained matching model, and if the cluster keywords are successfully matched, the corresponding cluster text data are connected to form a knowledge graph, so that the correlation of each node of the knowledge graph obtained in the end is high, and the efficiency of data analysis is improved, and the knowledge graph obtained in the end can enable users to grasp information more efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the scheme in the present application, a brief introduction will be given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1A flowchart of a text data visualization analysis method provided in one embodiment of the present application;

[0048] Figure 2 A directed acyclic graph provided in one embodiment of the present application;

[0049] Figure 3 A schematic diagram of a module of a text data visualization analysis device provided in an embodiment of the present application;

[0050] Figure 4 A schematic diagram of the structure of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0052] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly or implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0053] This application provides a method for visual analysis of text data. Figure 1 The figure is a flow chart of a text data visualization analysis method provided in one embodiment of the present application.

[0054] In this embodiment, the text data visualization analysis method includes:

[0055] S1. Obtain multiple text data to be processed;

[0056] In the present application, the text data to be processed can be directly received from the user or extracted from the database. The text data to be processed includes but is not limited to the daily text data generated by the user, and the common attributes of the multiple users are obtained by analyzing the daily text data generated by multiple users, so as to facilitate the analysis by relevant personnel.

[0057] Furthermore, the obtaining of a plurality of text data to be processed includes:

[0058] Send a call request to the database, the call request carrying a signature verification token;

[0059] The signature verification result returned by the database is received, and when the signature verification result is passed, the text data in the database is called, so as to obtain the multiple text data to be processed.

[0060] Specifically, since multiple text data to be processed may involve private data of users or companies, all text data to be processed will be saved in a preset database. Therefore, when obtaining the text data to be processed, the database will perform a signature verification step to ensure data security and avoid data leakage.

[0061] By verifying the signature, the security of the content in the database can be guaranteed, and the leakage of user privacy, such as text data, can be avoided.

[0062] S2, performing keyword extraction processing on the plurality of text data to be processed to obtain keywords and their weights corresponding to the text data to be processed;

[0063] Specifically, by performing keyword extraction processing on the text data to be processed, keywords corresponding to each text data to be processed and their weights are obtained, thereby realizing the extraction of attributes of the text data to be processed.

[0064] Furthermore, the keyword extraction process of the plurality of text data to be processed includes:

[0065] Using the Jieba word segmentation to perform word segmentation processing on the text data to be processed and to remove stop words, to obtain a corresponding plurality of words;

[0066] The TF-IDF algorithm is used to extract keywords from the multiple words, and the weights corresponding to the keywords are obtained.

[0067] Specifically, the Jieba word segmentation supports three word segmentation modes: precise mode, which separates the sentence most accurately; full mode, which scans out all the words that can be formed into words in the sentence; search engine mode, which segments long words again based on the precise mode. The Jieba word segmentation in precise mode is used in this application to facilitate accurate separation of the sentences to be processed.

[0068] The TF-IDF algorithm is a statistical method used to evaluate the importance of a word to a document in a corpus. The importance of a word increases with the number of times it appears in the document, but it also decreases with the number of times it appears in other documents in the corpus.

[0069] The stuttering toolkit in Python is directly used in this application. By importing this stuttering toolkit, each input text data to be processed can be processed and segmented to achieve word segmentation processing of the text data to be processed.

[0070] For example, after segmenting "the process after the investigation is damage assessment" using the precise pattern of Jieba word segmentation, the resulting words are "investigation / after / the / process / is / damage assessment".

[0071] After segmentation, the segmented words are segmented by Jieba and the existing stop word library is used to remove stop words. Specifically, the segmented words are searched in the stop word library one by one. If the words are found, the corresponding words are removed, thereby achieving the removal of stop words.

[0072] By using Jieba word segmentation to split text data into multiple independent words, it is convenient for subsequent keyword extraction, and keyword extraction is performed on the words obtained after each text to be processed is segmented, and the corresponding weights are obtained to achieve the extraction of attributes of the text to be processed.

[0073] Furthermore, the using the Jieba word segmentation to perform word segmentation processing on the text data to be processed includes:

[0074] Scan the text data based on a preset Trie tree to identify multiple segmentation combinations of words in the text data;

[0075] Constructing a directed acyclic graph based on all identified segmentation combinations, using the directed acyclic graph to perform dynamic programming to find the maximum probability path, determining the segmentation combination with the maximum probability, and segmenting the sentence to be processed based on the segmentation combination with the maximum probability;

[0076] For unrecognized words, a hidden Markov model is used for segmentation.

[0077] Specifically, the Trie tree, also known as a dictionary tree, is a common data structure and a prefix tree used for fast string matching in a string list. The sentence to be processed is scanned based on a preset Trie tree, and multiple segmentation combinations of words in the sentence to be processed are identified. The sentence to be processed is scanned and matched with the Trie tree to generate multiple segmentation combinations of words. The multiple segmentation combinations form a directed acyclic graph, and each node in the directed acyclic graph is a segmented word.

[0078] like Figure 2As shown, a directed acyclic graph is shown, which divides the sentence "Go to Peking University to play" into five segmentation combinations: go to / Beijing / University / Play, go to / Beijing / University / Play, go to / Beijing / University / Play, go to / Beijing / University / Play, go to / Beijing / University / Play, go to / Beijing / University / Play, and go to / Beijing / University / Play. These five segmentation combinations constitute a directed acyclic graph. The arc arrow represents the combination of the words between the starting point and the end point of the arc arrow. The horizontal arrow can represent the segmentation of the words.

[0079] Then, the directed acyclic graph is used to perform dynamic programming to find the maximum probability path. When the dictionary is used to generate the Trie tree, the number of times each word appears is converted into frequency. For the given multiple segmentation combinations, the frequency of occurrence of each segmentation combination is searched for each segmentation combination, that is, the probability of each node in the directed acyclic graph. The main function for calculating the maximum probability path is calc, and the function calculates the maximum probability path according to the constructed directed acyclic graph. The function calc is a bottom-up dynamic programming, which starts from the last word of the sentence to be processed and traverses each word of the sentence to be processed in reverse order, and calculates the probability logarithm score of each segmentation combination of the sentence to be processed. Then, the situation with the highest probability logarithm score is saved and output in such a segmentation combination mode. That is, the segmentation combination with the highest probability is obtained, and the sentence to be processed is segmented based on the segmentation combination.

[0080] And due to the limitation of the dictionary, it cannot contain all words, so the words that have not appeared in the dictionary will be segmented by the hidden Markov model. The hidden Markov model marks Chinese words according to the four states of BEMS. B refers to the starting position, E refers to the ending position, M refers to the middle position, and S refers to the position of a separate word. The stuttering analysis uses the above four states to mark Chinese words. For example, Beijing can be marked as BE, that is, Bei / B Jing / E, that is, Bei is the starting position and Jing is the ending position, for splitting. By segmenting the sentences to be processed in the above way, the segmentation combination closest to the actual situation can be obtained.

[0081] By segmenting the statement to be processed in the above manner, a segmentation combination closest to the actual situation can be obtained.

[0082] Furthermore, the using of the TF-IDF algorithm to extract keywords from the multiple words and obtaining the corresponding weights includes:

[0083] Obtain the number of times a first word appears in each document in the data to be processed and the total number of words, and obtain the word frequency of the first word by dividing the number of times the first word appears by the total number of words;

[0084] Obtaining the total number of documents in the data to be processed and the number of documents in which the first word appears, and calculating the inverse document frequency of the first word according to the total number of documents and the number of documents in which the first word appears;

[0085] Multiply the word frequency corresponding to the first word by the inverse document frequency to obtain the weight of the first word;

[0086] The first word whose weight is greater than a preset value is extracted as the keyword, and the keyword and its corresponding weight are obtained.

[0087] Specifically, a weight corresponding to the first word is obtained according to the word frequency and the inverse document frequency of the first word, wherein the inverse document frequency is the total number of documents in the data to be processed divided by (the number of documents in which the first word appears + 1), to obtain a first value, and then the logarithm of the first value is taken to obtain the weight corresponding to the first word;

[0088] By comparing the weight with the preset value, a weight greater than the preset value and its corresponding first word are obtained as the keyword.

[0089] Furthermore, after extracting keywords from the multiple words using the TF-IDF algorithm and obtaining the corresponding weights, the method further includes:

[0090] Sorting the keywords according to the weights corresponding to the keywords;

[0091] Based on the ranking results, a preset number of keywords are extracted.

[0092] Specifically, since multiple keywords will be obtained through the TF-IDF algorithm, and the number of keywords obtained by different text data to be processed may be inconsistent, the keywords corresponding to each text data to be processed are sorted in descending order, and the preset number of keywords are extracted to achieve the consistency of the number of keywords in each text data to be processed, that is, the consistency of the attribute dimensions.

[0093] In the present application, the preset number may be 4.

[0094] By unifying the attribute dimensions of each text data, it is easier to use visualization tools for subsequent visualization.

[0095] S3, using a visualization tool to perform visualization display according to the plurality of text data to be processed and the corresponding keywords and their weights, to obtain a scatter plot;

[0096] By using existing visualization tools, such as echarts, datav and other visualization tools, the text data to be processed is visualized to obtain the corresponding scatter plot.

[0097] The text data to be processed is represented as a point, and its corresponding keywords and their weights are used as its attributes. The visualization tools such as echarts and datav visualize the text data to be processed according to the attributes.

[0098] S4. Perform cluster analysis using a DBscan clustering model based on the scatter plot to obtain clusters;

[0099] After obtaining the scatter plot, the DBscan clustering model is used for clustering to obtain clusters. The DBscan is an unsupervised learning algorithm that can find the clustering attributes in the scatter plot, remove discrete points, and avoid interference from noise data.

[0100] The DBscan clustering model is a density-based clustering algorithm. This type of density-based clustering algorithm generally assumes that the category can be determined by the compactness of the sample distribution. Samples of the same category are closely connected. By classifying closely connected samples into one category, a cluster category is obtained. DBSCAN describes the compactness of the sample set based on a set of neighborhoods, and the parameter (∈, MinPts) is used to describe the compactness of the sample distribution in the neighborhood. Among them, ∈ describes the neighborhood distance threshold of a certain sample, and MinPts describes the threshold of the number of samples in the neighborhood of a certain sample with a distance of ∈.

[0101] Furthermore, after performing cluster analysis using the DBscan clustering model to obtain clusters, the method further includes:

[0102] Using the KMeans model, each cluster is processed to obtain the center point data;

[0103] Based on the center point data, the corresponding keywords and their weights are output.

[0104] Specifically, the KMeans model is used to process each cluster in the scatter plot, specifically taking K=1, to obtain the center point data of each cluster, and based on the center point data, the text data corresponding to the center point and its corresponding keywords are output. The center point data of each cluster can fully indicate the characteristics of the cluster, so the keywords corresponding to the center point data are extracted as the characteristic attributes of the cluster.

[0105] KMeans clustering algorithm, also known as k-means clustering algorithm, is a distance-based clustering algorithm that combines simplicity and classicity. It uses distance as the evaluation index of similarity, that is, it believes that the closer the distance between two objects, the greater their similarity. The algorithm believes that clusters are composed of objects that are close to each other, so the ultimate goal is to obtain compact and independent clusters.

[0106] By using the KMeans model to obtain the center point data of each cluster, the text data and keywords corresponding to the center point data are obtained. The keywords corresponding to the center point data can be used as the characteristic attributes of the cluster, thereby improving the clarity of the characteristics of the cluster data.

[0107] S5, based on each of the clusters, extracting the corresponding keywords and the text data to be processed to obtain cluster keywords and cluster text data;

[0108] After obtaining each cluster in the scatter plot, the text data to be processed and the keywords corresponding to each point in the cluster are extracted. The points correspond to the text data to be processed and the keywords. By obtaining each point in the scatter plot, the corresponding text data and keywords can be obtained, thereby finally obtaining cluster keywords and cluster text data.

[0109] By extracting only the data of the clusters and removing the discrete data outside the clusters, the correlation between the nodes of the subsequent knowledge graph is stronger. And removing discrete data is to remove noise data, so as to avoid the adverse effects of noise data on subsequent processes. And improve the processing efficiency of subsequent processing.

[0110] S6. Taking the clustered text data as nodes and the clustered keywords as attributes, the clustered keywords corresponding to different clustered text data are matched through a pre-trained matching model. If the clustered keywords are matched successfully, the corresponding clustered text data are connected to construct a knowledge graph.

[0111] Specifically, each clustered text data is taken as a node, which is directly represented by a point in practice, and each point is distinguished, and its corresponding cluster keyword is used as an attribute, and whether the clustered text data are connected to each other is determined according to the cluster keyword.

[0112] By using the pre-trained matching model, the similarity between the cluster keywords corresponding to the two cluster text data is obtained, and the similarity is compared with the preset value. If the similarity is greater than the preset value, the two cluster text data are connected in series. If the similarity between multiple cluster keywords in the two cluster text data is greater than the preset value, the greater the number, the closer the distance between the two cluster text data in the entire knowledge graph; conversely, if the similarity is less than the preset value, the two cluster text data are not connected. When the cluster keywords corresponding to all cluster text data are calculated in pairs, a knowledge graph is formed after all calculations are completed. Constructing a knowledge graph can more clearly display data and the relationship between data, allowing users to grasp information more efficiently.

[0113] The matching model is obtained by training with the CNN-DSSM model, wherein the CNN-DSSM model is a semantic matching model.

[0114] It should be emphasized that in order to further ensure the privacy and security of the data, all data of the knowledge graph can also be stored in a node of a blockchain.

[0115] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0116] By acquiring a plurality of text data to be processed and performing keyword extraction processing on the text data to be processed, keywords corresponding to the text data to be processed and their weights are obtained, that is, the attributes of the text data are obtained; according to the text data to be processed and the corresponding keywords and their weights, visualization tools are used for visualization display to obtain a scatter plot, each point in the scatter plot corresponds to the text data one by one, and visualization is performed, and then cluster analysis is further performed using the DBscan clustering model to obtain cluster clusters, remove text data with large differences, that is, remove noise data, and display in real time, extract the keywords and the text data to be processed in the cluster clusters, obtain cluster keywords and cluster keywords, and implement data screening, and finally, according to the cluster text data as nodes and the cluster keywords as attributes, the cluster keywords corresponding to different cluster text data are matched through a pre-trained matching model, and if the cluster keywords are successfully matched, the corresponding cluster text data are connected to form a knowledge graph, so that the correlation of each node of the knowledge graph obtained in the end is high, and the efficiency of data analysis is improved, and the knowledge graph obtained in the end can enable users to grasp information more efficiently.

[0117] like Figure 3 The figure shows a functional module diagram of the text data visualization analysis device of the present application.

[0118] The text data visualization analysis device 100 described in the present application can be installed in an electronic device. According to the functions to be implemented, the text data visualization analysis device 100 may include an acquisition module 101, a quantification module 102, a visualization module 103, a clustering module 104, an extraction module 105 and a construction module 106. The module described in the present application may also be referred to as a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0119] In this embodiment, the functions of each module / unit are as follows:

[0120] An acquisition module 101 is used to acquire a plurality of text data to be processed;

[0121] Further, the acquisition module 101 includes a request sending submodule and a calling submodule;

[0122] The request sending submodule is used to send a call request to the database, wherein the call request carries a signature verification token;

[0123] The calling submodule is used to receive the signature verification result returned by the database, and when the signature verification result is passed, call the text data in the database to obtain the multiple text data to be processed.

[0124] The method of verifying the signature by requesting the cooperation of the sending submodule and the calling submodule can ensure the security of the content in the database and avoid leakage of user privacy, that is, text data, etc.

[0125] A quantification module 102 is used to perform keyword extraction processing on the plurality of text data to be processed to obtain keywords and their weights corresponding to the text data to be processed;

[0126] Furthermore, the quantification module 102 includes a Jieba word segmentation processing submodule and a keyword extraction submodule;

[0127] The Jieba word segmentation processing submodule is used to perform word segmentation processing and stop word removal on the text data to be processed using the Jieba word segmentation to obtain a corresponding plurality of words;

[0128] The keyword extraction submodule is used to extract keywords from the multiple words using the TF-IDF algorithm and obtain the weights corresponding to the keywords.

[0129] The keyword extraction submodule extracts keywords from the words obtained after segmentation of each text to be processed, and obtains corresponding weights to extract the attributes of the text to be processed.

[0130] The Jieba word segmentation processing submodule directly utilizes the Jieba toolkit in Python. By importing this Jieba toolkit, it can process each input text data to be processed and segment it, thereby realizing the word segmentation processing of the text data to be processed.

[0131] The Jieba word segmentation processing submodule uses the Jieba word segmentation to split the text data into multiple independent words, which is convenient for subsequent keyword extraction.

[0132] Furthermore, the Jieba word segmentation processing submodule also includes a Trie tree unit, a dynamic programming unit, and a hidden Markov unit;

[0133] The Trie tree unit is used to scan the text data based on a preset Trie tree to identify multiple segmentation combinations of words in the text data;

[0134] The dynamic programming unit is used to construct a directed acyclic graph based on all identified segmentation combinations, use the directed acyclic graph to perform dynamic programming to find the maximum probability path, determine the segmentation combination with the maximum probability, and segment the sentence to be processed based on the segmentation combination with the maximum probability;

[0135] The hidden Markov unit is used to segment unrecognized words using a hidden Markov model.

[0136] By combining the Trie tree unit, dynamic programming unit and hidden Markov unit, the statement to be processed is segmented, and the segmentation combination closest to the actual situation can be obtained.

[0137] Furthermore, the keyword extraction submodule also includes a word frequency calculation unit, an inverse document frequency calculation unit, a weight calculation unit and a judgment unit;

[0138] A word frequency calculation unit, used for obtaining the number of times the first word appears and the total number of words in each document in the data to be processed, and obtaining the word frequency of the first word by dividing the number of times the first word appears by the total number of words;

[0139] an inverse document frequency calculation unit, configured to obtain the total number of documents in the data to be processed and the number of documents in which the first word appears, and calculate the inverse document frequency of the first word according to the total number of documents and the number of documents in which the first word appears;

[0140] A weight calculation unit, used for multiplying the word frequency corresponding to the first word by the inverse document frequency to obtain the weight of the first word;

[0141] The judgment unit is used to extract the first word whose weight is greater than a preset value as the keyword, and obtain the keyword and its corresponding weight.

[0142] Furthermore, the quantization module 102 further includes a sorting submodule and a sorting extraction submodule;

[0143] The sorting submodule is used to sort the keywords according to the weights corresponding to the keywords;

[0144] The sorting and extracting submodule is used to extract a preset number of keywords based on the sorting result.

[0145] Specifically, since multiple keywords will be obtained through the TF-IDF algorithm, and the number of keywords obtained for different text data to be processed may be inconsistent, the keywords corresponding to each text data to be processed are sorted in descending order through the sorting submodule, and the keyword extraction submodule is used to extract the preset number of keywords to achieve the consistency of the number of keywords in each text data to be processed, that is, the consistency of the attribute dimension.

[0146] The attribute dimensions of each text data are unified through the cooperation of the sorting submodule and the sorting extraction submodule, which is conducive to the subsequent visualization display using visualization tools.

[0147] A visualization module 103 is used to perform visualization display using a visualization tool according to the plurality of text data to be processed and the corresponding keywords and their weights to obtain a scatter plot;

[0148] A clustering module 104 is used to perform cluster analysis based on the scatter plot using a DBscan clustering model to obtain clusters;

[0149] Furthermore, the text data visualization analysis device 100 also includes a center point extraction module and an output module;

[0150] The center point extraction module is used to process each of the clusters using the KMeans model to obtain center point data;

[0151] The output module is used to output the corresponding keywords and their weights based on the center point data.

[0152] Specifically, the center point extraction module processes each cluster in the scatter plot by using the KMeans model, specifically taking K=1, to obtain the center point data of each cluster, and the output module outputs the text data corresponding to the center point and its corresponding keywords based on the center point data.

[0153] An extraction module 105 is used to extract the corresponding keywords and text data to be processed based on each cluster to obtain cluster keywords and cluster text data;

[0154] Specifically, after obtaining each cluster in the scatter plot, the extraction module 105 extracts the text data to be processed and the keywords corresponding to each point in the cluster. The points correspond to the text data to be processed and the keywords. By acquiring each point in the scatter plot, the corresponding text data and keywords can be obtained, thereby finally obtaining the cluster keywords and cluster text data.

[0155] Through the cooperation of the center point extraction module and the output module, the KMeans model is used to obtain the center point data of each cluster, and the text data and keywords corresponding to the center point data are obtained. The keywords corresponding to the center point data can be used as the characteristic attributes of the cluster, which improves the clarity of the cluster data features.

[0156] Construction module 106 is used to use the clustering text data as nodes and the clustering keywords as attributes, and to match the clustering keywords corresponding to different clustering text data through a pre-trained matching model. If the clustering keywords are matched successfully, the corresponding clustering text data will be connected to construct a knowledge graph.

[0157] Specifically, the construction module 106 takes each clustered text data as a node, which is actually directly represented by a point, and distinguishes each point, and uses its corresponding clustering keyword as an attribute, and determines whether the clustered text data are connected in pairs according to the clustering keyword.

[0158] The construction module 106 obtains the similarity between the cluster keywords corresponding to the two cluster text data by using the pre-trained matching model, and compares and judges based on the similarity and the preset value. If the similarity is greater than the preset value, the two cluster text data are connected in series, that is, connected. If the similarity between multiple cluster keywords in the two cluster text data is greater than the preset value, the greater the number, the closer the distance between the two cluster text data in the entire knowledge graph; conversely, if the similarity is less than the preset value, the two cluster text data are not connected. When the cluster keywords corresponding to all cluster text data are calculated in pairs, a knowledge graph is formed after all calculations are completed.

[0159] By adopting the above device, the text data visualization analysis device 100 uses the acquisition module 101, the quantification module 102, the visualization module 103, the clustering module 104, the extraction module 105 and the construction module 106 in coordination, so that while improving the data analysis efficiency, the text data with large differences are removed, so that the correlation between the nodes of the knowledge graph obtained in the end is higher, the relationship between the nodes of the knowledge graph is clearer, and the user can grasp the information more efficiently.

[0160] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0161] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0162] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0163] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the text data visualization analysis method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0164] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the text data visualization analysis method.

[0165] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0166] This embodiment implements the steps of the text data visualization analysis method of the above embodiment when the processor executes the computer-readable instructions stored in the memory, by acquiring multiple text data to be processed, and performing keyword extraction processing on the text data to be processed to obtain the keywords and their weights corresponding to the text data to be processed, that is, to obtain the attributes of the text data; according to the text data to be processed and the corresponding keywords and their weights, a visualization tool is used to perform a visualization display to obtain a scatter plot, and each point in the scatter plot corresponds to the text data one by one, and a visualization representation is performed, and then a cluster analysis is further performed using the DBscan clustering model to obtain cluster clusters, and the relatively different ones are removed. Large text data, i.e., noise data, is removed and displayed in real time, and the keywords and text data to be processed in the clustering cluster are extracted to obtain clustering keywords and clustering keywords to realize data screening. Finally, based on the clustering text data as nodes and the clustering keywords as attributes, the clustering keywords corresponding to different clustering text data are matched through a pre-trained matching model. If the clustering keywords are matched successfully, the corresponding clustering text data are connected to form a knowledge graph, so that the correlation between the nodes of the final knowledge graph is higher, and the efficiency of data analysis is improved. As well as the final knowledge graph, users can grasp information more efficiently.

[0167] The present application also provides another implementation mode, that is, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor, so that the at least one processor performs the steps of the above-mentioned text data visualization analysis method, by obtaining a plurality of text data to be processed, and performing keyword extraction processing on the text data to be processed, obtaining keywords and their weights corresponding to the text data to be processed, that is, obtaining the attributes of the text data; according to the text data to be processed and the corresponding keywords and their weights, using a visualization tool for visualization display, obtaining a scatter plot, each point in the scatter plot corresponds to the text data one by one, and performing visualization representation, and further using D The Bscan clustering model is used for cluster analysis to obtain cluster clusters, remove text data with large differences, that is, remove noise data, and display them in real time, extract the keywords and text data to be processed in the cluster clusters, obtain cluster keywords and cluster keywords, and implement data screening. Finally, based on the cluster text data as nodes and the cluster keywords as attributes, the cluster keywords corresponding to different cluster text data are matched through the pre-trained matching model. If the cluster keywords are successfully matched, the corresponding cluster text data are connected to form a knowledge graph, so that the correlation between the nodes of the final knowledge graph is higher, and the efficiency of data analysis is improved. As well as the final knowledge graph, users can grasp information more efficiently.

[0168] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0169] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A text data visualization analysis method, characterized in that: The method comprises: Get multiple text data to be processed; Perform keyword extraction processing on the plurality of text data to be processed to obtain keywords and their weights corresponding to the text data to be processed; Using a visualization tool to perform visualization according to the plurality of text data to be processed and the corresponding keywords and their weights, a scatter plot is obtained; According to the scatter plot, cluster analysis is performed using a DBscan clustering model to obtain clusters; Based on each of the clusters, extract the corresponding keywords and the text data to be processed to obtain cluster keywords and cluster text data; The clustered text data is used as a node and the clustered keywords are used as attributes. The clustered keywords corresponding to different clustered text data are matched through a pre-trained matching model. If the clustered keywords are successfully matched, the corresponding clustered text data are connected to construct a knowledge graph; The keyword extraction process of the plurality of text data to be processed comprises: Using the Jieba word segmentation to perform word segmentation processing on the text data to be processed and to remove stop words, to obtain a corresponding plurality of words; The TF-IDF algorithm is used to extract keywords from the multiple words, and the weights corresponding to the keywords are obtained.

2. The text data visualization analysis method according to claim 1, characterized in that: The obtaining of a plurality of text data to be processed comprises: Send a call request to the database, the call request carrying a signature verification token; The signature verification result returned by the database is received, and when the signature verification result is passed, the text data in the database is called, so as to obtain the multiple text data to be processed.

3. The text data visualization analysis method according to claim 1 is characterized in that: The using the Jieba word segmentation to perform word segmentation processing on the text data to be processed comprises: Scan the text data based on a preset Trie tree to identify multiple segmentation combinations of words in the text data; Constructing a directed acyclic graph based on all identified segmentation combinations, using the directed acyclic graph to perform dynamic programming to find the maximum probability path, determining the segmentation combination with the maximum probability, and segmenting the sentence to be processed based on the segmentation combination with the maximum probability; For unrecognized words, a hidden Markov model is used for segmentation.

4. The text data visualization analysis method according to claim 1, characterized in that: The using of the TF-IDF algorithm to extract keywords from the multiple words and obtain the corresponding weights includes: Obtain the number of times a first word appears in each document in the text data to be processed and the total number of words, and obtain the word frequency of the first word by dividing the number of times the first word appears by the total number of words; Obtaining the total number of documents in the text data to be processed and the number of documents in which the first word appears, and calculating the inverse document frequency of the first word according to the total number of documents and the number of documents in which the first word appears; Multiply the word frequency corresponding to the first word by the inverse document frequency to obtain the weight of the first word; The first word whose weight is greater than a preset value is extracted as the keyword, and the keyword and its corresponding weight are obtained.

5. The text data visualization analysis method according to claim 4 is characterized in that: After extracting keywords from the multiple words using the TF-IDF algorithm and obtaining the corresponding weights, the method further includes: Sorting the keywords according to the weights corresponding to the keywords; Based on the sorting results, a preset number of keywords are extracted.

6. The text data visualization analysis method according to any one of claims 1 to 5, characterized in that: After performing cluster analysis using the DBscan clustering model to obtain clusters, the method further includes: Using the KMeans model, each cluster is processed to obtain the center point data; Based on the center point data, the corresponding keywords and their weights are output.

7. A text data visualization analysis device, characterized in that: The device comprises: An acquisition module, used for acquiring a plurality of text data to be processed; A quantification module, used for performing keyword extraction processing on the plurality of text data to be processed, and obtaining keywords and their weights corresponding to the text data to be processed; A visualization module, used for performing visualization display using a visualization tool according to the plurality of text data to be processed and the corresponding keywords and their weights to obtain a scatter plot; A clustering module, used to perform cluster analysis based on the scatter plot using a DBscan clustering model to obtain clusters; An extraction module, used for extracting the corresponding keywords and text data to be processed based on each of the clusters to obtain cluster keywords and cluster text data; A construction module is used to use the clustered text data as nodes and the clustered keywords as attributes, and to match the clustered keywords corresponding to different clustered text data through a pre-trained matching model. If the clustered keywords are successfully matched, the corresponding clustered text data are connected to construct a knowledge graph; The quantification module includes: a Jieba word segmentation processing submodule and a keyword extraction submodule; The Jieba word segmentation processing submodule is used to perform word segmentation processing and stop word removal on the text data to be processed using the Jieba word segmentation to obtain a corresponding plurality of words; The keyword extraction submodule is used to extract keywords from the multiple words using the TF-IDF algorithm and obtain the weights corresponding to the keywords.

8. A computer device, characterized in that: The computer device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the text data visualization analysis method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the text data visualization analysis method according to any one of claims 1 to 6 is implemented.

Images