Systems and methods for using text analytics to transform, analyze, and visualize data - Patents.com

Abstract

According to one embodiment, described herein are systems and methods for using text analytics to transform, analyze, and visualize data, including support for data flows of unstructured text or other types of text data input. According to one embodiment, the system can be implemented in a cloud environment that enables self-service text analytics. A user can interact with the system via a user interface to apply natural language processing or other text analytics techniques to a data flow or dataset of input data to generate visualizations or other types of useful information related to the data.

Description

[Technical field]

[0001] Copyright Notice A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to anyone being able to reproduce the patent document or the patent disclosure as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all copyright rights whatsoever.

[0002] Claiming priority This application claims the benefit of priority to U.S. patent application Ser. No. 17 / 408,226, entitled "SYSTEM AND METHOD FOR USE OF TEXT ANALYTICS TO TRANSFORM, ANALYZE, AND VISUALIZE DATA," filed on August 20, 2021, the above-mentioned application and the contents thereof are incorporated herein by reference.

[0003] Technical Field The embodiments described herein relate generally to systems and methods for processing data flows or datasets of input data, and the use of text analytics to transform, analyze, and visualize data, including supporting data flows of unstructured text or other types of textual data. [Background technology]

[0004] Background technology Generally described, data analytics allows for computer-based analysis and interpretation of input data, usually large amounts of data, in order to derive additional useful information from that data. For example, within a business organization, various business intelligence (BI) tools may be used to provide business users with information describing organizational data in a presentation format that aids the users in making strategic business decisions. Data analytics may also be applied in other environments, such as science and engineering, media communications, electronic commerce, or online search applications.

[0005] Over the past few years, the use of business intelligence and other forms of data analytics has expanded beyond the analysis of relatively simple numeric or textual category label types of data. Today, much of the data of interest to a particular organization arrives in the form of unstructured text or other data, such as emails, text messages, responses to online surveys, or other types of free-form or unstructured text. Summary of the Invention [Problem to be solved by the invention]

[0006] The use of text analytics can be applied to these types of data to better understand and utilize such data, however, typical approaches to using text analytics generally require users to be familiar with applying techniques such as computer-based natural language processing, which can make them less accessible to business or everyday users. [Means for solving the problem]

[0007] overview According to one embodiment, described herein are systems and methods that use text analytics to transform, analyze, and visualize data, including support for data flows of unstructured text or other types of textual data inputs.

[0008] Additionally, various examples of algorithmic processes and user interfaces are described that can be used to enable text analysis in particular environments or use cases. Effect of the Invention

[0009] According to one embodiment, the system can be implemented in a cloud environment that enables self-service text analytics: users, for example, business users of an organization who may not be experts in the use of machine learning as applied to data processing, can interact with the system via a user interface to apply natural language processing or other text analytics techniques to a data flow or data set of input data to generate visualizations or other types of useful information related to the data. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 illustrates a system for using text analytics to transform, analyze, or visualize data, according to one embodiment. [Diagram 2] FIG. 1 further illustrates a system for using text analytics to transform, analyze, or visualize data, according to one embodiment. [Diagram 3] FIG. 1 further illustrates a system for using text analytics to transform, analyze, or visualize data, according to one embodiment. [Figure 4] FIG. 1 further illustrates a system for using text analytics to transform, analyze, or visualize data, according to one embodiment. [Diagram 5] FIG. 1 further illustrates a system for using text analytics to transform, analyze, or visualize data, according to one embodiment. [Figure 6] FIG. 1 further illustrates a system for using text analytics to transform, analyze, or visualize data, according to one embodiment. [Figure 7] FIG. 1 further illustrates a system for using text analytics to transform, analyze, or visualize data, according to one embodiment. [Figure 8] FIG. 1 further illustrates a system for using text analytics to transform, analyze, or visualize data, according to one embodiment. [Figure 9]FIG. 1 further illustrates a system for using text analytics to transform, analyze, or visualize data, according to one embodiment. [Figure 10] FIG. 1 illustrates a process for using text analytics to transform, analyze, or visualize data, according to one embodiment. [Figure 11] FIG. 1 illustrates the use of a user interface with the system to provide self-service text analysis, according to one embodiment. [Figure 12] FIG. 1 illustrates the use of a user interface with the system to provide self-service text analysis, according to one embodiment. [Figure 13] FIG. 1 illustrates the use of a user interface with the system to provide self-service text analysis, according to one embodiment. [Figure 14] FIG. 1 illustrates the use of a user interface with the system to provide self-service text analysis, according to one embodiment. [Figure 15] FIG. 1 illustrates an example of the use of a user interface to provide self-service text analytics, according to one embodiment. [Figure 16] FIG. 1 illustrates another example use of a user interface for providing self-service text analytics, according to one embodiment. [Figure 17] FIG. 1 illustrates another example use of a user interface for providing self-service text analytics, according to one embodiment. [Figure 18] FIG. 1 illustrates another example use of a user interface for providing self-service text analytics, according to one embodiment. [Figure 19] FIG. 1 illustrates a process for using a user interface to provide self-service text analytics, according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Detailed Description As mentioned above, much of the data of interest to a particular organization today arrives in the form of unstructured data, such as emails, text messages, responses to online surveys, or other types of free-form or unstructured text. However, common approaches to using text analytics on such data require users to be particularly adept at applying techniques such as computer-based natural language processing (NLP), making such capabilities difficult to utilize for business or everyday users.

[0012] According to one embodiment, described herein are systems and methods that use text analytics to transform, analyze, and visualize data, including support for data flows of unstructured text or other types of textual data inputs.

[0013] Additionally, various examples of algorithmic processes and user interfaces are described that can be used to enable text analysis in particular environments or use cases.

[0014] For example, according to various embodiments, the system may apply one or more algorithmic processes that perform Latent Dirichlet Allocation (LDA) clustering, term frequency-inverse document frequency (TF-IDF) based sentiment analysis, and / or assessment of reading grade level as a machine learning (ML) feature to control or supplement the application of text analytics to a particular data flow or dataset of input data.

[0015] According to one embodiment, the system can be implemented in a cloud environment that enables self-service text analytics: users, for example, business users of an organization who may not be experts in the use of machine learning as applied to data processing, can interact with the system via a user interface to apply natural language processing or other text analytics techniques to a data flow or data set of input data to generate visualizations or other types of useful information related to the data.

[0016] According to various embodiments, examples of the types of free-form or unstructured textual data that can be received as input and examined using the systems and methods described herein include email messages, text (e.g., SMS) messages, text responses to online surveys, tweets, customer service representative chat transcripts, medical transcription notes, online product / movie reviews, comments provided within a human capital management (HCM), human resources (HR), or enterprise resource planning (ERP) system, Slack chats, scraped web pages, song lyrics, or e-commerce product descriptions.

[0017] The above examples of various types of free-form or unstructured textual data are provided by way of example and for illustrative purposes, and according to various embodiments, the systems and methods described herein can similarly be used to apply text analytics to transform, analyze, and visualize other types of data flows of unstructured text or other types of textual data.

[0018] Data Analysis Environment FIG. 1 illustrates a system for using text analytics to transform, analyze, or visualize data, according to one embodiment.

[0019] The embodiment shown in Figure 1 is provided for purposes of illustrating one example of a computing environment that enables the use of text analytics to transform, analyze, and visualize other types of data flows of unstructured text or other types of textual data. According to various embodiments, the various components, processes, and functions described and illustrated herein may also be used in other types of data analysis systems or computing environments.

[0020] According to various embodiments, the components, processes, and functions illustrated in the figures and further described herein may be provided as software or program code executable by a computer system or other type of processing device or computing environment, which in some embodiments includes a cloud-based computing environment (cloud environment), such as, for example, an Oracle Analytics Cloud (OAC) environment.

[0021] As shown in FIG. 1, according to one embodiment, a data analysis system 100 may provide a data enrichment system 200 that includes device hardware 110 (e.g., processor, memory) and enables access by client devices / applications 120 to access and transform, analyze, or visualize data provided by the analysis system.

[0022] For example, according to one embodiment, the analytics system may be provided as a cloud environment such as OAC that enables access to client devices / applications via a cloud service such as Oracle Analytics Cloud Service.

[0023] According to other embodiments, the analysis system may be provided by or in association with other types of computing environments, including, for example, other types of cloud environments or on-premise (e.g., non-cloud) based computing environments.

[0024] According to one embodiment, the client application may be implemented as software or computer readable program code executable by a computer system or processing device and having a user interface 130, such as, for example, a software application user interface or a web browser interface. The client application may obtain or access data via an Internet / HTTP or other type of network connection to the analysis system or, in the example of a cloud environment, via cloud services provided by the environment.

[0025] According to one embodiment, the user interface may include or provide access to a variety of data flow action types that enable self-service text analysis, including enabling a user to view a data set or interact with the user interface to transform, analyze, or visualize data, e.g., generating graphs, charts, or other types of data analysis or visualization 140 of data flows associated with unstructured text or other types of text data, as described in more detail below.

[0026] According to one embodiment, the analysis system enables a dataset 240 to be obtained, received, or prepared from one or more data sources 250, for example, via one or more data source connections.

[0027] As mentioned above, examples of the types of data that can be transformed, analyzed, or visualized using the systems and methods described herein include HCM, HR, or ERP data, emails or text messages, or other free-form or unstructured text data.

[0028] For example, according to an embodiment that enables business intelligence (BI) tools to be used with organizational data, data may be obtained, received, or prepared via a business intelligence server 254 that communicates with one or more of a database 255, a data storage service 257, or other type of data repository or data source.

[0029] According to one embodiment, a request 142 for data analysis or visualization information may be received via a client application and user interface as described above and communicated to the analysis system (in the example cloud environment, via a cloud service).

[0030] The system can create a user / business context 264 associated with the request and retrieve an appropriate data set to address the user / business context 265 for use in generating and returning the requested data analysis or visualization information to the client 144. For example, according to one embodiment, the data analysis system can retrieve the data set using, for example, a SELECT statement or a logical SQL 266 instruction.

[0031] According to one embodiment, the system can create a model or dataflow 290 that reflects an understanding of the dataflow or dataset of input data by applying various algorithmic processes, including applying one or more natural language processing or other text analysis techniques, to the dataflow or dataset of input data, as further described herein, to generate visualizations or other types of useful information related to the data.

[0032] According to one embodiment, the data enrichment system may further include various components, processes, or functions, such as, for example, an ingest server 270, a preparation server 274, a transformation server 276, a recommendation server 278, a sentiment analysis server 280, a publishing server 282, a knowledge server 284, and an enrichment server 286, any or all of which may be used in various combinations by the system to further transform, analyze, or visualize the data.

[0033] For example, according to one embodiment, data sources can be sampled by an ingest server and the sampled data can be analyzed for enrichment. The identified data can be received and added to a distributed storage system, such as Hadoop Distributed Storage (HDFS), accessible to a data enrichment server. The data can be semantically processed by a pipeline having multiple processing stages.

[0034] For example, such processing stages may include a preparation stage controlled by one or more preparation servers, a publication stage controlled by a publication server, and / or an enrichment stage controlled by an enrichment server.

[0035] According to one embodiment, the preparation stage can be adapted to automatically detect the format of the data source and perform appropriate content extraction. Once the format of the data source is identified, the data source can be automatically prepared or normalized into a format that can be processed by the enrichment server.

[0036] According to one embodiment, inbound data sources can be loaded into a distributed storage system coupled to the enrichment server, which provides temporary storage space for storing ingested data files, intermediate processing files, and results prior to publication. The enhanced or enriched results can also be stored in the distributed storage system along with any metadata captured during the enrichment process and associated with the ingested data.

[0037] According to one embodiment, the data enrichment system can provide sentiment analysis through a sentiment analysis server that includes functionality for analyzing the sentiment of data from different data sources.

[0038] According to one embodiment, the publishing server can provide the data source metadata captured during enrichment to one or more visualization systems for analysis (e.g., to display recommended data transformations, enrichments, and / or other modifications). The publishing server can send the processed data to one or more data targets, such as, for example, another computing system, a database, or a service.

[0039] According to one embodiment, the data may be enriched to include additional content, such as, for example, association information for unknown or new words.

[0040] As indicated above, according to various embodiments, the components, processes, and functions shown in the figures and described herein may be provided as software or program code executable by a computer system or other type of processing device or computing environment, including in some embodiments a cloud environment such as an OAC.

[0041] For example, according to one embodiment, the data enrichment system and its various components, processes, and functions may be implemented in software (e.g., program code or instructions executable by one or more processors or computing devices provided within the cloud environment). According to various embodiments, data may be stored using one or more data structures that are organized in a variety of ways depending on how, what, and / or where the data is stored and stored within a data storage component of the cloud environment.

[0042] 2 and 3 further illustrate a system for using text analytics to transform, analyze, or visualize data, according to one embodiment.

[0043] As explained above, according to one embodiment, the system can create a model or data flow that reflects an understanding of the data flow or data set of input data.

[0044] As shown in FIG. 2, according to one embodiment, the model or dataflow may be further modified by applying various suitable natural language processing or other text analysis techniques to the input dataflow or dataset, including, for example, one or more dataflow actions 291, 292, which operate on the input dataflow or dataset.

[0045] For example, as shown in FIG. 3, according to one embodiment, the system may be adapted to apply one or more text classification 294, text transformation 295, text extraction 296, or document clustering 297 data flow actions that operate on a data flow or dataset of input data.

[0046] As will be further described below, according to one embodiment, a user can interact with the system via a user interface to control the use of dataflow actions to apply natural language processing or other text analysis techniques to a dataflow or dataset of input data to generate visualizations or other types of useful information related to the data.

[0047] Automatic topic naming determined by LDA clustering According to various embodiments, the system may apply one or more algorithmic processes that perform Latent Dirichlet Allocation (LDA) clustering to control or supplement the application of text analytics to a particular data flow or data set of input data.

[0048] Generally described, an LDA algorithm or process can be used to generate a model that allows for the observation of different data in a dataset based on a determination of similarity within the data. For example, as applied to data flows or documents of unstructured text or other types of textual data, LDA can be used to characterize or associate documents with topics (generally words or multi-word phrases) such that the presence of a word / phrase in the dataset can be attributed to one of those topics.

[0049] However, while LDA can be used to cluster an input data flow or dataset of documents into topics, typical LDA techniques do not, for example, provide labels or names for those topics.

[0050] According to one embodiment, the system can apply an LDA process that performs automatic naming of topics related to a data flow or dataset of input data using an approach that generally involves generating multiple multi-word topics or N-grams (e.g., bigrams, i.e., two-word topic titles, and trigrams, i.e., three-word topic titles), treating each generated title as if it were an entire document, and using a scoring process to determine which particular title has the greatest probability of being related to a particular topic. Given a particular document, the system can return the probability that the document belongs to each of the N topics.

[0051] 4-8 further illustrate a system for using text analytics to transform, analyze, or visualize data, according to one embodiment.

[0052] As shown in FIG. 4, according to one embodiment, the process performed by the system includes:

[0053] Receiving a collection of documents D300 as input or as a data flow. Such documents may be received in the form of unstructured data, such as emails, text messages, responses to an online survey, or other types of free-form or unstructured text. By way of illustration, a typical document might contain several long strings of text, perhaps between 40 and 3000 characters each.

[0054] From the collection of documents D, generate a vocabulary V1 (310) to be used in the LDA process. In order to exploit the information contained in word order, this vocabulary should include not only individual words 312, but also word pairs (bigrams 314) and / or word triples (trigrams 316), or longer phrases or sequences (if such sequences of words occur frequently enough in the collection of documents D).

[0055] According to one embodiment, the LDA process can be accomplished, for example, by using the SciKitLearn library class CountVectorizer and specifying the ngram_range parameter as ngram_range=(1,3), which instructs the system to allow all unigrams, bigrams, and trigrams to be considered as candidate vocabulary terms.

[0056] In such an embodiment, other CountVectorizer parameters can be set to prevent overfitting of the learning model. For example, the max_df parameter (maximum document frequency) can be set to something less than 1.0, such as 0.85, so that terms that appear in more than 85% of the documents are not considered distinctive enough for the vocabulary used by LDA. The max_features parameter (maximum size of the resulting vocabulary) can be set relatively low, such as 1000, so that the model does not overfit to spurious terms.

[0057] The description of various embodiments provided herein that utilize the SciKitLearn library for LDA processing purposes is provided for purposes of illustrating example LDA processing libraries. According to other embodiments, the analysis system may utilize other types of LDA processing libraries, algorithms, processes, or functions.

[0058] 5, according to one embodiment, an LDA process is performed by the system on a collection of documents D using vocabulary V1 to generate N topics 320. According to various embodiments, the value of N can be either (a) hard-coded to something useful for different use cases, such as N=6, (b) specified by the user, or (c) automatically determined by the system by trying different values ​​of N and measuring topic coherence.

[0059] As shown in FIG. 6, according to one embodiment, the system generates a second vocabulary V2 (340) from the collection of documents D that is much larger than vocabulary V1 (e.g., in this example, by including 314, 316, and also 344).

[0060] According to one embodiment, as shown above, the LDA process can be accomplished, for example, using the SciKitLearn library class CountVectorizer. In such an embodiment, to generate the second vocabulary, the parameter of max_features can be relaxed (e.g., to 100,000 instead of 1000), and the parameter of max_df can also be relaxed (e.g., to 0.95 instead of 0.85) by setting ngram_range=(2,3) to exclude single-word unigrams (e.g., 312, 342 in this example). The LDA process tracks the probability that each vocabulary term occurs in each topic.

[0061] As described above, the use of the SciKitLearn library for LDA processing purposes is provided for illustrative purposes, and according to other embodiments, the analysis system may utilize other types of LDA processing libraries, algorithms, processes, or functions.

[0062] According to one embodiment, the system determines the union U330 of the sets of the top M words for each of the N topics (where a typical value of M is 50), and then generates a topic title for each topic i of the N topics by doing the following:

[0063] As shown in FIG. 7, generating a set of candidate topic titles T350 by selecting from vocabulary V2 all multi-word topics or N-grams (e.g., bigrams, i.e., two-word topic titles, and trigrams, i.e., three-word topic titles) whose constituent words at least appear in the union U of the sets of the top M words for each of the N topics.

[0064] As shown in Figure 8, score each candidate topic title from T by:

[0065] Calculate its LDA score (treat the 2 or 3 word title as if it were the entire document, and calculate the probability that "document" is in each of the N topics). This produces a vector S of length N, which is the probability for each topic.

[0066] Calculating candidate topic title scores (352, 354), where s j is the j-th element of S, and α is typically 3.

[0067]

number

[0068] According to one embodiment, the system may then select the top-scoring candidate topic title as the label or name for that topic.

[0069] Using Sentiment Analysis According to various embodiments, the system may apply one or more algorithmic processes that perform word frequency-inverse document frequency (TF-IDF) based sentiment analysis and / or assessment of reading grade level as machine learning features to control or supplement the application of text analytics to a particular data flow or dataset of input data.

[0070] Generally described, a TF-IDF algorithm or process can be used to provide a value or indication of how important a particular word may be to a particular document within a collection of documents. The TF-IDF value generally increases in proportion to the number of times the word occurs in a document, offset by the number of documents within the collection of documents, reflecting the fact that some words may occur relatively more frequently. Such an algorithm or process can be used to perform sentiment analysis or provide an indication of some subjective information associated with a particular document.

[0071] Generally described, a reading (grade) level algorithm or process may be used to provide a value or indication of the reading level of a particular document or text. According to one embodiment, the reading grade level may be calculated based on, for example, the number of sentences in the document, the total number of words, the number of syllables, the presence of unusual words or phrases, or other characteristics indicative of a typical reading grade level.

[0072] FIG. 9 further illustrates a system for using text analytics to transform, analyze, or visualize data, according to one embodiment.

[0073] As shown in FIG. 9, according to one embodiment, once a vocabulary associated with a collection of documents D has been generated (e.g., by generating vocabulary V1 and / or vocabulary V2 as described above), the system may apply a TF-IDF algorithm or process, in conjunction with logistic regression, to perform sentiment analysis on the particular documents.

[0074] According to one embodiment, the TF-IDF process converts documents into numerical document vectors 362. At the same time, the system can perform a reading grade analysis 362, and that information is appended to the document vector so that the final document vector 366 includes a reading grade level indicator 368 for later use in machine learning.

[0075] In addressing certain use cases, for example where negative sentiment is correlated with relatively low reading grade levels, incorporating reading grade level indications within the document vectors improves the accuracy of the resulting model.

[0076] The text analysis process FIG. 10 illustrates a process for using text analytics to transform, analyze, or visualize data according to one embodiment.

[0077] As shown in FIG. 10 , according to one embodiment, in step 402, a data analysis system receives an analysis / visualization request related to one or more structured or unstructured data provided by a data source, and the system is adapted to provide a model or data flow describing one or more input documents D in the data using a number of data flow action components.

[0078] In step 404, the system creates a vocabulary V1 from document D that contains the individual words and larger N-grams (eg, bigrams, trigrams) that occur with relatively high frequency in D.

[0079] In step 406, the system applies an algorithm (eg, LDA) to document D using vocabulary V1 to generate N topics.

[0080] In step 408, the system creates a vocabulary V2 from document D that excludes individual words but includes the larger N-grams from V1 and additional N-grams that occur relatively infrequently in D.

[0081] In step 410, the system determines a union U of the sets of the top M words for each of the N topics, and generates a set T of candidate topic titles for N-grams in V2 where at least one constituent word occurs in U.

[0082] In step 412, the system scores each candidate topic title from T (LDA) and selects the top-scoring candidate topic title to associate with document D.

[0083] In step 414, the system may (optionally) perform a sentiment analysis associated with document D, which may include converting the document into a numerical vector (e.g., TF-IDF) and appending to that vector the reading grade level associated with the document's vocabulary (e.g., V1, V2, etc.).

[0084] In step 416, the system provides and / or graphically displays within a user interface as a visualization an analysis / visualization related to one or more structured or unstructured data provided by the data sources.

[0085] User Interface for Self-Service Analytics As discussed at the beginning, typical approaches for using text analytics with such data require the user to be particularly skilled in applying techniques such as computer-based natural language processing, making such capabilities difficult to utilize for business or everyday users.

[0086] According to one embodiment, the system can be implemented in a cloud environment that enables self-service text analytics, where business users of an organization, who may not be experts in the use of machine learning as applied to data processing, for example, can interact with the system via a user interface to apply natural language processing or other text analytics techniques to a data flow or dataset of input data and generate visualizations or other types of useful information related to the data.

[0087] 11-14 illustrate the use of a user interface with the system to provide self-service text analysis, according to one embodiment.

[0088] As shown in FIGS. 11-14 , according to one embodiment, a user interface provides access to data flow action types 420 that enable a user to specify, for example, one or more of add data 421, filter 422, train numerical prediction 423, train multiple classifiers 424, text classification 425, text transformation 426, text extraction 427, document clustering 428, or other types of data flow actions that operate on a data flow or dataset of particular input data (429), including a data flow of unstructured text or other types of textual data input.

[0089] According to one embodiment, the user interface organizes various machine learning, natural language processing, or other text analysis techniques into easily accessible data flow action types, such as, for example:

[0090] Text classification, e.g., sentiment, reading grade level, engagement level, novelty, humor, sarcasm, stance / rumor, language detection, neologism classification, outlier scores.

[0091] Text transformation, e.g. preparation, correction, language translation, summarization, vectorization for machine learning, TF-IDF, Word2Vec, Doc2Vec.

[0092] Text extraction, e.g. named entity recognition, part-of-speech tagging, coreference analysis, word counting. Document clustering, e.g., LDA.

[0093] The above examples are provided by way of example and for illustrative purposes, and according to various embodiments, the user interface may enable access to other types of data flow actions associated with other natural language processing techniques.

[0094] According to one embodiment, the user interface may enable a user to specify one or more data flow action types to use with a dataset, or otherwise interact with the user interface to transform, analyze, or visualize the data, e.g., to generate a graph, chart, or other type of data analysis or visualization.

[0095] According to one embodiment, the user interface allows the user to combine pairs of operations into a single operation: for example, a stop word removal operation and a lemma analysis operation can be combined into an action called "Prep" (with configuration options that the user can control independently), and similarly, a word count combination operation and a Group By operation can be combined into a single operation.

[0096] According to one embodiment, when used with other data flow environments, such as Oracle Cloud Infrastructure Data Flow, the user interface can leverage generic, e.g., Data Flow blocks that perform machine learning, such that the inclusion of text analytics is treated within the user interface as any other form of data flow operation. Advantages of this approach include, for example, a reduction in the number of action blocks required on the Data Flow tool palette, and also allows for the end user to be shown that text analytics can be considered along with the numerical data processing used in machine learning.

[0097] Example use cases 15-18 illustrate various use cases in which the system and user interface can be used to provide self-service text analysis, for example, in business, science and engineering, media and communications, e-commerce, or online search applications.

[0098] As shown in FIG. 15, according to one embodiment and an example HCM use case, a user can use the system and user interface to configure a model or data flow to include, in this example, one or more document clustering and text classification (sentiment analysis) data flow actions.

[0099] Once the data is processed, the system may perform one or more of LDA clustering, TF-IDF based sentiment analysis, and / or assessment of reading grade level as machine learning features to control or supplement the application of text analytics to the data, as described above.

[0100] As shown in FIG. 16, according to one embodiment, a user can use the system and user interface to modify a model or data flow to include, in this example, one or more text classification (sentiment analysis) and text extraction (word count) data flow actions.

[0101] As shown in FIG. 17, according to one embodiment, a user can interact with a user interface to modify a model or data flow to include, in this example, one or more text transformation (data preparation) and text classification TF-IDF operations, and train a multi-class model, in this example using a support vector machine (SVM) for classification and regression. When provided in association with an environment, such as an OAC, that provides a user interface that allows user control over the training of machine learning models, for example via a Data Flow ML block, the system can utilize such Data Flow ML blocks to reduce the number of options presented in the user palette (e.g., there is no need to provide an NLP-specific Data Flow block on the palette to train a custom ML model). This approach can help reduce user training time.

[0102] Continuing with the above example, as shown in FIG. 18, according to one embodiment, the system can apply the models created during the above data flows to transform, analyze, and visualize data flows including, for example, unstructured text or other types of textual data inputs, including reading grade level assessments.

[0103] According to various embodiments, the described techniques can be used, for example, to detect positive / negative sentiment in a particular document, to detect hate speech, or to provide a quick assessment of, for example, free-form HR survey results or employee performance reviews. The techniques can be applied to the assessment of other types of unstructured data as well, such as online news articles or other media content.

[0104] Self-service analytical processes FIG. 19 illustrates a process of using a user interface to provide self-service text analysis, according to one embodiment.

[0105] According to one embodiment, in step 432, the system receives, in a data analysis system, a request for analysis / visualization related to one or more structured or unstructured data provided by a data source, and the system is adapted to provide a model or data flow describing one or more input documents D in the data using a number of data flow action components.

[0106] At step 434, the client application and user interface are adapted to display within the user interface an analysis / visualization related to one or more structured or unstructured data provided by the data sources.

[0107] At step 436, one or more options are provided within the user interface for selecting, modifying, and / or combining operations of selected ones or types of data flow action components to be used with the input document.

[0108] At step 438, one or more options are provided within the user interface for selecting, modifying, and / or combining operations of selected ones or types of data flow action components to be used with the input document.

[0109] According to various embodiments, the teachings herein may be conveniently implemented using one or more conventional general-purpose or special-purpose computers, computing devices, machines, or microprocessors, including one or more processors, memories, and / or computer-readable storage media programmed according to the teachings of the present disclosure. As will be apparent to those skilled in the software art, appropriate software coding may be readily produced by skilled programmers based on the teachings of the present disclosure.

[0110] In some embodiments, the teachings herein may include a computer program product that is a non-transitory computer-readable storage medium or media having stored thereon instructions that can be used to program a computer to perform any of the processes of the present teachings. Examples of such storage media may include, but are not limited to, hard disk drives, hard disks, hard drives, fixed disks or other electromechanical data storage devices, floppy disks, optical disks, DVDs, CD-ROMs, microdrives, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards, nanosystems, or other types of storage media or devices suitable for non-transitory storage of instructions and / or data.

[0111] The foregoing description has been provided for purposes of illustration and description. It is not intended to be exhaustive or to limit the scope of protection to the precise form disclosed. Many modifications and variations will be apparent to those skilled in the art.

[0112] The embodiments were chosen and described in order to best explain the principles of the present teachings and their practical application, so that those skilled in the art can appreciate various embodiments, with various modifications suited to the particular use contemplated. It is intended that the scope be defined by the following claims and their equivalents.

Claims

1. 1. A system for using text analytics to transform, analyze, and visualize data, including support for a data flow of unstructured text or other types of text data inputs, comprising: A data analysis system including a processor, a memory, and a data enrichment system that accesses data and enables access by client devices / applications to transform, analyze, or visualize said data. Equipped with 1. A system adapted to perform one or more clustering or sentiment analysis processes to determine candidate topic titles associated with a particular data flow or dataset of input data and to control or supplement the application of text analytics to said data.

2. 2. The system of claim 1, wherein the clustering process or the sentiment analysis process includes a Latent Dirichlet Allocation (LDA) process that determines vocabulary associated with a collection of documents in the particular data flow or dataset of input data, generates a number of topics associated with the documents, calculates a Latent Dirichlet Allocation (LDA) score for two- or three-word topic titles in the particular data flow or dataset of input data, and selects the top-scoring candidate topic titles as labels or names for the topics.

3. 10. The system of claim 1, further comprising: performing a term frequency-inverse document frequency (TF-IDF) based sentiment analysis and / or a reading grade level assessment on one or more documents in the particular data flow or dataset of input data, wherein the reading grade level indication is incorporated into a document vector describing the one or more documents.

4. 10. The system of claim 1, wherein the system is implemented in a cloud environment that enables self-service text analytics, the system including a user interface that enables a user to interact with the system to apply natural language processing or other text analytics techniques to a data flow or data set of input data and generate visualizations or other types of useful information related to the data.

5. 5. The system of claim 4, wherein the user interface enables access to dataflow action types that enable the user to specify one or more text classification, text transformation, text extraction, document clustering, or other types of dataflow actions that operate on a particular dataflow or data set of input data, including a dataflow of unstructured text or other types of textual data input.

6. 1. A method for using text analytics to transform, analyze, and visualize data, including support for a data flow of unstructured text or other types of text data inputs, comprising: providing a data analysis system including a processor, a memory, and a data enrichment system that accesses data and enables access by a client device / application to transform, analyze, or visualize said data; performing one or more clustering or sentiment analysis processes to determine candidate topic titles associated with particular data flows or data sets of input data for use in controlling or supplementing the application of text analytics to said data; The method includes:

7. 7. The method of claim 6, wherein the clustering process or the sentiment analysis process includes a Latent Dirichlet Allocation (LDA) process that determines vocabulary associated with a collection of documents in the particular data flow or dataset of input data, generates a number of topics associated with the documents, calculates a Latent Dirichlet Allocation (LDA) score for two- or three-word topic titles in the particular data flow or dataset of input data, and selects the top-scoring candidate topic titles as labels or names for the topics.

8. 7. The method of claim 6, further comprising performing a term frequency-inverse document frequency (TF-IDF) based sentiment analysis and / or a reading grade level assessment on one or more documents in the particular data flow or dataset of input data, wherein the reading grade level indication is incorporated into a document vector describing the one or more documents.

9. 10. The method of claim 6, further comprising providing a user interface within a cloud environment enabling self-service text analytics that allows a user to interact with the data analytics system to apply natural language processing or other text analytics techniques to data flows or data sets of input data and generate visualizations or other types of useful information related to the data.

10. 10. The method of claim 9, wherein the user interface enables access to dataflow action types that enable the user to specify one or more text classification, text transformation, text extraction, document clustering, or other types of dataflow actions that operate on a particular dataflow or data set of input data, including a dataflow of unstructured text or other types of textual data input.

11. A computer program product which, when read and executed by a computer comprising one or more processors, causes said computer to carry out the method according to any one of claims 6 to 10.

Images