User interface commands for generating regular expressions

Abstract

Techniques for generating regular expressions are disclosed. In some embodiments, a regular expression generator can receive input data including one or more character sequences. The regular expression generator can convert the character sequences into a set of regular expression codes and / or span data structures. The regular expression generator can identify a longest common subsequence shared by the set of regular expression codes and / or spans and generate a regular expression based on the longest common subsequence. The generation of the regular expression can be achieved on an interactive user interface. A command can be applied to the one or more character sequences, and a regular expression is generated based on the applied command.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation-in-part of U.S. patent application Ser. No. 16 / 438,327, filed June 11, 2019, and entitled "USER INTERFACE FOR REGULAR EXPRESSION GENERATION," and claims priority under 35 U.S.C. §119(e) to U.S. Provisional Patent Application No. 62 / 684,498, filed June 13, 2018, and entitled "AUTOMATED GENERATION OF REGULAR EXPRESSIONS," and claims priority under 35 U.S.C. §119(e) to U.S. Provisional Patent Application No. 62 / 749,001, filed October 22, 2018, and entitled "AUTOMATED GENERATION OF REGULAR EXPRESSIONS." This application also claims priority under 35 U.S.C. §119(e) to U.S. Provisional Patent Application No. 62 / 865,797, entitled "AUTOMATED GENERATION OF REGULAR EXPRESSIONS," filed June 24, 2019, the entire contents of which are incorporated herein by reference for all purposes. [Background technology]

[0002] background Big data analytics systems can be used for predictive analytics, user behavior analytics, and other advanced data analytics. However, before any data analytics can be effectively performed to provide useful results, the initial data set may need to be formatted into a clean and curated data set. This data onboarding often presents a challenge for cloud-based data repositories as well as other big data systems where data from a variety of different data sources and / or data streams may be compiled into a single data repository. Such data may include structured data in multiple different formats, semi-structured data following different data models, and even unstructured data. Such repositories of data often include data representations in a variety of different formats and structures, and may also include duplicate and erroneous data. When these data repositories are analyzed for reporting, predictive modeling, and other analytical tasks, the low signal-to-noise ratio of the initial data set may lead to inaccurate or unuseful results.

[0003] Many current solutions to the problem of data formatting and preprocessing involve manual and ad-hoc processes to cleanse and curate data in order to manipulate the data into a common format before performing data analysis. While these manual processes may be effective for certain smaller data sets, such processes may be inefficient and impractical when attempting to preprocess and format large data sets. Summary of the Invention [Means for solving the problem]

[0004] overview Aspects described herein provide various techniques for generating regular expressions. As used herein, a "regular expression" may refer to a sequence of characters that defines a pattern that may be used to search for a match within a longer input text string. In some embodiments, a regular expression may be constructed using a wildcard matching language of symbols, and the pattern defined by the regular expression may be used to match character strings and / or extract information from character strings provided as input. In various embodiments described herein, a regular expression generator implemented as a data processing system may be used to receive and display input text data, receive a selection of a particular character subset of the input text via a client user interface, and then generate one or more regular expressions based on the selected character subset. After generating one or more regular expressions, a regular expression engine may be used to match the patterns of the regular expressions against one or more data sets. In various embodiments, the data that matches the regular expression may be extracted, reformatted, modified, etc. In some cases, additional columns, tables, or other data sets may be created based on the data that matches the regular expression.

[0005] According to some aspects described herein, a regular expression generator implemented via a data processing system can generate a regular expression based on a determined longest common subsequence (LCS) shared by different sets of one or more regular expression codes. The regular expression codes (which may also be referred to as category codes) may include, for example, L for letters of the English alphabet, N for numbers, Z for spaces, P for punctuation marks, and S for other symbols. Each set of one or more regular expression codes may be converted from a different sequence of one or more characters received as input data via a user interface. Regular expression codes that are excluded from an LCS may be represented as optional and / or alternative. In some embodiments, a regular expression code may be associated with a minimum number of occurrences of the regular expression code. Additionally or alternatively, a regular expression code may be associated with a maximum number of occurrences of the regular expression code. For example, a set of category codes may include L<0,1> to indicate that a particular portion of the LCS contains a character at most once, if at all. As described in more detail below, generalizing input data as intermediate regular expression code (IREC) can provide a variety of technical advantages, including using very little input data, which enables near-instantaneous generation of regular expressions that do not succumb to false positive or false negative matches in unseen data.

[0006] According to additional aspects described herein, a regular expression may be generated based on input data that includes three or more character sequences. When three or more character sequences are identified as input data, a regular expression generator that identifies the LCS of the character sequences may result in an exponential increase in runtime. To identify the LCS of all character sequences in a fully functional manner, the regular expression generator may perform an LCS algorithm for each distinct combination of two character sequences. Based on the results of the LCS algorithm, a fully connected graph may be generated, where each graph node represents a different character sequence, and the length of each graph edge corresponds to the LCS of the node that defines the graph edge. The order in which to select the character sequences may then be determined by performing a depth-first traversal of a minimum spanning tree on the fully connected graph.

[0007] Further aspects described herein relate to generating regular expressions based on inputs that include both positive and negative character sequence examples. A positive example may refer to a sequence of characters that match the regular expression to be generated, and a negative example may refer to a sequence of characters that do not match the regular expression to be generated. In some embodiments, when both positive and negative examples are received, the regular expression generator may identify a discriminator, i.e., a shortest subsequence of one or more characters that distinguishes the positive examples from the negative examples. The selected discriminator may be the shortest sequence (e.g., expressed in a category code) and may be either positive or negative, such that the positive examples will match and the negative examples will not match. The discriminator may then be hard-coded into the regular expression generated by the regular expression generator. In some cases, the shortest subsequence may be included in the prefix or suffix portion of the negative examples.

[0008] Further aspects described herein relate to one or more user interfaces to which input data may be provided to generate a regular expression. In some embodiments, the user interface may be displayed on a client device communicatively coupled to the regular expression generation server. The user interface may be generated by the server, by the client device, or programmatically by a combination of software components executed at the server and the client. The input data received via the user interface may correspond to a user selection of one or more character sequences that may represent positive or negative cases. In some cases, the user interface may support input data that includes a selection of a first character sequence within a second character sequence. For example, a user may highlight one or more characters within a larger, previously highlighted character sequence, and the second user selection may provide a context for the larger, first user selection. This allows the input data to be provided to the regular expression generator with greater specificity, providing "context" to the regular expression generator to enable it to generate a regular expression that avoids false positives. In response to a user selecting a character sequence via the user interface, the regular expression generator may generate and display a regular expression. For example, when a user highlights a first sequence of characters, the regular expression generator may generate and display a regular expression that matches the first sequence of characters, as well as other similar character sequences (e.g., consistent with the user's intent for matching sequences). When the user highlights a second sequence of characters, the regular expression generator may generate an updated regular expression that encompasses both the first sequence of characters and the second sequence of characters. Then, when the user highlights a third sequence of characters (e.g., within either the first or second sequence), the regular expression generator may update the regular expression again, and so on.

[0009] According to additional aspects described herein, regular expressions may be generated based on a longest common subsequence from one or more input sequence examples, but may also handle characters that are present in only some of the examples. To handle characters that are present in only some of the input examples, a span may be defined in which both the minimum and maximum occurrences of a regular expression code are tracked. If a span may not be present in all of the given input examples, the minimum number of occurrences may be set to zero. These minimum and maximum numbers may then be mapped to a regular expression multiplicity syntax. A longest common subsequence (LCS) algorithm may be run on spans of characters derived from the input examples, including "optional" spans (e.g., a minimum length of zero) that do not appear in all input examples. As described below, consecutive spans may be merged during the execution of the LCS algorithm. In such cases, when additional optional spans that are carried together end up appearing consecutively, the LCS algorithm may be run recursively on those optional spans as well.

[0010] Further aspects described herein relate to combinatorial searches in which the LCS algorithm performed by the regular expression generator may be run multiple times to generate a "correct" regular expression (e.g., a regular expression that properly matches all given positive examples and properly excludes all given negative examples) and / or may generate multiple correct regular expressions from which a most desirable or optimal regular expression may be selected. In some embodiments, the LCS algorithm may generally be run from right to left on the example input to generate a regular expression. However, for comparison purposes and to find alternative regular expressions, the LCS algorithm may be separately run in the reverse direction (e.g., in a left to right direction) on the example input. For example, example character sequences received as user input may be reversed before they are passed through the LCS algorithm, and the results from the LCS algorithm may then be returned reversed (including the original text fragment). Additionally, in some embodiments, the LCS algorithm may be run multiple times by the regular expression generator, both in normal character sequence order and in reverse order, with positioning at the beginning of a line, with positioning fixed at the end of a line, and without positioning at the beginning or end of a line. Thus, in some cases, the LCS algorithm may be run at least six times and the shortest successful regular expression may be selected from these runs. [Brief description of the drawings]

[0011] [Figure 1] FIG. 2 is a block diagram illustrating components of an exemplary distributed system for generating regular expressions in which various embodiments may be implemented. [Diagram 2] 1 is a flowchart illustrating a process for generating a regular expression based on input received via a user interface according to one or more embodiments described herein. [Diagram 3]1 is a flowchart illustrating a process for generating regular expressions using a Longest Common Subsequence (LCS) algorithm on a set of regular expression codes, in accordance with one or more embodiments described herein. [Figure 4] FIG. 1 is an illustrative diagram for generating a regular expression based on two example character sequences using the Longest Common Subsequence (LCS) algorithm on a set of regular expression codes, in accordance with one or more embodiments described herein. [Diagram 5] 1 is a flowchart illustrating a process for generating regular expressions using a Longest Common Subsequence (LCS) algorithm on a larger set of regular expression codes, according to one or more embodiments described herein. [Figure 6] FIG. 1 is an exemplary diagram for generating a regular expression based on five example character sequences using the Longest Common Subsequence (LCS) algorithm on a set of regular expression codes, in accordance with one or more embodiments described herein. [Figure 7] 1 is a flowchart illustrating a process for determining an order of execution for a Longest Common Subsequence (LCS) algorithm on a larger set of regular expression code, according to one or more embodiments described herein. [Figure 8A] 1 illustrates a fully connected graph used to determine the order of execution for a Longest Common Subsequence (LCS) algorithm on a larger set of regular expression code, according to one or more embodiments described herein. [Figure 8B] 1 illustrates a minimum spanning tree representation of a fully connected graph used to determine the order of execution for a Longest Common Subsequence (LCS) algorithm on a larger set of regular expression codes, in accordance with one or more embodiments described herein. [Figure 9] 1 is a flowchart illustrating a process for generating a regular expression based on example positive and negative character sequences according to one or more embodiments described herein. [Figure 10A] 1 is an example user interface screen illustrating generation of a regular expression based on an example positive character sequence, according to one or more embodiments described herein. [Figure 10B] 1 is an example user interface screen illustrating generation of a regular expression based on example positive and negative character sequences according to one or more embodiments described herein. [Figure 11] 1 is a flowchart illustrating a process for generating a regular expression based on a user data selection received in a user interface according to one or more embodiments described herein. [Figure 12] 1 is a flowchart illustrating a process for generating regular expressions and extracting data based on capturing groups via user data selections received in a user interface according to one or more embodiments described herein. [Figure 13] 1 is an exemplary user interface screen illustrating a tabular data display according to one or more embodiments described herein. [Figure 14] 1 is an example user interface screen illustrating generation of regular expressions and capturing groups based on selection of data from a tabular display, according to one or more embodiments described herein. [Figure 15] 1 is an example user interface screen illustrating generation of regular expressions and capturing groups based on selection of data from a tabular display, according to one or more embodiments described herein. [Figure 16A] 1 is an exemplary user interface screen illustrating generation of a regular expression based on selection of positive and negative examples from a table-style display, according to one or more embodiments described herein. [Figure 16B] 1 is an exemplary user interface screen illustrating generation of a regular expression based on selection of positive and negative examples from a table-style display, according to one or more embodiments described herein. [Figure 17] 11 is another example user interface screen illustrating generation of regular expressions and capturing groups based on selection of data from a tabular display, according to one or more embodiments described herein. [Figure 18] 1 is a flowchart illustrating a process for generating regular expressions, including optional spans, using a Longest Common Subsequence (LCS) algorithm in accordance with one or more embodiments described herein. [Figure 19] FIG. 1 is an exemplary diagram for generating a regular expression, including optional spans, using the Longest Common Subsequence (LCS) algorithm in accordance with one or more embodiments described herein. [Figure 20] 1 is a flowchart illustrating a process for generating regular expressions based on a combinatorial implementation of a Longest Common Subsequence (LCS) algorithm, according to one or more embodiments described herein. [Figure 21] FIG. 1 is a block diagram illustrating components of an exemplary distributed system in which various embodiments of the present invention may be implemented. [Figure 22] FIG. 1 is a block diagram showing components of a system environment in which services provided by an embodiment of the present invention can be provided as cloud services. [Figure 23] FIG. 1 is a block diagram illustrating an exemplary computer system upon which embodiments of the present invention may be implemented. [Figure 24] 1 illustrates a regular expression generator, according to some example embodiments. [Figure 25A] 1 illustrates a user interface for implementing a split command, according to some example embodiments. [Figure 25B] 1 illustrates a user interface for implementing a split command, according to some example embodiments. [Figure 26] 1 illustrates a user interface that displays the results of a split command on a dataset, according to some example embodiments. [Figure 27]1 illustrates a flowchart of a method for executing a split command, according to some example embodiments. [Figure 28] 1 illustrates a user interface for implementing a delete command, according to some example embodiments. [Figure 29] 1 illustrates a user interface that displays the results of a delete command on a dataset, according to some example embodiments. [Diagram 30] 1 illustrates a flowchart of a method for executing a delete command, according to some example embodiments. [Diagram 31] 1 illustrates a user interface for implementing an obfuscation command, according to some example embodiments. [Diagram 32] 1 illustrates a user interface that displays the results of an obfuscation command on a dataset, according to some example embodiments. [Diagram 33] 1 illustrates a flowchart of a method for executing an obfuscated command, according to some example embodiments. [Diagram 34] 1 illustrates a user interface for implementing a replace command, according to some example embodiments. [Diagram 35] 1 illustrates a user interface for implementing a replace command, according to some example embodiments. [Diagram 36] 1 illustrates a user interface that displays the results of a replace command on a dataset, according to some example embodiments. [Figure 37] 1 illustrates a flowchart of a method for executing a replace command, according to some example embodiments. [Figure 38] 1 illustrates a user interface for implementing row filtering commands according to some exemplary embodiments. [Figure 39] 1 illustrates a user interface for implementing row filtering commands according to some exemplary embodiments. [Diagram 40]1 illustrates a user interface that displays the results of a row filtering command on a dataset, according to some exemplary embodiments. [Diagram 41] 1 illustrates a flowchart of a method for executing row filtering commands according to some example embodiments. [Diagram 42] 1 illustrates a user interface displaying a view of a dataset in a single-level mode, according to some exemplary embodiments. [Diagram 43] 1 illustrates a user interface displaying highlighted data in nested full control mode, according to some exemplary embodiments. [Diagram 44] 1 illustrates a user interface displaying highlighted data in nested full control mode, according to some exemplary embodiments. [Diagram 45] 1 illustrates a user interface for providing examples, according to some exemplary embodiments. [Figure 46] 1 illustrates a user interface displaying an updated generated regex, according to some exemplary embodiments. [Figure 47] 1 illustrates a user interface displaying alternative data highlighting, according to some example embodiments. [Figure 48] 1 illustrates a user interface displaying an updated generated regex, according to some exemplary embodiments. [Figure 49] 1 is a flowchart of a method for performing multiple highlighting, according to some example embodiments. [Figure 50] 1 illustrates a flowchart of a method for providing negative examples, according to some example embodiments. [Figure 51] 1 illustrates a flowchart of a method for determining context from negative examples, according to some example embodiments. [Figure 52] 1 illustrates a diagram for generating regular expressions based on span highlight alignment, according to some exemplary embodiments. [Figure 53]1 illustrates a flowchart of a method for performing span highlight alignment, according to some exemplary embodiments. [Figure 54] 1 illustrates a flowchart of a method for tracking a span, according to some example embodiments. [Figure 55] 1 illustrates a user interface displaying punctuation spans and symbol spans according to some exemplary embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Detailed Description In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of various embodiments of the present invention. However, it will be apparent to one skilled in the art that embodiments of the present invention may be practiced without some of these specific details. In other instances, well-known structures and devices are shown in block diagram form.

[0013] The following description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments will provide those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes can be made in the function and arrangement of elements without departing from the spirit and scope of the invention as set forth in the claims.

[0014] Specific details are provided in the following description to provide a thorough understanding of the embodiments. However, it will be understood by those skilled in the art that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other examples, well-known circuits, processes, algorithms, structures and techniques may be shown without unnecessary detail to avoid obscuring the embodiments.

[0015] It is further noted that the particular embodiments may be described as a process which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. Additionally, the order of operations may be rearranged. A process is terminated when the operations are completed, but may include additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or to the main function.

[0016] The term "computer-readable medium" includes, but is not limited to, non-transitory media, such as portable or fixed storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. A code segment or computer-executable instructions may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means, including memory sharing, message passing, token passing, network transmission, etc.

[0017] Furthermore, the embodiments may be implemented in hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored on a machine-readable medium. A processor may perform the necessary tasks.

[0018] Described herein are various techniques (e.g., methods, systems, non-transitory computer-readable storage memories storing instructions executable by one or more processors, etc.) for generating regular expressions corresponding to patterns identified in one or more example input data. In certain embodiments, in response to receiving a selection of input data, one or more patterns in the input data are automatically identified, and regular expressions (or "regex" for short) may be automatically and efficiently generated to represent the identified patterns. Such patterns may be based on sequences of characters (e.g., sequences of letters, numbers, spaces, punctuation marks, symbols, etc.). Described herein are various embodiments including methods, systems, non-transitory computer-readable storage media storing programs, codes, or instructions executable by one or more processors, etc.

[0019] In some embodiments, regular expressions may be constructed using a symbolic wildcard matching language to match character strings and / or extract information from character strings provided as input.

[0020]

number

[0021] The input data received by the regular expression generation system may include, for example, one or more "positive" data examples and / or one or more "negative" data examples. As used herein, a positive example may refer to a character sequence received as input that will be matched by a regular expression generated based on that input. Meanwhile, a negative example may refer to an inputted character sequence that will not be matched by a regular expression generated based on that input.

[0022] Several technical advantages may be realized within the various embodiments and examples described herein. For example, some techniques described in this disclosure may improve the speed and efficiency of the regular expression generation process (e.g., regeX solutions may be generated in less than a second and user interfaces may be suitable for interactive real-time use). Various techniques described herein may also be decision-based, may not require training data, may generate solutions without requiring any initial regular expression input, and may be fully automated (e.g., generate regular expressions to the extent that they require no human intervention). Furthermore, various techniques described herein need not be limited with respect to the type of data input that can be effectively processed, and such techniques may improve the human readability of the resulting regular expressions.

[0023] Some embodiments described herein include one or more implementations of a Longest Common Subsequence (LCS) algorithm. The LCS algorithm may be used in some circumstances as a differencing engine (e.g., the engine behind the Unix “diff” utility) configured to determine and indicate differences between two text files. In some embodiments, input data (e.g., strings and other character sequences) may be converted into abstract tokens, which may then be provided as input to the LCS algorithm. Such abstract tokens may be, for example, tokens based on regular expression codes (e.g., Loogle codes or other character class codes) that represent regular expression character classes. Various different examples of such codes are contemplated and may be referred to herein as “regular expression codes” or “intermediate regular expression codes” (IREC). For example, the input character sequence “May 3” may be converted into the IREC code “LLLZN”, and the tokenized string may then be subjected to the LCS algorithm along with other tokenized strings. In some embodiments, IRECs (e.g., regular expression codes) that the input character sequences do not have in common may appear as options (e.g., optional spans) in the final generated regular expression. In particular embodiments, the regular expression codes may be category codes based on the Unicode category codes shown at https: / / www.regular-expressions.info / unicode.html#category, or may be general category property codes at http: / / unicode.org / reports / tr18 / #General_Category_Property. For example, code L may represent a letter, code N may represent a digit, code Z may represent a space, code S may represent a symbol, code P may represent a punctuation mark, etc.

[0024]

number

[0025] In addition, these different categories may be disjoint, or mutually exclusive, i.e., in this example, categories L, N, Z, P, and S may be made disjoint such that there is no overlap between members of the categories.

[0026] Further technical advantages may be realized in various embodiments including more efficient generation of regular expressions based on the use of regular expression codes (e.g., category codes), spans, etc. By using such code, computational resources need not be wasted if the LCS algorithm successfully identifies all or substantially all of the characters in the input string as different. Further technical advantages provided by various embodiments herein include improved readability of the generated regular expressions, as well as supporting both positive and negative examples as input data, and providing various advantageous user interface features (e.g., allowing a user to highlight a text fragment within a larger character sequence or data cell for extraction).

[0027] I. Overall Overview Various embodiments disclosed herein relate to generating regular expressions. In some embodiments, a data processing system configured as a regular expression generator can generate regular expressions by identifying a longest common subsequence (LCS) shared by different sets of regular expression codes (e.g., category codes). Each set of regular expression codes can be transformed from a sequence of characters received as input data via a user interface. Among the technical advantages described herein, abstracting the input data as intermediate code (e.g., regular expression codes, spans, etc.) can efficiently generate regular expressions using very little input data.

[0028] FIG. 1 is a block diagram illustrating components of an exemplary distributed system for generating regular expressions in which various embodiments may be implemented. As shown in this example, client device 120 may communicate with regular expression generation server 110 (or regular expression generator) and interact with a user interface to retrieve and display tabular data and generate regular expressions based on selection of input data (e.g., examples) via the user interface. In some embodiments, client device 120 may communicate with regular expression generator 110 via a client web browser 121 and / or a client-side regular expression application 122 (e.g., a client-side application that receives / consumes regular expressions generated by server 110). Within regular expression generator 110, requests from client device 120 may be received at a network interface over various communication networks and processed by an application programming interface (API), such as REST API 112. User interface data model generator 114 component, along with regular expression generator 110, may provide server-side programming components and logic to generate and render various user interface features described herein. Such features may include functionality that allows a user to retrieve and display tabular data from the data repository 130, select example input data to initiate generation of regular expressions, and modify and / or extract data based on the generated regular expressions. In this example, the regular expression generator component 116 may be implemented to generate regular expressions, including converting input character sequences into regular expression codes and / or spans, running algorithms (e.g., the LCS algorithm) on the input data, and generating / simplifying regular expressions.The regular expressions generated by the regular expression generator 116 may be sent by the REST service 112 to the client device 120, where Javascript code on the client browser 121 (or corresponding client-side application component 122) can then apply the regular expressions to all cells in the spreadsheet column rendered in the browser. In other cases, a separate regular expression engine component may be implemented on the server side to compare the generated regular expressions in the tabular data displayed on the user interface and / or other data stored in the data repository 130 to identify matching / non-matching data on the server side. In various embodiments, the matching / non-matching data may be automatically selected (e.g., highlighted) in the user interface and selected for extraction, modification, deletion, etc. Based on the generation of the regular expressions, any data extracted or modified via the user interface may be stored in one or more data repositories 130. Additionally, in some embodiments, the generated regular expressions (and / or the corresponding inputs to the LCS algorithm) may be stored in the regular expression library 135 for future retrieval and use. In some embodiments, the generated regular expressions need not actually be stored in a "library" but may be incorporated into a "transformation script." For example, such a transformation script may include programs, codes, or instructions that may be executable by one or more processing units to transform received data, as described in more detail in ET.S. Patent No. 10,210,246, which is incorporated by reference herein for all purposes. Other possible examples of transformation scripts may include "rename string," "big string data," or "guess gender from first name and create new column with gender," etc.

[0029] FIG. 2 is a flow chart illustrating a process 200 for generating a regular expression based on input received via a user interface, according to one or more embodiments described herein. In step 201, the regular expression generator 110 can receive a request from the client device 120 to access a regular expression generator user interface and view specific data via the user interface. The request in step 201 may be received via the REST API 112, and / or a web server, an authentication server, etc., and the user's request may be parsed and authenticated. For example, a user within a business or organization can access the regular expression generator 110 to analyze and / or modify transactional data, customer data, performance data, forecast data, and / or any other category of data that may be stored in the organization's data repository 130. In step 202, the regular expression generator 110 can retrieve and display the requested data via a user interface that supports the generation of a regular expression based on selected input data. Various embodiments and examples of such user interfaces are described in detail below.

[0030] In step 203, the user may select one or more input character sequences from data displayed in a user interface provided by the regular expression generator 110. In some embodiments, the data may be displayed in a tabular format in the user interface, including labeled columns with particular data types and / or categories of data. In such cases, the selection of input data in step 203 may correspond to the user selecting a data cell or selecting (e.g., highlighting) an individual text fragment within a data cell. However, in other embodiments, the regular expression generator 110 may support searching and displaying semi-structured and unstructured data via a user interface, and the user may select input data for regular expression generation by selecting a character sequence from the semi-structured or unstructured data. As will be described in the examples below, the user selecting an input character sequence from displayed tabular data is just one example of a use case. In other examples, the user (e.g., perhaps a software developer or power user attempting to build regular expressions for Linux command line tools such as grep, sed, or awk) may type in an example from scratch rather than picking an example from a spreadsheet.

[0031] In step 204, the regular expression generator 110 may generate one or more regular expressions based on the input data selected by the user in step 203. In step 205, the regular expression generator 110 may update the user interface, for example, to display the generated regular expressions and / or to highlight matching / non-matching data in the displayed data. In step 206, which may be optional in some embodiments, the user interface may support functionality that allows the user to modify the underlying data based on the generated regular expressions. For example, the user interface may support features that allow the user to filter, modify, remove, or extract specific data fields from the tabular data based on whether those fields match a regular expression. Filtering or modifying the data may include modifying the underlying data stored in the repository 130, and in some cases, the extracted data may be stored in the repository 130 as new columns and / or new tables.

[0032] While these steps provide a general, high-level overview of an exemplary user interaction with the user interface of regular expression generator 110, other embodiments may support a variety of additional features and functionality. For example, in some embodiments, a regular expression code (or category code) may be associated with a minimum number of occurrences of the code. Additionally or alternatively, a regular expression code may be associated with a maximum number of occurrences of the code. As an example, a set of regular expression codes may include the code L<0,1>, which indicates that a particular portion of an LCS contains a character either at least 0 times and at most 1 time.

[0033] Further, in some embodiments, the input data may include more than two character sequences. In such embodiments, various techniques may be used to determine an order for running the LCS algorithm on more than two character sequences such that the resulting regular expressions may be generated in a fully functional manner, avoiding the exponential increase in runtime caused by more than two input character sequences. Note that the regular expression generator 110 may instead run the LCS algorithm on two character sequences at a time and determine an order for selecting pairs of character sequences based on the graph. For example, a fully connected graph may indicate that a first run of the LCS algorithm (e.g., LCS1) should be run on sequence 1 and sequence 3, then a second run of the LCS algorithm (e.g., LCS2) should be run on LCS1 and sequence 2, etc. The graph may be a fully connected graph, where the nodes represent the character sequences and the edges connect the nodes to represent the length of the LCS shared by the connected nodes. Each node in the graph may be connected to every other node in the graph, and the order for selecting the character sequences may be determined by performing a depth-first traversal of a minimum spanning tree on the graph.

[0034] In further embodiments, the input data may be provided via the user interface in a number of different ways. For example, the input data may indicate a first user selection of one or more characters within a second user selection of a set of characters. For example, the user may highlight a character within a previously highlighted set of characters. Thus, the second user selection may provide a context for the first user selection, which may allow the input data to be provided to the regular expression generator 110 with greater specificity. In some embodiments, the regular expression generator 110 may generate and display regular expressions in near real-time in response to each user selection. For example, if the user highlights a first range of characters, the regular expression generator 110 may display a regular expression representing the first range of characters. And, if the user highlights a second range of characters within the first range of characters, the regular expression generator 110 may update the displayed regular expression.

[0035] Further, in some embodiments, the regular expression generator 110 can generate a regular expression based on an input that includes both positive and negative examples. As described above, a positive example may refer to a sequence of characters that should be included in the regular expression, and a negative example may refer to a sequence of characters that should not be included in the regular expression. In such cases, the regular expression generator 110 can identify a shortest subsequence of one or more characters that distinguishes a positive example from a negative example at a particular position. The shortest subsequence can then be hard-coded in the regular expression generated by the regular expression generator 110. In various examples, the shortest subsequence can be included in a prefix / suffix portion or a midspan within a negative example.

[0036] Further examples of automatically generating regular expressions according to certain embodiments are described below. These examples may correspond to various specific possible implementations of the general technique of FIG. 2 and may be implemented in software (e.g., code, instructions, programs, etc.) executed by one or more processing units (e.g., processors, cores) of the respective systems, hardware, or a combination thereof. The software may be stored on a non-transitory storage medium (e.g., on a memory device). The further examples described below are intended to be illustrative and non-limiting. These examples show various processing steps occurring in a particular order or sequence, but this is not intended to be limiting. In some alternative embodiments, the steps may be performed in some different order, or some steps may be performed in parallel.

[0037] In some examples, user input received via a user interface (e.g., step 203) may include one or more "positive examples" that are matched by the regular expression output, and zero or more "negative examples" that are not matched by the regular expression output. Optionally, one or more of the positive examples may be highlighted to select a particular range (or subsequence) of characters. In some cases, in step 204, the positive examples received via the user interface may be converted to spans of regular expression code (e.g., character category codes such as Unicode category codes). For each positive example, a sequence of spans may be generated. In some embodiments, a graph may be generated in which each vertex corresponds to one of the sequences of spans, and the edge weight is equal to the length of the output from an LCS algorithm run on those two sequences of spans that correspond to the edge endpoints. A minimum spanning tree may be determined for the graph. For example, in some embodiments, Prim's algorithm may be used to obtain a minimum spanning tree. A depth-first traversal may be performed on the minimum spanning tree to determine the traversal order, and then the LCS algorithm may be run on the first two elements of the traversal. One by one, each additional element of the traverse may then be merged in turn into the current LCS output by running the LCS algorithm again on the output of the previous LCS iteration and the next current traverse element. The final output of the LCS algorithm, which may be a sequence of spans, may then be converted into a regular expression. This conversion may be a one-to-one conversion in some embodiments, although certain optional embodiments described herein may not correspond to a one-to-one conversion. Finally, the resulting regular expression may be tested against all positive and negative examples received via the user interface in step 203. If any of the tests fail, the above process may be repeated using all positive examples and any negative examples that failed.

[0038] II. Regular Expression Generation Using the Longest Common Subsequence Algorithm on Regular Expression Code As mentioned above, some aspects described herein relate to generating regular expressions based on computing a longest common subsequence (LCS) shared by different sets of regular expression codes corresponding to input data.

[0039] 3 is a flow chart illustrating a process 300 for generating a regular expression using the LCS algorithm for a set of regular expression codes, according to one or more embodiments described herein. In step 301, the regular expression generator 110 may receive one or more character sequences as input data. As mentioned above, in some examples, the input data may correspond to positive case data selected from within tabular data displayed in a user interface, although it should be understood that in some embodiments, the user interface is optional and the input data may correspond to any character sequence received via any other communication channel (e.g., a non-user interface) in various examples.

[0040] In step 302, each character sequence received in step 301 may be converted to a corresponding regular expression code. In various embodiments, the regular expression code may be a Loogle code, a Unicode category code, or any other character class code that represents a regular expression character class. For example, an input character sequence "May 3" may be converted to the Loogle code "LLLZN". In some embodiments, the regular expression code may be a category code based on the Unicode category codes shown at https: / / www.regular-expressions.info / unicode.html#category. For example, code L may represent a letter, code N may represent a number, code Z may represent a space, code S may represent a symbol, code P may represent a punctuation mark, etc.

[0041]

number

[0042] In step 303, a longest common subsequence may be determined from among the set of regular expression codes generated in step 302. In some embodiments, an LCS algorithm may be executed using the two sets of regular expression codes as input. Various different characteristics of the execution of the LCS algorithm (e.g., direction of processing, positioning, pushing whitespace, coalescing low-cardinality spans, aligning on common tokens, etc.) may be used in different embodiments. In step 304, a regular expression may be generated based on the output of the LCS algorithm. In some cases, step 304 may include capturing the output of the LCS algorithm with the regular expression code and converting the regular expression code to a regular expression. In step 305, the regular expression may be simplified and output, for example, by displaying the regular expression to a user via a user interface.

[0043] FIG. 4 is an exemplary diagram for generating a regular expression using a Longest Common Subsequence (LCS) algorithm on a set of regular expression codes based on two example character sequences. Thus, FIG. 4 illustrates an example of applying the process described above in FIG. 3. As shown in FIG. 4, a regular expression in this example is generated based on two input strings, "iPhone 5" and "iPhone X". Each sequence in this example may be converted to a respective set of regular expression codes. Thus, iPhone 5 may be converted to "LLLLLLZN" and iPhone X may be converted to "LLLLLLZL". As shown in FIG. 4, these category codes are then provided as inputs to the LCS algorithm, which determines that both sets of IRECs (or category codes) contain six Ls and one Z. The Z category code that is excluded from the LCS may be represented as optional and / or alternative.

[0044]

number

[0045] The braces containing the number 6 indicate six instances of a character, and the question mark indicates that the last number / character is optional. Finally, a simplification process can be performed by the regular expression generator, during which the regular expression is simplified by inserting the common text fragment "iPhone" back into the final regular expression, resulting in a broader

[0046]

number

[0047] Replace the part. As shown in this example, the input string received by the regular expression generator 110 may be converted into "regular expression codes" that represent regular expression broad categories (which may also be referred to as "category codes"), and the LCS algorithm may be run on those regular expression codes. In some embodiments, Unicode category codes may be used for the regular expression codes. For example, the input text string may be converted into "regular expression codes" that represent regular expression broad categories (which may also be referred to as "category codes"), and the LCS algorithm may be run on those regular expression codes.

[0048]

number

[0049] 3 and 4 may be referred to as an indirect approach. However, in other embodiments, a direct approach may be used in which the LCS algorithm is performed directly on the character sequence received as input.

[0050] In some embodiments, the indirect approach may provide an additional technical advantage in that it does not require a large amount of training data, and may generate valid regular expressions with a relatively smaller number of input examples. This is because the indirect approach uses heuristics to reduce uncertainty in regular expression generation and to eliminate potential false positives and false negatives. For example, in generating a regular expression based on the input strings "May 3" and "Apr 11", a direct approach may require at least one example per month to generate a valid regular expression that matches the date pattern. Relying only on those two examples, the direct approach may generate a regeex of "[AM][ap][yr]

[13] 1?". ​​In contrast, the indirect approach uses heuristics to generate valid regular expressions based on Unicode broad categories.

[0051]

number

[0052] In addition, as noted above, one of the technical advantages described herein includes efficiently generating regular expressions using very little input data, even from a single example. For example, with respect to generating a regular expression from a single example "am", one heuristic may generate "am" for the regular expression,

[0053]

number

[0054] Either is likely to be correct, but programmed heuristics may implement user preferences and / or criteria to determine how to generate the optimal regular expression (e.g., whether it should also match "pm" or not).

[0055] In addition, the indirect approach also uses the generated regular expression

[0056]

number

[0057] can be simplified to make it more human readable, which can be beneficial in some embodiments, for example when outputting to non-sophisticated regular expression users who may not be familiar with the Unicode representation for regular expressions.

[0058] Additionally, in some embodiments, instead of treating each character independently when performing the LCS algorithm, sequential and equal regular expression codes may be converted into a span data structure (which may also be referred to as a span). In some cases, a span may contain a representation of a single regular expression code (e.g., a Unicode broad category code) along with a repeat count range (e.g., minimum and / or maximum). The conversion from regular expression codes to spans may facilitate a number of various additional features described below, such as recognizing alternations (e.g., disjunctions), and may also facilitate the merging of adjacent optional spans to further simplify the generated regular expression.

[0059] As mentioned above, the LCS algorithm can be configured to remember and retain underlying text fragments in the input character sequence that can potentially be inserted back into the final regular expression, such as the string "iPhone" in Figure 4. By tracking the text fragment that originally resulted in the category code assigned to that span, such an embodiment can allow literal text (e.g., am and pm) to be included directly in the generated regular expression, reducing false positives and making the regular expression output more human readable.

[0060] In some embodiments, a heuristic may be used to determine when to output a string literal in the generated regular expression, as opposed to a broadly matching regular expression code. As discussed above, in some cases, it may be desirable for a regular expression to require an exact match for a string fragment. Thus, in some embodiments, a heuristic may be used to determine whether an exact string fragment is output in the regular expression. For example, a heuristic may determine that if only one particular string fragment (e.g., "pm") has ever been encountered for a particular span, and if there are two or more examples for that span, then that exact string fragment is output in the regular expression. In the case of a punctuation span, the heuristic may lower the threshold to only one occurrence for that span (e.g., based on the assumption that punctuation is less likely to change). For example,

[0061]

number

[0062] For both, the threshold may be lowered to only one occurrence to output a literal string in the regular expression. The heuristic may be modified and / or adjusted to be somewhat restrictive based on a threshold number of examples required to output the exact string fragment in the regular expression. Note that if the heuristic errs in being overly restrictive (by outputting literal string fragments in the regular expression too easily), the user may compensate by introducing additional positive examples. Similarly, if the heuristic errs in being overly permissive (by outputting broad regular expression code too often), the user may compensate by introducing additional negative examples. The heuristic can be initially programmed (and subsequently adjusted) with an appropriate amount of restriction based on previous user interactions and experimentation.

[0063] As described above with reference to FIG. 4, a simplification process may be performed by the regular expression generator 110, during which the regular expression may be simplified using various techniques. In some cases, the simplification may include replacing long form regular expression repeat codes (e.g., using curly braces) with regular expression shorthand codes (e.g., *, +, and ?). For example, if a span comes from an example string fragment that represents a repeat between, for example, 0 and 3 (minimum and maximum), the regular expression generator 110 may simplify the expression by outputting * instead of {0,3} in the regular expression to prevent false negatives and improve readability. Furthermore, instead of frequently using the {a,b} syntax, the regular expression generator 110 may instead use the + syntax when a>2 or b>4. This makes the generated regular expression more readable and increases the likelihood of avoiding false negatives.

[0064] In some cases, a bank of predefined character ranges may be used. For example, the regular expression Unicode character category for the character

[0065]

number

[0066] which may not be as well known or as easily legible as older character ranges such as [A-Z]. Thus, for English input, regular expression generator 110 may select [A-Z], [az], and [A-Za-z] as

[0067]

number

[0068] may be substituted with if a given example can be successfully matched with one of those substitutions.

[0069] Additionally, regular expression generator 110 may be configured in some embodiments to default to a maximum tolerance with respect to whitespace in a regular expression. As described herein, regular expression generator 110 may be configured to generate more specific regular expressions, for example, in some cases by preferring literal string fragments over broad matching codes and applying minimum and maximum bounds to the repeat output within curly braces. However, with respect to whitespace, regular expression generator 110 may be configured in some embodiments to default to a maximum tolerance. That is, for any whitespace (e.g., even a single space that appears between words), regular expression generator 110 may map it to a regular expression category code.

[0070]

number

[0071] and may specify repetition to be +, meaning one or more. Thus, such an embodiment may anticipate and match extra whitespace, or even tabs and carriage returns that can be matched exactly (e.g., as in fully unstructured text).

[0072] In some cases, repetitions in the generated regular expression may be expanded into literal repeating regular expression code.

[0073]

number

[0074] For example, the regular expression generator 110 may generate a regular expression for a Unicode category code that is two characters long (e.g.,

[0075]

number

[0076] ), and the repetition count is 4 or less, or the Unicode category code is 3 characters long (e.g.

[0077]

number

[0078] ), and the number of repetitions is 3 or less, or if the length of the Unicode category code is greater than 3 characters and the number of repetitions is 2 or less. Additionally, in some embodiments, regular expression generator 110 may be configured to escape literal characters when they need to be output (e.g., a regular expression "special character").

[0079] In various additional embodiments, the regular expression generator 110 may be configured to detect various types of paired brackets (e.g., { and}) and generate regular expressions with negated character classes (e.g., [^}]*). For example, within text highlighted (or otherwise selected) by a user, or within positive / negative examples or other input to the system, open or closed brackets may be detected and matched or paired with corresponding brackets. In some examples,

[0080]

number

[0081] A number of different types of brackets may be detected in the input text, including some or all of the following: . Thus, as an example, for HTML tags, something like <[az]+(?:[az]+=[az]+> may be generated instead of <[^>]*>. When detecting and pairing different kinds of brackets, regular expression generator 110 may be configured to properly detect and pair brackets nested within brackets, and may be configured to ignore brackets that overlap highlighted (or otherwise user-selected) portions of the input. Also, in some cases, regular expression generator 110 may not use the new negated character class between brackets if the content is unique; i.e., if the content is always all the same (e.g., continuing with the HTML example, all identified tags are <h1> ), the literal may be output in the generated regular expression in place of a negated bracket character class (e.g., instead of <[^>]*>).< / h1> <h1>(only output).

[0082] In an exemplary embodiment, when highlighting is exactly enclosed by parentheses, collapsing within the parentheses may be allowed. This allows for the extraction of parenthetical annotations. For example, to extract Windows NT from (Windows NT), the generated regeex is

[0083]

number

[0084] It would be more general and would yield fewer false negatives. III. Regular Expression Generation Using the Longest Common Subsequence Algorithm on Combinations of Regular Expression Codes Further aspects described herein relate to the generation of regular expressions based on input data that includes three or more strings (e.g., three or more distinct character sequences). When three or more strings are identified as input data, the regular expression generator 110 may use a performance optimization feature in which an optimal order is determined for a sequence of LCS algorithm executions. As described below, the performance optimization function for three or more strings may include constructing a graph with vertices corresponding to each string and edge lengths / weights between each string and every other string that may be based on the size of the LCS output. A minimum spanning tree may then be derived using those edge weights, and a depth-first traversal may be performed to determine the order of the input strings. Finally, a sequence of LCS algorithms may be performed using the determined order of the input strings.

[0085] FIG. 5 is a flow chart illustrating a process 500 for generating regular expressions using a Longest Common Subsequence (LCS) algorithm for a larger set of regular expression codes (e.g., three or more character sequences). Thus, steps 502-505 in this example may correspond to step 303 described above in FIG. 3. However, because this example concerns generating regular expressions based on three or more input character sequences, the LCS algorithm may be run multiple times. For example, to avoid exponential growth in runtime for three or more input strings, the LCS algorithm may be run multiple times, with each run being run for only two input strings. For example, regular expression generator 110 may run a first run of the LCS algorithm on two strings (e.g., two input character sequences or two transformed regular expression codes), then run a second run of the LCS algorithm on the output of the first LCS algorithm and a third string, then run a third run of the LCS algorithm on the output of the second LCS algorithm and a fourth string, and so on.

[0086] To improve and / or optimize the performance of such embodiments, it may be desirable to determine an optimal order for an input string (e.g., an input character sequence or regular expression code) in which to run a sequence of LCS algorithms. For example, a good order for taking the input string may affect the readability of the generated regular expression, such as by minimizing the number of optional spans. To keep the generated regex succinct, it is preferable that additional strings that are LCSed to the current regex already have some similarity to the current regex (the intermediate result from LCSing a string already seen).

[0087] Thus, in step 501, multiple (e.g., three or more) input character sequences are converted into regular expression codes. In step 502, the order in which to process the regular expression codes is determined using the LCS algorithm. The determination of the order in step 502 is described below with reference to FIG. 7. In step 503, the first two regular expression codes in the determined order are selected (for the first iteration of step 503) or the next regular expression code in the determined order is selected (for a subsequent iteration of step 503). In step 504, the LCS algorithm is performed on two input strings corresponding to the format of the regular expression codes. In the first iteration of step 504, the LCS algorithm is performed on the first two regular expression codes in the determined order, and in subsequent iterations of step 504, the LCS algorithm is performed on the next regular expression code in the determined order and the output of the previous LCS algorithm (which may be in the same regular expression code format). In step 505, the regular expression generator 110 determines whether there are additional regular expression codes in the determined order that have not yet been provided as input to the LCS algorithm. If there are such additional regular expression codes in the determined order, processing returns to step 503 for another run of the LCS algorithm, otherwise, in step 506, a regular expression is generated based on the output of the last run of the LCS algorithm.

[0088] FIG. 6 is an exemplary diagram for generating a regular expression based on five example input character sequences. In this example, each input character sequence is converted into a regular expression code, and then the LCS algorithm is run repeatedly based on the determined order of the regular expression codes. Thus, FIG. 6 illustrates an example of applying the process described above in FIG. 5. In this example, the determined order for the five regular expression codes is Code #1 through Code #5, and each code is input to the LCS algorithm in the determined order to generate a regular expression output. The final regular expression output (Reg Ex #4) corresponds to the final regular expression generated based on all five of the input character sequences.

[0089] FIG. 7 is a flow chart illustrating a process for determining an order of execution for a Longest Common Subsequence (LCS) algorithm on a larger set (e.g., three or more) of regular expression codes. Thus, as shown in this example, steps 701-704 may correspond to the order determination in step 502 described above. In step 701, the LCS algorithm may be run on each distinct pair of regular expression codes corresponding to the input data, and the resulting output LCS may be stored for each run. Thus, for k input data, this may represent all (k(k-1)) / 2 possible pairings of strings run through the LCS algorithm, or in some embodiments k(k-1). For example, if an input character sequence of k=3 is received, the LCS algorithm may be run three times in step 701; if an input character sequence of k=4 is received, the LCS algorithm may be run six times in step 701; if an input character sequence of k=5 is received, the LCS algorithm may be run ten times in step 701, and so on. In step 702, a fully connected graph may be constructed from k nodes where the edge weight of (k(k-1)) / 2 edges represents a string with the length of the raw LCS output between two nodes. In step 703, a minimum spanning tree may be derived from the fully connected graph in step 702. In step 704, a depth-first traversal may be performed on the minimum spanning tree. The output of this traversal may correspond to the order in which the regular expression codes are input into a sequence of LCS algorithm executions.

[0090] With brief reference to Figures 8A and 8B, an example of a fully connected graph generated based on a received input character sequence for k=5 is shown in Figure 5, and a minimum spanning tree representation for the fully connected graph is shown in Figure 8B.

[0091] In some embodiments, the approaches described in Figures 5-8B may provide additional technical advantages in terms of performance. For example, certain conventional implementations of the LCS algorithm require O(n 2 ), where n is the length of the string. Extending such an implementation to k strings instead of just 2 leads to exponential runtime performance O(n k ), because the LCS algorithm may be required to search a k-dimensional space. Such traditional implementations of the LCS algorithm may not perform well or may not be well suited for real-time online user experiences.

[0092] As mentioned above, the LCS algorithm may be run (k(k-1)) / 2 times, where sometimes the overlap is exactly the same as seen before, since the LCS algorithm may be the case where raw input examples from the user are converted into Regex category codes. Thus, in some cases, memorization may be implemented, where a cache can be used to map previously seen LCS problems to previously worked LCS solutions.

[0093] An exemplary embodiment also provides an effective way to control the cache size if the cache table becomes too large. Parts of the cache may be removed. For example, a hash map table may be used that can be truncated. The hash map table may be arbitrarily truncated.

[0094] IV. Generating Regular Expressions Based on Positive and Negative Pattern Matching Examples Further aspects described herein relate to generating regular expressions based on input data corresponding to both positive and negative examples. As described above, a positive example may refer to an input data character sequence that is specified as an example string that should match a regular expression generated by the regular expression generator. Meanwhile, a negative example may refer to an input data character sequence that is specified as an example string that should not match a regular expression generated by the regular expression generator. As described below, in some embodiments, the regular expression generator 110 may be configured to identify a position and a shortest subsequence of characters that distinguishes a positive example from a negative example at that position. The shortest subsequence may then be hard-coded into the generated regular expression, such that the positive examples will match the regular expression and the negative examples will be excluded (e.g., not matched) by the regular expression.

[0095] 9 is a flow chart illustrating a process 900 for generating a regular expression based on example positive and example negative character sequences. In step 901, the regular expression generator 110 may receive one or more input data character sequences corresponding to the positive examples. In step 902, the regular expression generator 110 may generate a regular expression based on the received positive examples. Thus, steps 901-902 may include some or all of the steps performed in FIG. 3 or FIG. 5 discussed above to generate a regular expression based on the input data character sequences.

[0096] In step 903, the regular expression generator 110 may receive one additional input data character sequence corresponding to a negative example. Thus, the negative example is specifically designated not to match the regular expression generated in step 902. In some embodiments, the negative example received in step 903 may be first tested against the regular expression generated in step 902, and if it is determined that the negative example does not match the regular expression, no further action is taken. However, in this example, it may be assumed that at least one of the negative examples received in step 903 matches the regular expression generated in step 902. Thus, in step 904, a disambiguation position may be determined within the regular expression generated in step 902. In some embodiments, the disambiguation position may be selected as either a prefix position (e.g., at the beginning of the regular expression) or a suffix position (e.g., at the end of the regular expression). For example, the regular expression generator 110 may determine a first number of characters that would be required in a prefix to distinguish positive cases from negative cases and a second number of characters that would be required in a suffix to distinguish positive cases from negative cases. The regular expression generator 110 may then select a suffix or prefix based on the shortest number of replacement characters required. In some cases, using a prefix as a disambiguation location may be preferred (e.g., weighted) for readability. In yet other examples, the disambiguation location may be a midspan location that does not correspond to a prefix or suffix of the regular expression.

[0097] In step 905, the regular expression generator 110 can determine a replacement sequence of a custom character class (or more specifically, a sequence of a custom "bracket" regular expression character class) that, when inserted into the regular expression at the determined position, can distinguish the positive example from the negative example. In some embodiments, in step 905, the regular expression generator 110 can take a text fragment from each of the positive and negative examples that corresponds to the disambiguation position (or replacement position), and then use the text fragments to determine a discriminator to be used as a replacement sequence that distinguishes the positive example from the negative example. Additionally, the discriminator replacement sequence determined in step 905 can include multiple different replacement sequences of the custom character class, which can be replaced either at the same position or at different positions in the regular expression.

[0098] As mentioned above, in some cases, the determination of the replacement sequence in step 905 may be performed in conjunction with the determination of the disambiguation position (or replacement position) in step 904. For example, the regular expression generator 110 may determine one or more replacement sequences that may distinguish positive cases from negative cases at a first possible replacement position. The regular expression generator 110 may also determine one or more other replacement sequences that may distinguish positive cases from negative cases at a second different possible replacement position. In this example, when selecting between the different possible replacement positions and corresponding replacement sequences, the regular expression generator 110 may apply a heuristic formula to perform the selection based on one or more of the size of the characters at the replacement position and the number and / or size of the corresponding replacement sequences. Finally, in step 906, the regular expression may be modified by inserting one or more of the determined replacement sequences at the determined positions to replace previous portions of the regular expression. Optionally, following modification of the regular expression in step 906, the positive and / or negative examples may be tested against the modified regular expression to verify that the positive examples match the regular expression and that the negative examples do not match the regular expression.

[0099] 10A and 10B are exemplary user interface screens illustrating the generation of a regular expression based on examples of positive and negative character sequences. Thus, the examples shown in FIGS. 10A and 10B may correspond to a user interface displayed during execution of the process of FIG. 9 described above. In FIG. 10A, a user provides three positive examples 1001 of data input character sequences, and regular expression generator 110 generates a regular expression 1002 that matches each of the positive examples. Next, in FIG. 10B, a user provides one negative example 1004, and regular expression generator 110 generates a revised regular expression 1005 that is based on both the current set of positive examples 1003 and the current set of negative examples 1004.

[0100] As noted above, in some embodiments, when both positive and negative examples are received, the regular expression generator 110 may identify a discriminator, i.e., the shortest subsequence of one or more characters that distinguishes positive examples from negative examples. The selected discriminator may be the shortest sequence (e.g., represented by a categorical code), may be either positive or negative, and positive examples will match while negative examples will not. In some cases, the discriminator may then correspond to a substitution subsequence that can be hard-coded into the regular expression at step 905. As an example, in "[AL][a-z]+", [AL] would be a positive discriminator that would match (or recognize) "Alley", "Avenue", and "Lane" assuming it is applied to street suffixes, but would not match (or tolerate) anything else. As another example, in "[BC][o][a-z]+", [BC][o] is a positive discriminator consisting of a sequence of two character classes that would match "Boulevard" and "Court". As yet another example, in "[^A][a-z]+", [LA] could be a negative discriminator that would not tolerate "Alley" and "Avenue". In some cases, the algorithm may generate a negative lookbehind to discriminate correctly. For example, (?<!Av)[A-Za-z]+ would exclude "Avenue" but would recognize "Alley".

[0101] As another example, if the user supplies positive examples "202-456-7800" and "313-678-8900" as well as negative examples "404-765-9876" and "515-987-6570", in one embodiment, the regular expression generator 110 generates the regular expression

[0102]

Number

[0103] That is, replacement character subsequences may be identified based on a determination that for a suffix of a regular expression, phone numbers ending in 00 distinguish positive cases from negative cases (e.g., assuming the goal is a regular expression that matches business phone numbers). While this is an example of a negative example by suffix (or, more specifically, an example of addressing negative cases by using a positive suffix), various other embodiments may support replacement at either the prefix, suffix, or mid-span position. In an example of replacement at a mid-span position, characters that are offset into the span may be tracked and split at the mid-span point.

[0104] To determine whether to use a prefix or a suffix, in some embodiments, k a and a heuristic is used in which the minimum score across all prefix / suffix combinations is selected:

[0105]

number

[0106] k a = number of characters considered to disambiguate the affix (prefix or suffix) |F p |= number of unique text fragments from positive examples needed to disambiguate an affix |F n |= number of unique text fragments from negative examples needed to disambiguate an affix |E p |= Number of (complete) positive cases provided by the user |E n |= number of (complete) negative examples provided by the user In the above example, the heuristic will choose the longer disambiguated text fragment (so, for example, k a The heuristics are designed to favor shorter disambiguating text fragments over prefixes (hence, for example, a penalty of 0.1 for suffixes), to improve readability. Finally, the heuristics are designed to favor disambiguating (e.g., replacing) longer prefixes or suffixes over disambiguating by using a larger number of string fragments (hence, for example, a square of the number of string fragments to be replaced).

[0107] As mentioned above, some embodiments may also support negative midspan cases, as well as negative look-behind and negative look-ahead cases.

[0108] Once the prefix / suffix and k (the number of characters to disambiguate) are determined, the regular expression generator 110 can further determine how to express the disambiguation in the generated regular expression. The generated regular expression may be permissive for affixes (e.g., prefixes or suffixes) that appear to be positive examples, or may exclude affixes that appear to be negative examples.

[0109]

number

[0110] If usePermissive is greater than zero, then what appear to be positive examples are passed through by generating regular expressions, one for each character position, that allow characters taken from the positive examples. In other cases, regular expression generator 110 may take the approach of not allowing what appear to be negative examples by generating regular expressions, one for each character position, that do not allow characters taken from the negative examples.

[0111] As another example, the regular expressions generated for positive example 8am and negative example 9pm may be

[0112]

Number

[0113] This uses the caret syntax. In some cases, the regular expression generator 110 may be configured to prefer shorter regular expressions, which may not only be more readable for the user but also more likely to be correct. The principle is that characters that appear frequently are more likely to appear again in the future, and therefore, emphasis should be placed on frequently appearing characters. When there are fewer unique characters |F p | (since those that appear do so more frequently and have less uniqueness), this is rewarded heuristically by having it in the denominator.

[0114] Referring again to the heuristic of the usePermissive example above, determining one unique positive suffix is not a major feature when there is only one positive example from the user. Therefore, in this heuristic, a low |E p | is penalized by having it in the numerator (i.e., a high |E p | is rewarded in this heuristic).

[0115] Furthermore, in some embodiments, the negative examples may be based on lookbehind and / or lookahead. For example, the user provides a positive example of "323 - 1234" and a negative example of "202 - 754 - 9876", in which case it involves the use of the regex lookbehind syntax (?<!) to exclude phone numbers with area codes.

[0116] In some cases, the negative examples may be based on an optional span. For example, a user may provide positive examples of "ab" and "a2b" and a negative example of "a3b". In this case, an example implementation may fail because it may try to discriminate based only on the required span, the "2" digit being in the optional span. In this example, failure may refer to a situation where the generated regular expression (correctly) matches all of the positive examples and (incorrectly) matches one or more of the negative examples. In such cases, the user may be alerted to the failure and provided with options via a user interface to manually repair the generated regular expression and / or to remove some of the negative examples.

[0117] V. User Interface for Generating Regular Expressions Additional aspects described herein include several different features and functions within a graphical user interface related to the generation of regular expressions. As described below, some of these features may include various options for user selection, as well as highlighting of positive and negative cases, color coding of positive and negative cases, and multiple overlapping / nested highlighting within data cells.

[0118] FIG. 11 is a flow chart illustrating a process 1100 for generating a regular expression based on a user data selection received within a user interface. The exemplary process of FIG. 11 may correspond to any of the previously described examples of generating a regular expression based on an input data character sequence. However, FIG. 11 describes a process for a user interface that may be generated and displayed on a client device 120. In step 1101, in response to a request from a user via the user interface, the regular expression generator 110 may retrieve data (e.g., from the data repository 130) and render / display the data in a tabular format within a graphical user interface. It should be understood that while this example uses tabular data, other examples may not require the use and display of tabular data. For example, in some cases, a user may directly type raw data (rather than selecting data from a user interface). Furthermore, when data is presented via a user interface, the data need not be in a tabular format, but may be unstructured data (e.g., a document) or semi-structured (e.g., a spreadsheet of unformatted / unstructured data items such as tweets or posts). In various examples, the tabular data may correspond to transactional data, customer data, performance data, forecast data, and / or any other category of data that may be stored in the data repository 130 for a business or other organization. At step 1102, a user selection of input data may be received via a user interface. The selected input data may correspond, for example, to an entire data cell or a subsequence of characters within a data cell selected by the user. At step 1103, the regular expression generator 110 may generate a regular expression based on the input data (e.g., a data cell or a portion thereof) received at step 1102. At step 1104, the user interface may be updated in response to the generation of the regular expression.In some cases, the user interface may simply be updated to display the generated regular expression to the user, while in other cases, the user interface may be updated in various other ways, described below. As shown in this example, the user may select multiple different input data character sequences via the user interface, and in response to each new input data received, regular expression generator 110 may generate an updated regular expression that encompasses both the first and second (positive) instances of the character sequence. Then, if the user highlights a third sequence of characters (e.g., outside of both character sequences, or within the first or second character sequences), regular expression generator 110 may update the regular expression again, and so on. In some embodiments, regular expression generator 110 may execute an algorithm in real time (or near real time), and an entirely new regular expression may be generated in response to each new keystroke or each new highlighted section made by the user. Furthermore, if the user highlights (or otherwise selects) a portion of a previous highlight, the old highlight may be removed and a new highlight may be added.

[0119] Thus, as shown in Figure 11, in response to a user selection of a character sequence via the user interface, the regular expression generator 110 may generate and display a regular expression. For example, when a user highlights a first sequence of characters, the regular expression generator may generate and display a regular expression that represents the first sequence of characters. When a user highlights a second sequence of characters, the regular expression generator may generate an updated regular expression that encompasses both the first sequence of characters and the second sequence of characters. Then, when a user highlights a third sequence of characters (e.g., within either the first or second sequence), the regular expression generator may update the regular expression again, and so on.

[0120] FIG. 12 is another flow chart showing a process 1200 for generating regular expressions and extracting data based on capture groups via user data selection received in a user interface. In step 1201, the regular expression generator 110 can retrieve data (e.g., from the data repository 130) and render / display the data in a table format in a graphical user interface, as described above in step 1101. In step 1202, the regular expression generator 110 can receive a selection of a user highlighting a text fragment in a particular data cell. In step 1203, the regular expression generator 110 can generate a regular expression based on the positive examples of the selected data cell, and in step 1204, create a regular expression capture group based on the text fragment highlighted in the cell. In step 1205, the regular expression generator 110 can determine one or more additional cells in the displayed tabular data that match the generated regular expression, and in step 1206, extract the corresponding text fragment in the additional cell that matches the generated regular expression.

[0121] Thus, in addition to providing a positive example, a user may select a text fragment within any of the selected positive examples (e.g., via mouse text highlighting). In response, the regular expression generator 110 may create a regular expression capturing group to extract that text fragment from the example and extract corresponding fragments from all other matches in the text to which the regular expression is applied. Extracting text fragments from matching data cells may also include deletions and modifications, and in some cases may be used to create new columns of data from existing columns of semi-structured or unstructured text.

[0122] Using the example where the user selects a positive data example, if the user highlights a year, the regular expression generator 110 generates the regular expression

[0123]

number

[0124] As shown in this example, regular expression generator 110 has added parentheses around the year and also converted the old parentheses around the month and day (used for alternation) into "non-capturing" groups by use of the ?:regex syntax. In some embodiments, the extraction / capture group may be required to be on a span boundary, and in such embodiments, regular expression generator 110 may take the highlighted character range as input and expand it to encompass the nearest locator span boundary. However, in other examples, mid-span extraction / capture may be supported by the user interface.

[0125] In some embodiments, the user interface may support input data from a user that includes a selection of a first character sequence within a second character sequence. For example, a user may highlight one or more characters within a larger previously highlighted character sequence, and the second user selection may provide a context for the larger first user selection. Such embodiments may enable input data to be provided to the regular expression generator 110 with greater specificity.

[0126] Additionally, in some examples, actions may be initiated and dialogs may be opened in response to a user making a selection (e.g., highlighting text) within the user interface. In some cases, the dialog may be a non-modal dialog, such as a floating toolbox window that does not interfere with user interaction with the main screen. The dialog may also change appearance and / or functionality depending on what primary operation the user is performing. Thus, in such cases, the user does not need to search for further menu items after highlighting the selected text to initiate modifying, extracting, etc., capture group text fragments.

[0127] Further, in certain embodiments, the user interface provided for generating regular expressions may include three highlighting modes: nested auto, nested manual, and single level. Nested auto is also known as nested (auto-outer). Nested manual is also known as nested (full-control). In these examples, single level may refer to one level of highlighting (or other form of text selection), which may cause a regular expression to be generated without capturing groups. Nested manual highlighting mode allows for the identification of two levels of highlighting (or other form of text selection), which may cause a regular expression to be generated with capturing groups. Nested auto highlighting may be identical to nested manual highlighting mode, except that the outer highlighting (or other form of text selection) may be automatically set to be the contents of the entire text (e.g., an entire spreadsheet cell).

[0128] In some cases, the default mode of operation may be for the entire cell to be identified as the highlighted region, and the user may further highlight one or more additional subsequences within the highlighted cell. In other modes, the user may be permitted to manually specify both highlights within a data cell of a tabular data display. In yet other modes, the user may be permitted to manually specify an outer highlight without an inner highlight. These other modes may be more suited to semi-structured data, e.g., columns of data consisting of tweets or other long strings such as browser "user agent" strings. "Semi-structured" data refers to data that may be displayed in a tabular format within a user interface, but the columns within the table consist of unstructured text.

[0129] In some such embodiments, the selection (e.g., highlighting) of the inner and outer by the user via the user interface may be differentiated by color coding. For example, the outer highlighting of positive cases may be shown in a first text / background color combination, and the inner highlighting of positive cases may be shown in a different, contrasting text / background color combination.

[0130] As discussed above, a user may specify a selection of a capture group via selection of a character subsequence. A GUI may be used to facilitate user selection via highlighting (or other indication). An example is shown in FIG. 13, where an exemplary user interface screen is shown along with a tabular data display. In this example, FIG. 13 shows highlighting within a column value, e.g., caused by a user dragging a mouse across one or more desired elements of the column value. Note that the "cell" in which the user highlighting is performed may exhibit a color change indicating selection of the column value. This color change may be interpreted as an automated highlighting in response to the user highlighting.

[0131] 14 and 15 are exemplary user interface screens illustrating the generation of regular expressions and capture groups based on the selection of data from a tabular display. In these examples, FIGS. 14 and 15 show an additional user interface window that is automatically displayed upon detection of a user highlight 1401 in the tabular data display. The window includes a field 1402 for displaying positive cases, a field for displaying negative cases, and a field for displaying a regular expression that is dynamically (and nearly instantly) generated in response to the selection of a positive case from the tabular data display. In these examples, a user highlight in the column value 1401 may be equivalent to a user highlight in an automated highlight. Thus, a user highlight of an area code populates the positive case field 1402 with not only the user-highlighted area code 1401, but also the remainder of the phone number.

[0132] However, it should be understood that user highlighting is not limited to performance within automatic highlighting. For example, user highlighting may alternatively be performed within other user highlighting. As another example, user highlighting may alternatively be performed without inner highlighting (e.g., further highlighting within the highlighted text). These alternatives are particularly suited to semi-structured data, such as columns of data that contain "tweets" or other long strings (e.g., browser "user agent" strings).

[0133] Additionally, once a corresponding regular expression has been generated, other column values ​​1402 that match the regular expression may be identified based on additional automated highlighting. In the examples shown in Figures 14 and 15, the additional automated highlighting indicates elements of these other column values ​​that match the capturing groups of the generated regular expression. The additional automated highlighting may be performed using colors that are different from the colors used for user highlighting.

[0134] As shown in FIG. 15, additional user highlighting is shown to indicate user selection of other examples. Additional user highlighting may be performed in a manner similar to that described above. Thus, the user interface of FIG. 15 shows a population of other examples in the field 1502 for displaying positive examples. This may occur in response to detection of additional user highlighting. Additionally, the generated regular expression 1503 may be dynamically and nearly instantly updated so that it matches all of the positive examples 1502. In response to generation of the updated regular expression, automated highlighting of other column values ​​1504 that match the updated regular expression may also be updated. In some implementations, dynamic color coding may also be used. For example, matches may be color coded using a first color (e.g., blue), positive examples may be color coded using a second color (e.g., green), and negative examples may be color coded using a third color (e.g., red). Within a tabular data view (such as a spreadsheet) or other infinite scrolling data view (e.g., of semi-structured or unstructured data), the newly visible data may be color-coded as the view is scrolled down to actual additional data.

[0135] 16A and 16B are example user interface screens illustrating the generation of regular expressions based on the selection of positive and negative examples from a table-like display. In FIGS. 16A-16B, individual examples from positive example field 1602 may be removed from positive example field 1603 and / or moved to negative example field 1603. Within the user interface, this may be performed, for example, by the user clicking (e.g., right-clicking) on ​​one of the examples to select it. Selection may cause the user interface to display a menu 1602 that includes delete and modify options. Clicking an option then performs the corresponding function.

[0136] In the example shown in FIGS. 16A and 16B, the result of a user selection of the change option is to move the selected example to the negative example field 1603 and update regular expression 1601 to regular expression 1604, which may be generated dynamically and nearly instantaneously (e.g., in one embodiment, between 30 ms and 9000 ms, or 100 ms median). In response to generation of the updated regular expression 1604, automated highlighting of other column values ​​that match the updated regular expression may also be updated within the tabular data display. Additionally, automated highlighting may be performed for some or all of the negative examples, including any column values ​​that correspond to negative examples, which may be highlighted using a color different than any of the colors used above, or may otherwise be distinguished within the user interface using other visual techniques.

[0137] In some embodiments, designating a negative example via a user interface need not require first designating the example as a positive example and then converting it to a negative example, as shown in Figures 16A and 16B. Rather, a negative example may be designated in a variety of ways. For example, a user may select (e.g., right-click) a column value via a user interface (e.g., one of the other column values ​​on which automated highlighting has been performed to show that it matches the generated regular expression), thereby causing the display of a menu containing options (e.g., "Create a new counterexample") to designate the selected column value as a negative example.

[0138] 16A and 16B, in response to generating the updated regular expression 1604, the automated highlighting of other column values ​​that match the updated regular expression may also be updated. In these examples, the updated regular expression specifies a phone number that ends with a "9."

[0139] With brief reference to FIGS. 14 and 15, when the "Extract" button is clicked or otherwise selected by the user, operations may be initiated to extract highlighted text fragments in all cells that match the current regular expression 1403 or 1503. Although not shown in FIGS. 14 and 15, in some embodiments, the user interface may provide other selectable buttons in addition to or instead of the "Extract" button. For example, a "Replace" button may be presented as an option to replace the user-highlighted element with a user-specified element. Additionally or alternatively, one or more "Delete" buttons may be presented as an option to effectively replace the user-highlighted element with nothing. For example, one or both of a "Delete Fragment" operation and / or a "Delete Line" operation may be implemented, which will delete either the user-highlighted text fragment or the entire line, respectively. Additional operations that may be implemented in various embodiments may include a "keep lines" operation, a "split" operation (e.g., highlighting commas and then extracting the comma-separated components into separate new columns), and an "obfuscate" operation (e.g., replacing highlighted text / capture groups with "#" or other symbol sequences).

[0140] In this example, in response to the "Extract" button being selected, the extract operation may be added to a list of transformation scripts to be executed by downstream operations. In some embodiments, the list of transformation scripts may be displayed in a portion of the user interface for review / modification by the user. Alternatively, the extract operation may be performed on the fly to generate a new column containing the contents of the Regex capture group (e.g., elements corresponding to user-highlighted portions of positive cases). In the example shown in Figures 14 and 15, a new column and / or a new table of area codes may be generated in response to selection of the "Extract" button.

[0141] FIG. 17 is another example user interface screen illustrating generation of regular expressions and capturing groups based on selection of data from a tabular display in accordance with one or more embodiments described herein.

[0142] A. Multiple highlighting In some embodiments, the user interface may also support multiple highlighting per example. Figures 42, 43, 44, 45, 46, 47, and 48 show user interfaces 4200, 4300, 4400, 4500, 4600, 4700, and 4800 for implementing multiple highlighting, according to some exemplary embodiments. Different highlighting methods are described below. In an exemplary embodiment, inner highlighting and outer highlighting can be differentiated by using different color codes.

[0143] FIG. 42 illustrates a user interface 4200 displaying a view of a dataset in a single-level mode, according to some example embodiments. FIG. 42 illustrates how highlighting appears on the user interface while in single-level highlighting mode. In FIG. 42, a user highlights version number 4235 (e.g., "5.0") of an application 4236 (e.g., Mozilla). In response to the user highlighting version number 4235, the user interface data model generator highlights version numbers 4237 ("6.1"), 4238 ("1.9"), 4239 ("2.2"), and 4240 ("3.6").

[0144] Based on the highlighting, a Regex by Example dialog box 4230 appears. The Regex by Example dialog box 4230 includes a specified example 4233 specified by the user. In this example, a version number "5.0" is specified by the user. The Regex by Example dialog box 4230 also indicates that the highlighting mode 4250 is single level 4251.

[0145] FIG. 43 illustrates a user interface 4300 displaying highlighted data in nested full control mode, according to some exemplary embodiments. In the example illustrated in FIG. 43, “nested (full control)” 4351 is selected from the highlight mode panel 4350 of the dialog box 4330. Also illustrated in FIG. 43, the user displays the outer highlight as the software name 4236 (e.g., “Mozilla”) and version number 4235 (e.g., “5.0”). The example Regex dialog box 4330 indicates that the highlight mode 4350 is in nested (full control) 4351. Additionally, the data set is currently in multi-highlight mode 4360. The highlighting performed in FIG. 43 performs the outer highlight in nested full automatic mode.

[0146] In an exemplary embodiment, an in progress outer highlight, which is an outer highlight that is currently being executed, may be treated in a first class manner with its own urgency color, such as gold. In an exemplary embodiment, the highlight may appear in different colors that may be used to indicate the urgency, priority, and / or importance of the highlight.

[0147] FIG. 44 illustrates a user interface 4400 displaying highlighted data in a nested full control mode, according to some exemplary embodiments. As shown in FIG. 44, a multi-highlight mode 4460 is selected. A user displays an outer highlight as a software name 4236 and a version number 4235. The software name 4236 (e.g., Mozilla) and the version number 4235 (e.g., 5.0) selected as the outer highlight may be highlighted in a first color. A user can identify an inner highlight after selecting the outer highlight. For example, a user can select only the version number 4235 (e.g., "5.0") as the inner highlight. The inner highlight may be highlighted in a second color different from the first color. Since the inner highlight and the outer highlight appear in different colors, the outer highlight and the inner highlight can be easily identified. The example shown in FIG. 44 is in a nested full control mode highlight. Thus, the highlight mode panel 4450 indicates that the data set is in nested full control mode 4451 .

[0148] In an exemplary embodiment, the inner and outer highlighting may be differentiated by using different colors or color codes, for example, the outer highlighting of positive cases may be shown in black text on a light green background, and the inner highlighting of positive cases may be shown in light green text on a dark green background.

[0149] In FIG. 44, the user has selected version number "5.0," identified as element 4235. When the user selects version number "5.0" (element 4235), the user interface data model generator may automatically select version numbers "3.6" (element 4240) and "5.1" (element 4241). Because the user interface data model generator has highlighted version numbers in addition to the version number selected by the user, the user may provide additional examples if the additional version numbers identified by the user interface data model generator are not desired by the user. Alternatively, if the user agrees with the additional highlighting performed by the user interface data model generator, the user may proceed to apply commands to the highlighted data.

[0150] The Regex by Example dialog box 4430 displays the specified example 4433. Additionally, the Regex by Example dialog box 4430 indicates that the highlighting mode 4450 is nested (full control) 4451. The Regex by Example dialog box 4430 also indicates that it is in multiple highlighting mode 4460. The generated regular expression 4432 is also identified in the Regex by Example dialog box 4430.

[0151] The user may provide further examples, as shown in Figures 45, 46 and 47. Figure 45 shows a user interface 4500 where two positive examples are provided, according to some exemplary embodiments. Figure 46 shows a user interface 4600 where two positive examples are provided, according to some exemplary embodiments. Figure 47 shows a user interface 4500 where three positive examples are provided, according to some exemplary embodiments.

[0152] As shown in FIG. 45, to provide a positive example, a user may indicate an outer highlight as a software name 4536 and a version number 4535. The software name 4536 (e.g., Windows NT) and the version number 4535 (e.g., 6.1) selected as the outer highlight may be highlighted in a first color. After the user selects the outer highlight, the user may identify an inner highlight. For example, the user may select the version number 4535 (e.g., "6.1") as the inner highlight. The inner highlight may be highlighted in a second color that is different from the first color. A regular expression 4532 may be generated based on the highlight.

[0153] The highlighting example provided by the user is identified in the specified example 4533 in the Regex by Example dialog box 4530 .

[0154] Figure 46 illustrates a user interface 4600 displaying an updated generated regex, according to some example embodiments. As shown in Figure 46, the generated regex 4632 is updated based on two specified examples 4633 (e.g., "Mozilla 5.0" and "Windows NT 6.1"). As additional outer and inner highlighting is provided by the user in the specified example 4633, the generated regex 4632 changes accordingly. The generated regex 4632 appears in the Regex by Example dialog box 4630.

[0155] 47 illustrates a user interface 4700 displaying alternative data highlighting, according to some exemplary embodiments. As shown in FIG. 47, a version number with an underscore can be provided as an example for generating the regeex.

[0156] As shown in FIG. 47, to provide another positive example, a user may indicate an outer highlight as a software name 4735 and a version number 4735. The software name 4735 (e.g., MacOS X) and the version number 4736 (e.g., 10_6_8) selected as the outer highlight may be highlighted in a first color. After the user selects the outer highlight, the user may identify an inner highlight. For example, the user may select the version number 4736 (e.g., 10_6_8) as the inner highlight. The inner highlight may be highlighted in a second color that is different from the first color.

[0157] The highlighting examples provided by the user are identified in the specified examples 4733 in the Regex by Example dialog box 4730. As shown in FIG. 47, the generated Regex 4732 is updated based on the three specified examples 4733 (e.g., "Mozilla 5.0", "WindowsNT 6.1" and "MacOS X 10_6_8"). As additional outer and inner highlighting is provided by the user in the specified examples 4733, the generated Regex 4732 changes accordingly.

[0158] Figure 48 illustrates a user interface 4800 displaying an updated generated regex, according to some example embodiments. As shown in Figure 48, in response to a user selecting a version number with an underscore, the generated regex 4832 is updated to include the underscore information.

[0159] The generated regeex 4832 is updated based on three specified examples 4833 (e.g., "Mozilla 5.0", "Windows NT 6.1" and "MacOS X 10_6_8"). As additional outer and inner highlighting is provided by the user in the specified examples 4833, the generated regeex 4832 changes accordingly. The generated regex 4832 is displayed in the Regex by Example dialog box 4830.

[0160] FIG. 49 is a flowchart of a method 4900 for performing multiple highlighting, according to some exemplary embodiments.

[0161] In step 4910, the user may select data (e.g., a data fragment) within the data set. That is, the user may select an outer highlight (e.g., "Mozilla 5.0") and an inner highlight (e.g., "5.0"). The user may perform outer and inner highlighting on the first data record 4320.

[0162] In step 4920, after the user performs the initial highlighting, the user interface data model generator may automatically highlight data (e.g., data fragments) in the same record and / or other records in the dataset that correspond to the highlighting made by the user in step 4910.

[0163] In step 4930, a regular expression may be generated based on the highlighting provided by the user in step 4910 and based on additional highlighting performed by the user interface data model generator in step 4920.

[0164] If the user is not satisfied with the highlighting results, the user can highlight additional data to provide additional examples. Thus, steps 4910, 4920, and 4930 can be repeated until the user is satisfied with the final highlighting on the data set. Alternatively, the user can modify the generated regular expression (e.g., 4832) to generate the desired highlighting.

[0165] In an exemplary embodiment, for multiple highlights in nested (auto outer) mode, the search is performed for the minimum number of surrounding spans required. When the user interface is in nested (auto outer) mode with multiple highlights, the backend algorithm can enter a special mode. The reason for the special mode is that multiple highlights per example tend to mean that the example strings are longer than normal. Since the LCS algorithm has an exponential runtime, this would take too long to run on the full length of the outer highlight. In the special mode, the algorithm starts with the inner highlight and gradually grows the outer highlight around the inner highlight until a satisfactory regeex is generated.

[0166] Exemplary code for implementing a special mode may include some or all of the following code:

[0167]

number

[0168]

number

[0169] Multiple highlighting allows a user to easily select data (e.g., data fragments) to which any of the commands (e.g., extract, split, delete, obfuscate, replace, and filter lines) can be applied.

[0170] For multiple-highlight-per-example capabilities, individual highlights may be clickable instead of the entire row. In some cases, a checkbox (or other technique in the user interface) may be used to toggle between single-highlight-per-example and multiple-highlight-per-example modes. For example, in a mode supporting multiple-highlight-per-example, a user may select multiple different highlighted portions within a single data example (e.g., a spreadsheet cell) to provide multiple different examples for which regular expressions may be generated. In multiple-highlight-per-example, both positive and negative "examples" may be identified and shown as color codes in a single list of examples. For example, in multi-highlight mode, clicking on a highlight will make only that one sub-example negative. The functionality of multiple-highlight-per-example is illustrated in Figures 42-49.

[0171] As shown in this series of examples, the user sequentially selects an outer highlighted region followed by an inner highlighted region to generate / update the corresponding regular expression. Additionally, as shown in these figures, the user can specify multiple different inner / outer highlighted examples within a single data cell, and the generated regular expression can be updated with each new example highlighted (or otherwise selected by the user) via the user interface.

[0172] In some embodiments, the special multi-highlight nested auto-outer look mode may be limited in diameter to a certain number of spans (e.g., 10 spans). In such a case, the condition for the final do-while loop may be like this: while (!result.isSuccess && !reachedMaximal && numLookAheadAndBehindSpans < 10).

[0173] Multiple highlighting may be used when implementing certain commands. B. Command As mentioned above, the user interface may support additional types of commands in addition to the extract command in various embodiments, and elements of the user interface data model generator may be configured to implement those commands.

[0174] 24 illustrates a regular expression generation server 2400 according to some exemplary embodiments. The regular expression generation server 2400 corresponds to the regular expression generator server 111 illustrated in FIG. 1. The regular expression generation server 2400 may include multiple processors and memories. The regular expression generation server 2400 may include a regular expression generator 2410 and a user interface data model generator 2420. The regular expression generator 2410 may also be referred to as a regeex generator. The regular expression generator 2410 may correspond to the regular expression generator 116 of FIG. 1. The user interface data model generator 2420 may correspond to the user interface data model generator 114 of FIG. 1.

[0175] The regular expression generator 2420 can execute several different commands, including extract, split, delete, obfuscate, replace, and filter lines. Thus, the regular expression generator 2420 can include an extract data generator 2411, a split data generator 2412, a delete data generator 2413, an obfuscate data generator 2414, a replace data generator 2415, and a line filtering data generator 2416. Each component of the regular expression creation server 2400 is described in more detail below.

[0176] Although extract, split, delete, obfuscate, replace, and line filtering commands are described, example embodiments may be configured to perform additional commands. Also, regular expression generator 2410 may include command generators other than those shown in FIG.

[0177] 1. Extraction As mentioned above, in the extract command, regular expression pattern matches may be identified and matching data may be extracted into newly created columns. The extract function is described above with respect to Figures 14 and 15.

[0178] 2. Split Figures 25A, 25B, and 26 show examples of a split command implemented via a user interface. Figures 25A and 25B show user interface 2500 and user interface 2501 for implementing a split command, according to some exemplary embodiments. Figure 26 shows the results of a split command on a data set, according to some exemplary embodiments.

[0179] The split command can result in the creation of multiple different columns based on the presence of a particular regular expression used as a delimiter. For example, if a comma (",") is given as the pattern to be matched, the comma is used as the delimiter during the extract operation. Each data item to be extracted that contains a single comma is split at the comma and extracted into two different columns (i.e., the data before the comma and the data after the comma). If the data item to be split contains two commas, the data is split into three parts based on the comma and extracted into three new columns. Thus, the split command can perform extraction based on a delimiter (into one or more new columns) rather than extraction based on a pattern match (where matches would be put into a single column).

[0180] In other examples, split commands may be implemented for certain types of content, such as using curly or square brackets, asterisks, '<' or '>', or hyphen and dash characters in the pattern to be matched (or delimiters). Additional symbols may be used as delimiters.

[0181] As shown in Fig. 25A, a data set is displayed on a user interface 2500. The data set may be a spreadsheet. The data set may include a column 2510 containing a phone number. The phone number may include a first hyphen 2511 and a second hyphen 2512. If a split command is implemented on the phone number column 2510, the phone number may be split as desired by the user.

[0182] 27 illustrates a flowchart of a method 2700 for executing a split command, according to some exemplary embodiments. In the example of FIG. 27, a hyphen character is used as a delimiter to split data.

[0183] In step 2710, the user may select a hyphen in the first record 2520 in the "phone number" column 2510. The hyphen may be selected by the user highlighting the hyphen on the display of the device, for example with a mouse or gesture, if the user is on an interactive user interface. In the example shown in FIG. 25A, the user has highlighted the first hyphen 2511 in the first record 2520.

[0184] In step 2720, after the user highlights the first hyphen in the first record 2520, the user interface data model generator can automatically highlight all of the first hyphens in the phone number column that correspond to the selection made by the user for each of the data records (e.g., records 2-25 as shown in FIG. 25A). That is, the user interface data model generator can determine what highlighting should be done for the remaining records (e.g., 2-25) in the phone number column based on the highlighting done by the user for the first record 2520. Although 25 records are shown in the drawings, exemplary embodiments can apply the split command to more or less than 25 records. Additionally, the data set can include a large number of records, such as thousands or thousands of records. The regular expression can determine what additional highlighting should be done within the same record or additional records based on the initial highlighting done by the user.

[0185] In step 2730, after the highlighting has been performed, a regular expression can be generated based on the highlighting. Figure 25A shows a generated regular expression 2532 that is generated based on the highlighting performed by the user and the automatic highlighting performed by the regular expression generator. The generated regular expression is updated based on the highlighting changes made by the user.

[0186] In step 2740, the "Regex by Example" dialog box 2530 automatically appears on the screen. In the example shown in Figure 25A, the "Regex by Example" dialog box 2530 appears after highlighting has been performed on all records (e.g., records 1-25) in the phone number column 2510, however, the "Regex by Example" dialog box 2530 can also appear after initial input (e.g., highlighting of the first record by the user).

[0187] 25A shows the specified example 2533, allowing the user to easily highlight selected data or modify the highlighting made to selected data. The user can modify the highlighting by changing the highlighting in the data in the "phone number" column 2510, by changing the highlighting in the specified example 2533, or by modifying the generated regular expression 2532. Changes made to the "phone number" column 2510 are reflected in the generated regular expression 2532.

[0188] In an example embodiment, the minimum spanning tree can be used on the examples, but not on the highlights. Thus, if there are multiple highlights for each example given and the highlights overlap, then only a single copy of the longest common subsequence algorithm needs to be run for each example. If the highlights for a given example do not overlap, then the longest common subsequence algorithm can be run for each of the highlights.

[0189] In an exemplary embodiment, the longest span list is selected for a minimum spanning tree (MST) vertex and fed first into the LCS queue. The LCS queue contains the spans to which the longest common subsequence algorithm is applied. With one example representing multiple highlights per example contained in the array passed to graphLcs(), the longest such example is selected to be a vertex in the graph of the minimum spanning tree that determines the ordered list of spans to be passed to the LCS queue.

[0190] At step 2750, a selection of a "Split" command is received. For example, a user may select the "Split" button 2531 on the Regex by Example dialog box 2530. As shown in the Regex by Example dialog box 2530, the multi-highlight checkbox 2534 and the highlight mode drop-down list are disabled when Split is selected.

[0191] In an exemplary embodiment, the checkbox may be animated to alert the user when the UI automatically checks the checkbox based on the user providing a second highlight. An exemplary embodiment activates the checkbox based on a change in state caused by the user interacting elsewhere in the UI. A checkbox is activated when the server system causes the checkbox to automatically change its state based on user interaction elsewhere on the screen.

[0192] After a selection of the "split" command is received, split data generator 2412 of regular expression generator 2410 can automatically split the phone number based on the highlighted delimiters in step 2760. The example shown in Figure 25B occurs after a user selects the "split" command 2531 in Figure 25A.

[0193] As shown in FIG. 25B, the split command changes the highlight setting to multi-highlight. In FIG. 25A, prior to the selection of the split command, the multi-highlight mode 2534 is not selected. In FIG. 25B, after the selection of the split command, the multi-highlight mode 2534 is selected. In some embodiments, when the "Split" command is selected, the user interface may enable a multi-highlight checkbox and a highlight mode drop-down list. For example, referring to FIG. 25B, when a user clicks on the "Split" command, the highlight mode may automatically change to single level and then the multi-highlight text box may be enabled.

[0194] In step 2770, the split command results can be displayed. As shown in the user interface 2600 shown in FIG. 26, the phone number from the phone number column 2510 is now split into three columns 2610, 2620, and 2630. Column 2610 contains the portion of the phone number that appears before the first hyphen, column 2620 contains the portion of the phone number that appears between the first and second hyphens, and column 2630 contains the portion of the phone number that appears after the second hyphen. The phone number is split into three columns because there are two hyphens acting as delimiters. For some numbers, one column may contain the area code along with the local code because the area code appears before the first hyphen. For phone numbers, the fields will be populated based on the information in the phone number. If the phone number has only one hyphen, then only two columns of information will appear for that number.

[0195] Three additional columns are created in the spreadsheet view as shown in FIG. 26. Because the phone number is split up, the user can more easily identify the desired information. For example, the user can more easily identify the area code in the data record. The user can also perform additional processing on each of columns 2610, 2620, and 2630. Thus, the user can more easily utilize the data. For example, the user can identify the customer's primary area code.

[0196] When a split command is executed, the executed split command (e.g., phone column split) may be added to a transformation script for the dataset. The transformation script indicates the commands that are applied to columns of data. The transformation script can be used to apply the same commands to another dataset.

[0197] Thus, the exemplary embodiments provide a fast and efficient user interface for splitting data within a dataset. Additionally, the regular expression generator can be configured to identify settings that are more likely to produce results desired by the user.

[0198] Although the flow diagram is described with specific steps, the order of the steps can be changed, for example, a regular expression can be generated based on initial highlighting performed by a user.

[0199] 3. Delete Figure 28 illustrates a user interface 2800 for implementing a delete command, according to some exemplary embodiments. Figure 29 illustrates a user interface 2900 displaying the results of a delete command on a dataset, according to some exemplary embodiments. In a delete command, a user may highlight (or otherwise select) an example text, generate a regular expression that corresponds to the pattern, and then delete data that matches the pattern from a spreadsheet or other data view. The delete function replaces the data with an empty string (e.g., the delete command can be implemented by converting the generated regular expression into three capture groups).

[0200] As shown in FIG. 28, a data set is displayed in a user interface 2800. The data set can include a column 2810 that contains a street address. In the example shown in FIG. 28, a user wants to remove a street number 2811 from an address. If a delete command is implemented in the "street_address" column 2810 of record 2820, the street address information can be deleted as desired by the user. In FIG. 28, the user has selected some positive examples to identify street numbers in the "street address" column of the spreadsheet. These examples appear as designated examples 2833. The user then clicks the "Delete" button to initiate the delete operation. FIG. 29 shows the resulting spreadsheet with the street number deleted. Thus, the delete command is essentially replaced with (empty string). This can be achieved by converting the generated legex into three capture groups, as discussed below with respect to the "Replace" command.

[0201] FIG. 30 illustrates a flowchart of a method 3000 for executing a delete command, according to some example embodiments.

[0202] In step 3010, the user may select a portion of the address information for deletion. In the example shown in Figure 28, the street number portion 2811 of the address information has been selected for the first record 2820. The street number portion 2811 of the address information may be selected by the user highlighting the street number portion on the display of the device, for example with a mouse or gestures, if the user is on an interactive user interface.

[0203] In step 3020, after the user highlights the street number portion 2811 in the first record 2820, the user interface data model generator can automatically highlight all of the street number portions in the "street_address" column 2810 for each of the data records (e.g., records 2-25 shown in FIG. 28). That is, the user interface data model generator can determine what highlighting should be performed on the remaining records (e.g., 2-25) in the "street_address" column 2810 based on the highlighting performed on the first record 2820. Although 25 records are shown in the drawings, exemplary embodiments can apply the delete command to more or less than 25 records. A regular expression can determine what additional highlighting should be performed based on the initial highlighting by the user.

[0204] In step 3030, a regular expression is generated. In FIG. 28, the generated regular expression 2832 is shown. The generated regular expression allows the user to easily highlight which data to select. In the case of a delete command, the generated regular expression allows the user to easily highlight which data to delete. The generated regular expression corresponds to the highlighting made by the user and the regular expression generator. The user can modify the highlighting by changing the highlighting in the data in the "street_address" column 2810 or by changing the highlighting in the regular expression 2832. Changes to the "street_address" column 2810 are reflected in the generated regular expression 2832. The generated regular expression is updated based on the highlighting changes made by the user.

[0205] In step 3040, a "Regex by Example" dialog box 2830 may be automatically displayed on the screen after highlighting has been applied to all street numbers in the "street_address" column 2810. In the example shown in Figure 28, the "Regex by Example" dialog box 2830 appears after highlighting has been applied to all records (e.g., records 1-25) in the "street_address" column 2810, but the "Regex by Example" dialog box 2830 may also appear after initial input (e.g., highlighting the first record). The specified example 2833 identifies the example provided by the user.

[0206] A selection of a delete command may be received at step 3050. For example, a user may select the “Delete” button 2831 in the “Regex by Example” dialog box 2830.

[0207] After receiving the selection of the delete command, the delete data generator 2413 of the regular expression generator 2410 can automatically delete the street numbers based on the highlighting in step 3060. In the example shown in Figure 28, all of the "street_address" records include highlighting, so the delete command is applied to each of the records shown in Figure 28.

[0208] The results of the deletion may be displayed in step 3070. As shown in user interface 2900 displayed in Figure 29, the street numbers have been removed from the street addresses in the "street address" column 2810. The street numbers are deleted in place.

[0209] After a deletion command is executed, the executed deletion command (e.g., delete street address column) may be added to a transformation script for the dataset. The transformation script indicates the commands that are applied to columns of data. The transformation script can be used to apply the same commands to another dataset.

[0210] Thus, the illustrative embodiments provide a quick and efficient user interface for deleting data in a data set.

[0211] 4. Obfuscation Figure 31 illustrates a user interface 3100 for implementing an obfuscation command, according to some example embodiments. Figure 32 illustrates a user interface 3200 showing the results of an obfuscation command on a data set, according to some example embodiments.

[0212] With the obfuscate command, a user may highlight (or otherwise select) example text, generate a regular expression that corresponds to the pattern, and then obfuscate (e.g., obscure rather than delete) data that matches the pattern from a spreadsheet or other data view. Data may be obfuscated so that all of the data in a record is not visible. For example, a user may want to obfuscate information for security or privacy reasons.

[0213] In Figure 31, the user has selected the two middle digits 3112 in the "ssn" column 3110 of the spreadsheet. The user then clicks the "Obfuscate" button to begin the obfuscation operation. Figure 32 shows the resulting spreadsheet after replacing the two middle digits of the entire "ssn" column with two pound signs.

[0214] As shown in Figure 31, a data set is displayed in a user interface 3100. The data set includes a column 3110 directed to, for example, social security numbers. Although social security numbers are used to illustrate the obfuscation command, any of the data within the column of data may be obfuscated as desired by the user. For example, a user may wish to obfuscate any type of sensitive information, such as credit card information or bank account information.

[0215] The social security number can be divided into a first field 3111, a second field 3112, and a third field 3113. The first field 3111 can be before the first dash, the second field 3112 can be between the dashes, and the third field 3113 can be after the second dash of the social security number. In the example shown in FIG. 31, the user wishes to obfuscate the second field 3112 of the social security number (e.g., the digits between the first dash and the second dash) so that not all of the social security number is visible. Although the second field is obfuscated, the user can choose to obfuscate any or all of the fields within the social security number.

[0216] FIG. 33 illustrates a flowchart of a method 3300 for executing an obfuscated command, according to some example embodiments.

[0217] In step 3310, a selection of a second field 3112 in the social security number column 3110 is accepted. The selected field 3112 is selected for the first record 3120. The second field 3112 may be selected, for example, by using a mouse or gestures on a display of a device when the user is on an interactive user interface. The second field selected by the user may be highlighted in a first color. The selection made by the user is provided as a designated example 3133 in the dialog box 3130.

[0218] In step 3320, all of the social security numbers in the social security number column may be highlighted to correspond to the highlighting performed in step 3320. That is, all second fields within the social security numbers in column 3110 will be highlighted in a first color and all first fields 3111 and third fields 3113 within the social security numbers in column 3110 will be highlighted in a second color. The fields are highlighted in different colors to allow a user to easily distinguish between fields that are to be modified and fields that are not to be modified.

[0219] The user interface data model generator can highlight all records (e.g., 2-25) to correspond to the highlighting performed by the user on the first record 3120. That is, the user interface data model generator can determine what highlighting should be performed on the remaining records (e.g., 2-25) in the social security number column based on the highlighting performed on the first record 3120. Although 25 records are shown in the drawings, example embodiments can apply the obfuscation command to more or less than 25 records.

[0220] In step 3030, a regular expression may be generated. In FIG. 31, the generated regular expression 3132 is shown in a dialog box by example 3130. The generated regular expression allows the user to easily highlight which data should be obfuscated. The user may modify the highlighting by changing the highlighting in the data in the "ssn" column 3110 or by changing the highlighting in the generated regular expression 3132. Changes to the "ssn" column 3110 are reflected in the generated regular expression 3132. That is, the generated regular expression is updated based on the highlighting changes made by the user to the data set.

[0221] In step 3340, after highlighting has been applied to the social security number, a "Regex by Example" dialog box 3130 can automatically appear on the screen. In the example shown in FIG. 31, the "Regex by Example" dialog box 3130 appears after highlighting has been applied to all records (e.g., records 1-25) in the social security number column 3110, but the "Regex by Example" dialog box 3130 can also appear after initial input (e.g., highlighting the first record). The "Regex by Example" dialog box 3130 includes the specified example 3133 entered by the user. The "Regex by Example" dialog box 3130 also includes the generated regular expression 3132.

[0222] At step 3350, a selection of an “Obfuscate” command is accepted. For example, a user may select the “Obfuscate” button 3131 in the “Regex by Example” dialog box 3130. As shown in FIG. 31, a user may select the Obfuscate button 3131.

[0223] In step 3360, after the obfuscation command is received, the obfuscation generator 2414 of the regular expression generation server 2400 may automatically obfuscate all of the second fields of the social security numbers in the “ssn” column 3110.

[0224] At step 3370, the obfuscation results may be displayed. Figure 32 illustrates a user interface 3200 displaying the results of an obfuscation command on a dataset, according to some example embodiments. As illustrated in Figure 32, the second fields 3112 selected by the user are replaced with "##" in place of the highlighted fields to be obfuscated. Thus, all of the second fields 3112 of the social security numbers in the social security column 3110 have been replaced with "##". That is, the second fields 3112 for all social security numbers have been obfuscated.

[0225] After an obfuscation command is executed, the executed obfuscation command (e.g., obfuscate_column_ssn) may be added to a transformation script 3230 for the dataset. The transformation script indicates the commands that are applied to columns of data. The transformation script can be used to apply the same commands to another dataset.

[0226] Thus, the illustrative embodiments can provide a quick and efficient user interface for obfuscating data in a dataset while maintaining the privacy of user information.

[0227] 5.Replacement According to some example embodiments, Figure 34 shows a user interface 3400 for implementing a replace command, and Figure 35 shows a user interface 3500 for implementing a replace command. In the replace command, a user can highlight (or otherwise select) an example text, generate a regular expression that corresponds to the pattern, and then replace the text that matches the pattern with other selected text.

[0228] The replace command includes dynamically displaying a two-column table showing the before and after examples in a dialog. To utilize the contents of a capture group in a replace expression, users can use $1, which is standard Regex replacement syntax from the Java and Javascript Regex API.

[0229] FIG. 37 illustrates a flowchart of a method 3700 for executing a replace command, according to some example embodiments.

[0230] In step 3710, the user may select within the "street_address" column 2810 the field that the user wishes to replace with other data. In the example shown in FIG. 34, the user wishes to replace "Drive" with "Dr." in the street address. As shown in FIG. 34, the user may select "Drive" from the fifth record 3420 of the dataset. The data to be replaced may be selected by the user, if on an interactive user interface, by highlighting the data to be replaced on the display of the device, for example with a mouse or gestures.

[0231] In step 3720, after the user highlights the data to be replaced in the fifth record (e.g., "Drive"), the user interface data model generator can automatically highlight all fields in the remaining records that correspond to the highlighting performed in the fifth record 3420. The user interface data model generator can automatically highlight the street address that contains "Drive". As shown in FIG. 34, since records 7, 13, 16, 18, and 20 contain "Drive", "Drive" is automatically highlighted by the user interface data model generator in records 7, 13, 16, 18, and 20. "Drive" can be highlighted in a first color in records 5, 7, 13, 16, 18, and 20. The rest of the street address that contains "Drive" can be highlighted in a second color different from the first color. Thus, the portion to be replaced can be easily identified.

[0232] In step 3730, a regular expression 3432 may be generated. As shown in FIG. 35, a three-part generated regular expression 3532 is shown. The three-part generated regular expression allows the user to easily replace the "Drive" portion of the data, the portion before "Drive", or the portion after "Drive". In this example, the user replaces the "Drive" portion of the data with "Dr", and the user can immediately see the results in a preview spreadsheet in the pop-up "Regex by Example" dialog box 3530.

[0233] In Figure 34, a user has selected a sufficient number of examples 3433 (positive and negative) from the "street_address" column of a spreadsheet to generate a regular expression that corresponds to addresses ending in "Drive." Address 3434 is a positive example, and address 3435 is a negative example.

[0234] In step 3740, after highlighting has been performed on all records, the "Regex by Example" dialog box 3430 automatically appears on the screen. In the example shown in FIG. 34, the "Regex by Example" dialog box 3430 appears after highlighting has been performed on all records containing data to be replaced, but the "Regex by Example" dialog box 3430 can also appear after initial input (e.g., highlighting "Drive" in the fifth record). The "Regex by Example" dialog box 3430 can include the specified example 3433 and the generated regular expression 3432. Address 3434 is a positive example and address 3435 is a negative example.

[0235] A selection of the "Replace" command is accepted at step 3750. For example, a user may select the "Replace" button 3431 on the "Regex by Example" dialog box 3430 to initiate a replace command.

[0236] In step 3760, panel 3538 and "Replace with" field 3537 may be displayed to assist the user in determining what information is being replaced. As shown in FIG. 35, in panel 3538 the user can see the initial address and how it will appear when replaced. For example, the word "Drive" in the address will appear as "Dr." in the address, as shown by element 3539. Panel 3538 may assist the user in providing a preview of how the data will appear if the replace command is applied.

[0237] As shown in FIG. 35, regeex 3532 is converted into a triple capture group. The generated regeex is "^(.*?)(D[az]+)()$". Thus, the user can easily see what data appears before highlighting (^(.*?)), on highlighting ((D[az]+)), and after highlighting (()). In the example shown in FIG. 35, no data appears after highlighting (()). The data on highlighting is the data selected by the user. In this example, the data on highlighting is "Drive", which is the data the user selected for replacement.

[0238] The user can also identify what the selected word will be replaced with. The "Replace with" field 3537 identifies the word (e.g., "Dr") with which the selected word (e.g., "Drive") should be replaced. The user can modify the replacement word in the "Replace with" field 3537. The replacement word selected by the user in the "Replace with" field 3537 is applied to the highlighted record. If the user agrees with the replacement, the user can select the Create button 3536 and the replacement data generator 2415 can make the replacement.

[0239] As shown above, the replace command may also include dynamically displaying a two-column table in the dialog showing the before and after examples (e.g., a preview spreadsheet in the pop-up "Regex by Example" dialog box). To utilize the contents of the capture group in the replace expression, the user can use the standard regex replacement syntax, $1, from the Java and Javascript regex API. Furthermore, if the user selects the replace command and the user is in a mode where they generate capture groups (i.e., nested-auto-outer), the generated regex displayed in the dialog gets two additional capture groups for a total of three, namely, before highlight, on highlight, and after highlight. The user can then refer to these as $1, $2, and $3 in the replace expression. Furthermore, if there are no capture groups, the system can wrap the entire generated regex in a capture group and the user can refer to the original text with $1.

[0240] As disclosed in the exemplary embodiment, highlighting modes can include nested auto, nested manual, and single level. Nested auto is also known as nested (auto outer). Nested manual is also known as nested (full control).

[0241] Substitutions can be made in step 3770. After clicking the "Create" button 3536 to confirm the changes, the data is substituted.

[0242] In step 3780, the replacement results may be displayed. Figure 36 illustrates a user interface 3600 that displays the results of a replace command in a dataset, according to some example embodiments. As shown in the user interface 3600 illustrated in Figure 36, records containing "Drive" are now replaced with "Dr.". This causes records containing "Drive" in the "street_address" column 2810 to be replaced with "Dr.".

[0243] After a substitution command is performed, the substitution command that was performed (e.g., a street address column substitution) may be added to a transformation script for the dataset. The transformation script indicates the commands that are applied to columns of data. The transformation script can be used to apply the same commands to another dataset.

[0244] Thus, the illustrative embodiments provide a quick and efficient user interface for replacing data in a dataset.

[0245] 6. Row Filtering Figures 38 and 39 show user interfaces 3800 and 3900 for implementing row filtering commands, according to some exemplary embodiments. Figure 40 shows a user interface 4000 that displays the results of a row filtering command on a dataset, according to some exemplary embodiments.

[0246] In a filtering operation, a user can highlight (or otherwise select) example text, generate a regular expression that corresponds to the pattern, and then filter the data to include (or exclude) data that matches the pattern. Examples of the "row filtering" operation are shown in Figures 38, 39, 40, and 41.

[0247] 41 illustrates a flowchart of a method 4100 for executing a row filtering command, according to some example embodiments. In the example shown in FIG. 41, a user wants to filter records in a dataset to identify records that have "Avenue" in the street address.

[0248] In step 4110, the user may select the data to be used to filter the records. In the example shown in FIG. 38, the user has selected "Avenue" from the "street address" column 2810. The data may be selected by the user by highlighting the data to be used for filtering. For example, if the user is on an interactive user interface, the data may be selected with a mouse or via gestures on the device's display.

[0249] In step 4120, after the user highlights the data used for filtering, the user interface data model generator can automatically highlight the corresponding data in other records. For example, the user interface data model generator may highlight the word "Avenue" in records 8, 13, and 19. That is, the user interface data model generator can determine what highlighting should be performed for the remaining records (e.g., 2-25) in the "street_address" column 2810 based on the highlighting performed for the first record 3820.

[0250] In step 4130, a regular expression may be generated based on the highlighting selected by the user. Figure 39 shows a dialog box 3930 containing the generated regular expression.

[0251] In FIG. 39, the generated regular expression 3932 is shown, allowing the user to easily identify the data to be used as the basis for filtering. In FIG. 39, three examples have been highlighted by the user and are displayed in the specified examples 3933. Thus, three addresses appear in the specified examples 3933. The user has selected a sufficient number of examples 3933 (positives 3934 and negatives 3935) from the "street_address" column of the spreadsheet to generate a regular expression that corresponds to addresses ending with "Avenue". Although three examples are shown, the user may use one or more examples based on the results desired by the user.

[0252] The positive and negative examples may be displayed in different colors or in any manner in which the positive and negative examples are displayed differently in the dialog box 3930 and in the data set displayed on the user interface. In the specified example 3933 shown in FIG. 39, two addresses containing "street" are selected by the user as negative examples from the "street address" column 2810. That is, they are examples of data that the user does not want to appear in the records. However, if additional examples are provided by the user, a more accurate Regex expression can be generated that is applied to the filtering.

[0253] A user can modify the highlighting by changing the highlighting in the data in the "street_address" column 2810 or by changing the highlighting in the regular expression 3932. Changes to the "street_address" column 2810 are reflected in the generated regular expression 3932. In other examples, a user may use similar techniques to define regular expressions that correspond to patterns that should be filtered out (rather than retained).

[0254] In step 4140, after the highlighting has been performed, the "Regex by Example" dialog box 3930 automatically appears on the screen. In the example shown in Figure 39, the "Regex by Example" dialog box 3930 appears after highlighting has been performed for all records (e.g., records 1-25) in the "street_address" column 2810, although the "Regex by Example" dialog box 3930 can also appear after initial input (e.g., highlighting the first record).

[0255] At step 4150, a selection of a "Row Filtering" command is accepted. For example, a user may select the "Row Filtering" button 3931 in the "Regex by Example" dialog box 3930. The user may click on the "Row Filtering" button 3931 in the user interface to initiate a filter option that filters out all other types of addresses, leaving only "Avenue" addresses, as shown in FIG.

[0256] In step 4160, a confirmation regarding the row filtering command is accepted. Specifically, a selection is accepted to keep the record identified by example 3933 (Keep button 3937) or to delete the record identified by example 3933 (Delete button 3938). After selecting the "Keep" button 3934 or the "Delete" button 3935, the user can select the Create button 3936 to begin filtering.

[0257] In step 4170, the row filtering data generator 2416 of the regular expression generator 2410 may automatically filter the records according to the filtering criteria entered by the user.

[0258] Step 4180 displays the results of the row filtering. As shown in user interface 4000 shown in FIG. 40, the records have been filtered so that only records with "Avenue" in the address are shown. In this example, the spreadsheet contains more than 25 records. For illustration purposes, the first 25 records with "Avenue" in the "street_address" are shown.

[0259] When a row filtering command is executed, the executed row filtering command (e.g., street address column row filtering) may be added to a transformation script for the dataset. The transformation script indicates the commands that are applied to columns of data. The transformation script can be used to apply the same commands to another dataset.

[0260] Thus, the illustrative embodiments provide a quick and efficient user interface for filtering rows of data in a dataset.

[0261] VI. Context of Negative Examples In an exemplary embodiment, for negative examples in the nested manual, a search can be performed for contexts that correspond to positive examples.

[0262] When the user interface is in nested manual mode, an example may be received having an outer highlight of "Windows NT 6.1" and an inner highlight of "6.1". For example, in response to an initial example provided by a user, the UI may highlight "Windows NT 6.0". The UI may highlight "Windows NT 6.0" because, via a regular expression generator, "Windows NT 6.0" is determined to be a match with "Windows NT 6.1", which was initially provided as an example. The user may then select "Windows NT 6.0" to indicate that "Windows NT 6.0" is not a desired match with "Windows NT 6.1". That is, the user may provide a negative example by selecting "Windows NT 6.1". If the user clicks on "Windows NT 6.0" to create a negative example, the UI may send only "6.0" as the negative example. Thus, according to an exemplary embodiment, the UI may search for a matching "Windows NT" context before executing the LCS. The exemplary embodiment searches for a context based on the negative example provided.

[0263] FIG. 50 shows a flowchart of a method 5000 for providing negative examples, according to some exemplary embodiments.

[0264] In step 5010, an initial selection may be accepted. A regular expression is generated based on the initial selection.

[0265] For example, a user may select an outer highlight of "Windows NT 6.1" and an inner highlight of the "6.1" of "Windows NT 6.1". The outer and inner highlights may be created by highlighting data. For the outer highlight, the user may highlight "Windows NT 6.1" and for the inner highlight, the user may highlight the "6.1" of the already highlighted "Windows NT 6.1". The outer highlight corresponds to the application name and the inner highlight corresponds to the version number. The outer highlight may appear in a different color than the inner highlight. Highlighting may be performed by using, for example, a mouse or gestures on the display of the device when the user is on an interactive user interface.

[0266] In step 5020, after the user highlights the desired data fragment (e.g., inner and outer highlighting), the user interface data model generator can automatically highlight the corresponding data in the same record or the remaining records. For example, the user interface data model generator can highlight all instances of "6.1" in the record. However, the regular expression generator can also highlight all instances of "Windows NT 6.0" from the records in the dataset. Thus, to ensure high accuracy in the results, the user can provide negative examples.

[0267] In step 5030, the user can provide negative examples. The user may provide negative examples because they do not want all of the applications and version numbers automatically highlighted by the UI. For example, the user may select "Windows NT 6.0" to provide a negative example. The UI may then identify all records that contain a version number of "6.0". However, this may result in identifying applications other than "Windows NT" that contain a "6.0" version number.

[0268] In step 5040, the UI determines the context from the negative examples. Instead of highlighting all version numbers, including the version number "6.0", the UI determines the context from the examples provided by the user. That is, the UI not only identifies the version number "6.0" from the data records, but also identifies the application name "Windows NT" from the records when identifying records containing negative examples. Thus, instead of simply searching for records containing "6.0" when identifying records containing negative examples, the UI searches for "Windows NT 6.0" when identifying records containing negative examples.

[0269] Determining context from negative examples is described in more detail below with respect to FIG.

[0270] In step 5050, the highlighting on the dataset is updated according to the identified context: the UI highlights words containing "Windows NT 6.0" rather than just "6.0," thereby making the negative examples more accurate.

[0271] The generated regular expressions may be updated based on the context identified from the negative examples. Thus, the exemplary embodiment provides a more accurate method of providing negative examples. The context is determined before running the LCS algorithm to determine the longest common subsequence shared by different sets of one or more regular expression codes. By determining the context from the provided examples, a more accurate regular expression can be generated.

[0272] Figure 51 illustrates a flowchart of a method 5100 for determining context from negative examples, according to some exemplary embodiments. Figure 51 describes step 5040 of Figure 50 in more detail.

[0273] In step 5110, after a negative example is given, it is determined whether there are multiple highlights and whether the multiple highlights are nested (inner + outer highlights). As shown in Figure 51, the context of a negative example is not determined unless a negative example is given (e.g., step 5030 of Figure 50).

[0274] In step 5120, context is obtained from the data to the left of the negative example. Specifically, the context is obtained from the data to the left of where the negative highlighting is embedded. The embedded location of the negative highlighting may be determined prior to obtaining the context. Code for performing this step may include: val eLookBehindStart = rsslice(0, r.highlightSpans.min).foldRight(es.slice(0, highlightSpanRange.start).reverse)((sElem,esLookBehind) => esLookBehind.dropWhile(_.cc == sElem.cc)).length.

[0275] In one exemplary embodiment, the furthest left span is used. eLookBehindStart gets the span number of the leftmost span from the left walk (from the left edge of the negative highlight).

[0276] Step 5120 may be an iterative process. For example, filtering may be performed for each span to the left of where the negative highlighting is embedded. The highlighting of the fragment of the dataset may be removed for each span of the fragment that does not match a span of a negative example. With every iteration, the list of spans under consideration will be shortened.

[0277] In step 5130, context is obtained from the data to the right of the negative examples. Specifically, context is obtained from the data to the right of where the negative highlighting is embedded. Code for this step may include: val eLookAheadEnd = rsslice(r.highlightSpans.max+1, rslength).foldLeft(es.slice(highlightSpanRange.end, es.length))((esLookAhead,sElem) => esLookAhead.dropWhile(_.cc == sElem.cc)).length.

[0278] In one exemplary embodiment, the farthest right span is used. eLookAheadEnd gets the span number of the rightmost span from the right walk (from the right edge of the negative highlight).

[0279] Step 5130 may be an iterative process. For example, filtering may be performed for each span of the fragment to the right of where the negative highlight is embedded. The highlights of the fragments of the dataset may be removed for each fragment that does not match a span of the negative example. With every iteration, the list of spans under consideration will be shortened.

[0280] In step 5140, fragment filtering is performed: Highlighted fragments that do not correspond to the left and right contexts of the negative examples (eg, the contexts identified in steps 5120 and 5130) are removed.

[0281] Although contexts are used on the left and right sides of negative examples, for example, if only left context data or only right context data is present, only the left or right context may be used to identify the context for the negative examples. Also, while in the example described in Figure 51 the left context is identified before the right context, the right context may be identified before the left context. Additionally, although the above examples are described with respect to determining context for negative examples, context may also be determined with respect to positive examples.

[0282] Exemplary code for searching for contexts corresponding to negative examples in a nested manual mode may include some or all of the following code:

[0283]

number

[0284]

number

[0285] VII. Regular Expression Generation Using the Longest Common Subsequence Algorithm over Spans Further aspects described herein relate to the generation of regular expressions based on the LCS algorithm from one or more data input character sequences, however, the regular expression generator 110 may also handle characters that are present in only some of the examples. To handle characters that are present in only some of the input examples, spans may be defined in which both the minimum and maximum occurrences of a regular expression code are tracked. For example, for character sequence inputs of "9pm" and "9 pm", there is an optional space between the digits and the "pm" text. In such a case, if a certain span (e.g., a single space between "9" and "pm") may not be present in all of the given input examples, the minimum number of occurrences may be set to zero. These minimum and maximum numbers may then be mapped to a regular expression multiplicity syntax. A longest common subsequence (LCS) algorithm may be run on spans of characters derived from the input examples, including "optional" spans that do not appear in all input examples (e.g., a minimum length of zero). As described below, consecutive spans may be merged during the execution of the LCS algorithm. In such cases, when additional optional spans carried together end up occurring consecutively, the LCS algorithm may be run recursively on those optional spans as well. That is, while execution of the LCS algorithm is recursive in nature, in these cases the entire LCS algorithm may be run recursively (e.g., running the recursive LCS algorithm recursively). Among other technical advantages, this may enable shorter, cleaner, and more readable regular expression generation. For example, (am|am) (i.e., with optional whitespace before am) may be generated without running the LCS algorithm recursively, while running the LCS algorithm recursively may result in the regular expression being generated as the shorter, cleaner (?am).

[0286] FIG. 18 is a flow chart illustrating a process 1800 for generating a regular expression including optional spans using a Longest Common Subsequence (LCS) algorithm, according to one or more embodiments described herein. In step 1801, the regular expression generator 110 may receive as input data one or more character sequences corresponding to positive regular expression examples. In step 1802, the regular expression generator 110 may convert the character sequences into regular expression code. Thus, steps 1801 and 1802 may be similar or identical to the previous corresponding examples described above. Then, in step 1802, the regular expression code may be further converted into a span data structure (or spans). As described above, each span may include a data structure that stores a character class code (e.g., a regeex code) and a repeat count range (e.g., a minimum count and / or a maximum count). In step 1804, the regular expression generator 110 may execute the LCS algorithm to provide a set of spans as input to the algorithm. The output of the LCS algorithm in this example may include an output set of spans that includes at least one span that has a minimum repeat count range equal to zero that corresponds to the optional span in the output of the LCS algorithm. Finally, in step 1805, the regular expression generator 110 may generate a regular expression based on the output of the output of the LCS algorithm that includes the optional span.

[0287] In some embodiments, to improve readability and achieve an appropriate level of restriction, the regular expression generator 110 may use single span alternation. To illustrate, after the LCS algorithm is executed in step 1804, its output (i.e., intermediate output before the final conversion to a regular expression) may be a sequence of spans, each with a minimum and maximum repeat count. A simple direct conversion from a list of spans to a regular expression may result in multiple regular expression codes marked as optional (e.g., with a question mark "?" modifier code). In contrast, in some cases, it may be desirable to generate a regular expression that includes alternation, which may be expressed as vertical bar separated alternatives enclosed in parentheses (see https: / / www.regular-expressions.info / altemation.html). For example, a direct conversion of a span over time to a regular expression may be:

[0288]

number

[0289] However, in this example, the regular expression generator 110 converts the regular expression into a more readable

[0290]

number

[0291] To perform that conversion, the regular expression generator 110 may keep track of the original string fragments underlying the span, and may determine that if all original string fragments appear in at least two given positive examples, then a list of those original string fragments should be output as alternatives separated by vertical bars (e.g., rather than multiple optional regular expression codes).

[0292] FIG. 19 is an exemplary diagram illustrating the generation of a regular expression using a Longest Common Subsequence (LCS) algorithm, where the generated regular expression includes an optional span. In this example, two input data character sequences are "8am" and "9pm". The input data character sequences are first converted to regular expression codes (step 1802) and then converted to spans (step 1803), as described above. The spans may be provided as input to the LCS algorithm (step 1804), and the LCS output may include an optional span Z. <0, 1> to indicate that the optional single space can be a digit and a two-character text sequence. That is, the superscript notation in this example can include two digits, a minimum repeat count range (e.g., 0) and a maximum repeat count range (e.g., 1) that apply to the preceding code (e.g., Z=space). Finally, a regular expression may be generated based on the output span of the LCS algorithm, where the optional span is represented by the corresponding regular expression code "pZ * " may be converted to "

[0293] In some embodiments, the rendering and use of optional whitespace by regular expression generator 110 during the execution of the LCS algorithm may provide further technical advantages with respect to performance and readability. For example, when generating a regular expression, it may be desirable to be able to handle both characters that are common among all given examples and characters that are present in only some of those examples.

[0294] In one embodiment, for each span data structure, both the minimum number of occurrences of the category code and the maximum number of occurrences of the category code may be tracked. If no spans are present in one or more of the given examples, the minimum is set to zero. As another example, to generate a regular expression for handling spelled out months, the minimum and maximum numbers may then be mapped to a regular expression multiplicity syntax with curly braces (e.g., [A-Za-z]{3,9}).

[0295] In some embodiments, regular expression generator 110 may keep track of the minimum and maximum occurrences for each span, but may also handle additional implementation details. For example, as a result of the combination of handling optional spans and performing LCS on spans of characters, regular expression generator 110 may be configured to detect and merge consecutive spans through the execution of the LCS algorithm. In addition, it may be desirable for any additional optional spans carried together to appear consecutively from time to time and for the LCS algorithm to be performed recursively on them as well. For example, in some cases, regular expression generator 110 modifies and / or extends the LCS algorithm to favor (or weight) fewer transitions between optional and required sequence elements (e.g., spans). For example, grouping optional spans together may minimize the number of grouping parentheses that must be used within a regular expression, thus improving human readability of the generated regular expression. In some cases, if the resulting lengths are equal even after considering optional spans, regular expression generator 110 may indicate a preference for alternatives with fewer transitions between optional and required spans. For example, in some cases, a standard LCS algorithm may be implemented to favor the selection of the longer sequence at its decision point. However, at decision points where the alternatives are of equal length, a configuration preference may be programmed into regular expression generator 110. One such configuration preference may be, for example, to favor the shorter sequence (once optional spans are considered). Thus, a customized LCS in this configuration may simultaneously optimize longer sequences (of required spans) and shorter sequences (of the sum of required and optional spans).

[0296] In some embodiments, the regular expression generator 110 implementation of the LCS algorithm may be configured to select a shorter output. As mentioned above, the LCS algorithm may be used generally to find the longest common subsequence. For generation of regular expressions using spans, the LCS algorithm may be used to identify the longest sequence of required sequence elements (in our case, the required spans). However, in embodiments including optional spans, the LCS algorithm may be configured to preserve the original goal of maximizing the number of common sequence elements (e.g., spans) while simultaneously minimizing the number of optional spans. That is, in the original LCS algorithm, an arbitrary choice may be made as to whether to consume sequence elements (e.g., spans) from the first example string or the second example string (with a view to maximizing the number of common sequence elements). In such a case, the regular expression generator 110 may implement a modified version of the LCS algorithm that may select the one that results in a shorter overall regular expression length when optional spans are also taken into account. The shorter the regular expression in the final result, the better the readability.

[0297] Furthermore, in some embodiments, regular expression generator 110's implementation of the LCS algorithm may be configured to favor a greater number of required spans, i.e., when given a choice, regular expression generator 110 may select an LCS output with fewer optional spans to improve human readability.

[0298] In some embodiments, the generated regular expression may be more readable if it starts with a required span (which may also serve as a mental anchor for a human reader) rather than starting the regular expression with an optional span. Thus, in some cases, if the resulting choices have an equal number of transitions, the choice with the earlier non-optional span may be selected. Additionally, the LCS algorithm performed by the regular expression generator 110 may be configured in some embodiments to push all whitespace (including optional spans corresponding to whitespace) to the right in the regular expression. By pushing all whitespace to the right, the chances that spans of whitespace may be merged together may be increased, which may simplify the resulting regular expression and improve readability. Thus, during execution of the LCS algorithm, if it is determined that two sets of substrings have the same LCS, instead of arbitrarily selecting one of the two sets of substrings, the set that facilitates improved readability may be selected. Additionally, in some embodiments, the LCS algorithm may be configured to favor a greater number of required spans and / or a smaller number of optional spans to improve readability.

[0299] As mentioned above, in some cases, the negative examples may be based on an optional span. For example, a user may provide positive examples of "ab" and "a2b" and a negative example of "a3b". In this case, an example implementation may fail because it may attempt to distinguish based only on the required span, the "2" digit being in the optional span. In such a case, the user may be alerted to the failure and provided with options via a user interface to manually repair the generated regular expression and / or remove some of the negative examples.

[0300] In some embodiments, there may be an isSuccess returned as part of the JSON coming back from the REST service. In some embodiments, the generated regeex may be a different color (e.g. red) when isSuccess=false.

[0301] As mentioned above, regular expression generator 110 can use single span alternation in some cases. In addition, in some embodiments, regular expression generator 110 can be configured to perform multi-span alternation. That is, sometimes some spans in a line may

[0302]

number

[0303] To detect these, regular expression generator 110 may first discover which spans from the output of the LCS algorithm are actually used in all of the given input positive examples. These common spans, along with the phantom span before the first span and the phantom span after the last span, may constitute the anchor points. Between every pair of consecutive anchor points (that also have at least one non-anchor span between them), there is a "bridge" that must be crossed. Then, for each bridge, regular expression generator 110 may (a) determine the string fragments covered by the bridge span for each input positive example, (b) convert each string fragment into a regular expression using techniques described herein, and then (c) remove duplicates from the above and put them into a regular expression alternation syntax.

[0304] Instead of alternatives within an alternation appearing in any order, they could also be sorted alphabetically. By having a more deterministic output, the generated regex may change less with slight changes in the input examples, such as during typing when the regex is generated in real time with each character the user types. Furthermore, some advanced regex interpreters can explicitly use a greedy vs. non-greedy strategy when encountering an alternation, depending on whether a greedy quantifier is present. However, other regex interpreters, such as those found in standard Java and Javascript APIs, may simply try alternatives within an alternation in the order in which they appear. To compensate for this commonly found weakness, the regex generation may output longer alternatives within an alternation first. Also, alphabetical order may be used as a secondary sorting criterion to break any ties.

[0305] In some embodiments, the regular expression generator can be configured to collapse spans that appear in a small number of positive examples to a non-greedy wildcard .*?, i.e., the input examples may be a bit verbose. The regular expression generator may use the wildcard .*? when it detects significant variance in the number of spans and original fragment content, rather than trying to match every single word and their specific word length. In some embodiments, for a span to be considered for participation in such collapsing to the wildcard .*?, etc., the span must meet one of the following three conditions: (1) the span may contribute (strictly) less than one-third of the total number of input examples, (2) the span is a SPACE span, and (3) all of the original text fragments for the span occur rarely and there is a "unique punctuation" in the solution. "Rarely" may mean (strictly) less than two-thirds of the number of examples. "Unique punctuation" means that there is a SYMBOL or PUNCTUATION span somewhere in the solution, or the solution is positioned to the beginning of the text with ^ or to the end of the text with $.

[0306] In some cases, the eol wildcard may be prevented from touching a hidden wildcard inside the highlighting. That is, recall that when ^ or $ are used, the wildcard appears at the opposite end. For example, ^([AZ]+).*? matches the first word. Indiscriminate hiding of wildcards may end up with something like ^(.*?).*?, which would give the Regex interpreter nothing concrete to match against. To solve this, if ^ is used and the last span in the capture group is part of a hide to wildcard .*?, one extra span after the capture group is used in the output before the end of line wildcard is emitted to be used for the negative character class that touches the wildcard. Similarly, if $ is used and the first span in the capture group is part of a hide to wildcard, one extra span before the capture group is used explicitly in the output.

[0307] In some embodiments, symbols and punctuation marks may no longer be spannable, for example to allow a highlighting endpoint to come between two punctuation marks. In such an example, each punctuation mark and each symbol may take up its own PUNCTUATION or SYMBOL span. In other cases, "A.,S" might result in three spans, LETTER (with original fragment A), PUNCTUATION with original fragment (.,), and LETTER (with original fragment S). However, in these embodiments, there may be four spans. In this example, it may be useful if the comma acts as a delimiter in a comma-separated list, especially if there are multiple user highlights per example. In this example, the dot (period) may be redundant and should not be in the same span as the comma.

[0308] Additionally, for spans of type ALPHANUMERIC, the regular expression generator can track how many letter and number spans have been replaced. Since many aspects of the algorithm, such as determining the beginning and end of the inner highlighting in the overall solution, keep track of positions by span index, replacing multiple LETTER and NUMBER spans with a single ALPHANUMERIC span throws away these indexes and positions. It may therefore be necessary to maintain a mapping from pre-substitution indexes to post-substitution indexes. Furthermore, these need to be tracked on a per-example basis, since each example will not, on average, contain all indices from the overall solution (including optional spans for spans that are not represented in 100% of the examples). A class member numPreSubstitutedSpans has been added to the Span class, with the type and default value as follows: numPreSubstitutedSpans:Map[ID,Int]=orig.flatMap(_.fromExamples).distinct.map((_,1)).toMap In some embodiments, publicly available Regex APIs can be used to find the end offset of a capture group. For example, as described in connection with a replace command that may be implemented in some user interfaces, when using the replace command in nested auto outer mode, an additional capture group may be generated after the regular capture group (similar to the previous one, for a total of three). The user interface may perform this replacement, and similarly to above, techniques may be employed to find the right parenthesis of a capture group to avoid having to implement a full parser in the user interface. This may be more complicated than finding the left parenthesis of a capture group, because it may be necessary to find the right parenthesis of a capture group ( <stufff>) before the right parenthesis of a non-capturing group (?: <stuff>), because there is no easy way to distinguish it from a ?:. Distinguishing a left parenthesis may possibly be possible by using a lookahead to look for ?:. In some embodiments, a technique may be employed that requires the search to have an example of something that is known to match the regular expression. By comparing the captured groups concatenated together with the original example text, it may be ensured that the regular expression is completely covered with capturing groups without any gaps. The code for this technique is shown below:

[0309]

number

[0310] VIII. Regular Expression Generation Using a Combinatric Longest Common Subsequence Algorithm Further aspects described herein relate to combinatorial searches in which the LCS algorithm performed by regular expression generator 110 may be run multiple times to generate a "correct" regular expression (e.g., a regular expression that properly matches all given positive examples and properly filters out all given negative examples) and / or to generate multiple correct regular expressions from which the most desirable or optimal regular expression may be selected. For example, during a combinatorial search, the entire LCS algorithm and regular expression generation process may be run multiple times, including different combinations / permutations of text processing directions, different positioning, and other different characteristics of the LCS algorithm.

[0311] FIG. 20 is a flow chart illustrating a process 2000 for generating a regular expression based on a combinatorial execution of the Longest Common Subsequence (LCS) algorithm. In step 2001, the regular expression generator 110 may receive an input data character sequence corresponding to a positive example. In step 2002, the regular expression generator 110 may iterate over a variety of different combinations of execution techniques for the LCS algorithm. As shown in this example, during each iteration of step 2002, the regular expression generator 110 may select a different combination of the following LCS algorithm execution parameters (or characteristics): locator (i.e., no positioning, positioning at the beginning of a line, positioning at the end of a line), processing direction (i.e., right-to-left order, left-to-right order), whitespace pushing (i.e., with or without whitespace pushing), and span collapse (i.e., with or without collapse of spans). In step 2003, an LCS algorithm is run on the input data character sequence (or on the regular expression code if the input character sequence was first transformed), and the LCS algorithm is configured based on the parameters / characteristics selected in step 2002. In step 2004, the output of the LCS algorithm may be stored by the regular expression generator 110 and may include data such as whether an LCS was successfully identified by the algorithm and the length of the corresponding regular expression. In step 2005, the process may iterate until the LCS algorithm has been run with all possible combinations of the parameters / characteristics of the combinatorial search. Finally, in step 2006, a particular output from one of the LCSs may be selected as the optimal output (e.g., based on success and regular expression length), and a regular expression may be generated based on the selected LCS algorithm output.

[0312] In various embodiments, combinatorial searches such as those described above with reference to FIG. 20 may be performed on various different combinations of parameters / characteristics. For example, in some embodiments, the LCS algorithm may use a caret symbol ^ to locate a regular expression to the beginning of the text and / or a dollar symbol $ to locate a regular expression to the end of the text. In some cases, such locate may result in generating a shorter regular expression. Locators may be particularly useful when a user wants to find a particular pattern at the beginning and / or end of a string. For example, a user may want a product name at the beginning. To avoid confusing the LCS algorithm with a variable number of words that describe the product name, a caret may be used to locate the regular expression to the beginning of the string, as shown in the image below.

[0313] Additionally, in some embodiments, the LCS algorithm may be run with input data that is either forward or reverse (or equivalently, the LCS algorithm may be configured to receive input data in normal order and then reverse the order before running the algorithm). Thus, in some embodiments, a combinatorial search of the LCS algorithm that may be run against an input character sequence or code pair may be as follows:

[0314] 1. Normal (right-to-left) ordering, no positioning relative to the start or end 2. Normal (right-to-left) order, using the caret ^ to position relative to the beginning of the line 3. Normal (right-to-left) ordering, using dollar to position to the end of the line 4. Reverse (left-to-right) order, no positioning to the beginning or end 5. Reverse (left to right) order, use the caret ^ to position to the beginning of the line 6. In reverse (left to right) order, use dollar $ to position to the end of the line In this example, out of the six runs of the LCS, the shortest resulting regular expression may be selected (step 2006).

[0315] In some embodiments, the combinatorial search of the LCS algorithm may iterate over the greedy quantifier "?" and the non-greedy quantifier "??". For example, by default, if there is an optional span, a single question mark is issued, e.g., for a first name and last name with an optional middle initial:

[0316]

number

[0317] If no satisfactory regular expression is found when using greedy quantifiers, the combinatorial search replaces all question mark quantifiers with double question mark quantifiers (e.g.

[0318]

number

[0319] The double question mark corresponds to a non-greedy quantifier, which can instruct downstream regular expression matchers to enter backtracking mode in order to find a match.

[0320] Additionally, in some embodiments, the combinatorial search of the LCS algorithm may also iterate over whether to prefer whitespace on the right. For example, in some embodiments that push whitespace to the right, as described above, a strategy may be used in the hope that, for example, when the LCS algorithm is faced with an arbitrary choice of otherwise equal options, the whitespace spans will be merged together, resulting in a smaller number of overall spans. This feature adds another option to the combinatorial search, i.e., to either push whitespace to the right or to perform according to a traditional LCS approach that leaves the decision arbitrary.

[0321] Additionally, in some embodiments, the combinatorial search of the LCS algorithm may also iterate to scan / unscan for literals common to all examples by running LCS on the original string. In such embodiments, the LCS algorithm may be configured to identify and align common words. As used herein, a "common word" may refer to a word that appears in all positive examples. Once a common word is identified, its span type may be converted from LETTER to WORD, and then subsequent runs through the LCS algorithm may naturally align to it.

[0322] Thus, in the following example, the combinatorial search may be iterated over several parameters / properties, amounting to 96 runs of the full LCS algorithm. The various parameters / properties to be iterated over in this example are as follows: Positional specifier (3) (value = ^, $, or neither) Push Blank (2) (value = Yes or No) · Merge low density spans into wildcards (2) (value = Yes or No) · Greedy quantifier? (2) (value = Yes or No) · Alignment on common tokens in LCS algorithm (2) (value = Yes or No)

[0323]

number

[0324] As mentioned above, in this example, the complete LCS algorithm is executed 96 times (eg, 3*2*2*2*2*2=96).

[0325] However, in other embodiments, the regular expression generator 110 may provide a performance boost whereby only the first three properties from the list above (positional specifiers, pushing whitespace, and coalescing low-cardinality spans into wildcards) may participate in the combinatorial search. This may result in a much smaller number of full LCS algorithms being executed (e.g., 3*2*2=12 times). In such an embodiment, only the last three properties from the list above (greedy quantifiers, alignment on common tokens in the LCS algorithm, and

[0326]

number

[0327] ) do not participate in the combinatorial search, but these properties may be tested individually and sequentially at the end. Technical advantages may be realized in such embodiments because dividing the search space in this way may still result in a satisfactory regular expression being found, with approximately an 8x speedup in performance.

[0328] To illustrate, the following example of a combinatorial search may provide a performance advantage over the previous example. In this example, the combinatorial search may be performed based on the following parameters / characteristics to be iterated over: Positioning (3): BEGINNING_OF_LINE_MODE (beginning of line mode), END_OF_LINE_MODE (end of line mode), NO_EOL_MODE (no end of line mode) Order / Direction (2): Right-to-Left (normal) LCS vs. Left-to-Right (reverse) LCS · push(2): whether to try to push whitespace to the right in the LCS algorithm · Compress to Wildcards (2): Compress long sequences of spans that only occur occasionally into wildcards. * Whether to compress to ? The combinatorial trick in this example may result in running the full algorithm 3*2*2*2=24 times. Regular expression generator 110 may then take the best of the 24 results of the LCS algorithm, where "best" may mean that (a) the LCS algorithm was successful, and (b) the shortest regular expression was generated. Regular expression generator 110 may then perform three additional tasks: 1. A sequence of letters and numbers that is uninterrupted by spaces, punctuation, or symbols.

[0329]

number

[0330] This can be useful for hexadecimal digits found in IPv6 addresses from clickstream logs. 2. Try using the non-greedy quantifier ?? instead of the greedy quantifier ?. 3. Try to align on the literal.

[0331] A. Span highlight alignment A positive or negative example may include multiple highlights. Each example provided may include multiple highlights (e.g., outer and inner highlights). Thus, the exemplary embodiments provide a method for efficient and accurate processing when multiple examples each have their own highlights.

[0332] Each example is broken down into pre-highlighting, on-highlighting, and post-highlighting. The full algorithm is run on each set, with each set including pre-highlighting, on-highlighting, and post-highlighting. User highlighting on multiple examples is supported even with a large degree of variability between the examples provided, and highlighting failures are avoided.

[0333] FIG. 52 illustrates a diagram 5200 for generating a regular expression based on span highlight alignment, according to some example embodiments.

[0334] FIG. 52 illustrates performing highlight alignment on two data examples, "Jane Doe" 5210 and "David Williams Jr." 5220. As shown in FIG. 52, the last names of each entry (e.g., Doe and Williams) are highlighted. For example, a user may select "Doe" as an example in a first record and "Williams" as an example in another data record. For ease of explanation, two examples are shown. However, highlight alignment can be performed on multiple data examples.

[0335] In the example shown in FIG. 52, regex is generated in parts (e.g., three parts). There could be more or less parts depending on the information in the data cells of the data record. Regex can be generated for all of the "before highlight" fragment, the "highlight above" fragment, and the "after highlight" fragment. A first regex 5231 is generated for the "before highlight" fragment, a second regex 5232 is generated for the "highlight above" fragment, and a third regex 5233 is generated for the "after highlight" fragment.

[0336] The generated regulex spans (i.e., intermediate results before the final regulex generation) are concatenated together. The generated regulex spans for the parts (e.g., the pre-highlight fragment, the above-highlight fragment, and the post-highlight fragment) are concatenated together.

[0337] 53 shows a flowchart of a method 5300 for performing span highlight alignment according to some exemplary embodiments. The steps of FIG. 53 may be performed by a regular expression generator such as the regular expression generators shown in FIG. 1 and FIG. 24.

[0338] In step 5310, an initial highlight selection may be accepted. For example, a user may select a text fragment from a first record in a column of data. In the example shown in FIG. 52, the user has highlighted last name 5210 "Doe" in the name column. The selected fragment may be identified as an "above highlight" fragment. The fragment before the selected fragment may be identified as a "before highlight" fragment. The fragment after the selected fragment may be identified as a "after highlight" fragment.

[0339] At step 5320, a second highlight selection may be accepted. For example, the user may select a text fragment from a second record in the same data column (e.g., the name column) as the first highlight selection. In the example shown in FIG. 52, the last name 5220 "Williams" in the name column is highlighted. In the second example shown in FIG. 52, the user has also highlighted a last name. The selected fragment may be identified as an "over highlight" fragment. The fragment before the selected fragment may be identified as a "before highlight" fragment. The fragment after the selected fragment may be identified as an "after highlight" fragment. The user may wish to provide multiple examples, and thus the user may provide additional highlighted selections. However, the examples provided may not be clearly aligned.

[0340] In step 5330, alignment of the fragments is performed. Fragments occurring before the highlighting are aligned together, fragments occurring after the highlighting are aligned together, and fragments after the highlighting are aligned together. Thus, in the example shown in Figure 52, fragment 5211 containing "Jane" and fragment 5221 containing "David" are aligned because they occur before the highlighted fragment. Fragment 5210 "Doe" and fragment 5220 "Williams" are aligned together because they occur on the highlighting. Fragment 5215, which does not contain any characters, and fragment 5222 are aligned because they occur after the highlighting.

[0341] In step 5340, a regex is generated for the "before highlight" fragment. For example, a first regex 5231 may be generated. The first regex 5231 may be generated by a regex generator based on the data before the highlighted fragment.

[0342] In step 5350, regex for the "highlighted" fragment is generated. For example, a second regex 5231 may be generated. A second regex 5232 may be generated by a regex generator based on the data in the highlighted fragment.

[0343] In step 5360, regex for the "post-highlight" fragment is generated. For example, a third regex 5231 may be generated. A third regex 5233 may be generated by the regex generator based on the data after the highlighted fragment.

[0344] As shown in Figure 53, regex is generated in three parts 5231, 5232, 5233. A first regex 5231 is generated for the "before highlight" fragment, a second regex 5232 is generated for the "on highlight" fragment, and then a third regex 4233 is generated for the "after highlight" fragment. Although an order for generating the first, second, and third regex has been described, the order for generating the regex may vary. Additionally, although three parts have been described, highlighting may be performed on more than three parts of the example data.

[0345] In step 5370, the spans for the three regexxes are concatenated. That is, instead of concatenating the three resulting regexxes together, the spans (i.e., intermediate results before the final regexx generation) are concatenated together.

[0346] The longest common subsequence for the three spans concatenated is determined in step 5380. The LCS may be determined for the output of running an LCS algorithm on the pre-highlight span, the above-highlight span, and the post-highlight span.

[0347] Thus, the illustrative embodiments provide a more accurate method of determining the longest common subsequence because the longest common subsequence is determined based on similarly located data.

[0348] An exemplary embodiment also provides an alphanumeric span. By determining the start and end of the inner highlight, keeping track of the position by the span index, and replacing multiple letter and number spans with a single alphanumeric span, the index and position can be discarded. Thus, the exemplary embodiment provides a mapping from pre-replacement index to post-replacement index. A mapping is generated that associates the index of the alphanumeric span with the index of one or more numbers and one or more letters.

[0349] Furthermore, since each example may not contain all of the indices from the global solution, the mapping is tracked on an example-by-example basis.A number of pre-permuted spans can be identified.

[0350] 54 shows a flowchart of a method 5400 for tracking spans, according to some exemplary embodiments. Specifically, the exemplary embodiments provide a method for determining how many letter spans and how many number spans of an alphanumeric span are replaced.

[0351] In step 5410, one or more character spans and one or more number spans are replaced with a single alphanumeric span. For example, an HTML hex color code can be replaced with a single alphanumeric span instead of three different spans, such as a character span and a number span. BUY NOW! In an HTML hexadecimal color code such as , a single alphanumeric span, FF030A, is used instead of three different spans (e.g., a letter span, a number span, and another letter span).

[0352] Some example data fragments may contain hex "digits" A-F, which may occur anywhere within the hex code, so an alphanumeric span may be used to easily identify matches for such example data fragments.

[0353] In step 5420, an example data fragment is identified that includes an alphanumeric span that replaces one or more character spans and one or more numeric spans.

[0354] A mapping is generated that identifies pre-substitution indexes and post-substitution indexes for the data fragment example, at step 5430. A mapping is generated for each data fragment example.

[0355] In step 5440, a number of pre-permuted spans may be identified. 55 illustrates a user interface 5500 displaying punctuation spans and symbol spans, according to some exemplary embodiments. As shown in FIG. 55, a user has provided specified examples 5533. The user specified examples include examples 5534 and 5535. Regex 5532 is generated based on specified examples 5533.

[0356] As shown in FIG. 55, symbols and punctuation marks are no longer spannable such that a highlighting endpoint can come between two punctuation marks.

[0357] Each punctuation mark and each symbol can have its own punctuation span or symbol span. For example, an example containing "A.,S" would contain the four spans "A", ".", "," and "S".

[0358] Exemplary embodiments provide improved regeex generation aesthetics. Specifically, fragments can be associated with the examples that gave rise to them. Fragments are associated back to the examples that gave rise to them. In one exemplary embodiment, the examples that gave rise to a fragment can be added to a list of example numbers.

[0359] Additionally, certain exemplary embodiments provide an even more detailed way of associating a fragment with the example that gave rise to it. An example can be tied to a particular text fragment, rather than simply to the span object as a whole. In this way, the user can more easily determine which examples are tied to a fragment, since the user can see the example itself, instead of just seeing the example number in a list of example numbers.

[0360] For example, for the two input examples below, Example 1: "a#s#" and Example 2: "a#", the solution spans could include [(LETTER, [("a",[1,2])]), (SYMBOL, [("#",[1,2])]), (LETTER, ["s",[1]]), (SYMBOL, ["#",[1]])]. By having the example numbers available for each individual fragment, the algorithm can walk back and forth (left or right) when searching for the start and / or end of the highlight (or beyond the highlight to establish context) when processing a particular example. Each example can be decomposed into spans, and an LCS merging of those spans into its own set of spans is performed.

[0361] For two other input examples, Example 1: "8pm" and Example 2: "9am", the solution span could include: [(NUMBER, [("8",[1]),("9",[2])]), (LETTER, [("pm",[1]),("am",[2])]), (PUNCTUATION, [(".",[1,2])])]. Thus, a fragment is associated with the particular example that gave rise to it.

[0362] In another example, the Regex generated aesthetic may include that spans are not merged unless they come from the same set of examples. To facilitate matching, in an example embodiment, spans may not be merged unless they come from the same set of examples.

[0363] Thus, embodiments of the present invention provide various mechanisms for generating precise regular expressions that can be applied to data to obtain data results desired by users, allowing users to more easily and efficiently obtain desired information without extensive searching and data manipulation.

[0364] IX. Hardware Overview 21 shows a simplified diagram of a distributed system 2100 for implementing an embodiment. In the illustrated embodiment, the distributed system 2100 includes one or more client computing devices 2102, 2104, 2106, and 2108 coupled to a server 2112 via one or more communication networks 2110. The client computing devices 2102, 2104, 2106, and 2108 may be configured to run one or more applications.

[0365] In various embodiments, the server 2112 may be adapted to execute one or more services or software applications that enable automated generation of the regular expressions described in this disclosure. For example, in a particular embodiment, the server 2112 may receive user input data sent from a client device, the user input data being received by the client device via a user interface displayed on the client device. The server 2112 may then convert the user input data into a regular expression that is sent to the client device for display via the user interface.

[0366] In particular embodiments, the server 2112 may also provide other services or software applications, which may include non-virtualized and virtualized environments. In some embodiments, these services may be provided as web-based or cloud services, such as in a Software as a Service (SaaS) model, to users of the client computing devices 2102, 2104, 2106, and / or 2108. Users operating the client computing devices 2102, 2104, 2106, and / or 2108 may utilize one or more client applications to interact with the server 2112 to utilize the services provided by these components.

[0367] In the configuration shown in Figure 21, the server 2112 may include one or more components 2118, 2120, and 2122 that implement the functions performed by the server 2112. These components may include software components that may be executed by one or more processors, hardware components, or a combination thereof. It should be appreciated that a wide variety of system configurations are possible that may differ from the distributed system 2100. Thus, the embodiment shown in Figure 21 is an example of a distributed system for implementing the system of the embodiments and is not intended to be limiting.

[0368] A user may use client computing devices 2102, 2104, 2106 and / or 2108 to run one or more applications that generate regular expressions according to the teachings of this disclosure. The client devices may provide an interface that allows a user of the client device to interact with the client device. The client devices may also output information to the user via the interface. Although FIG. 21 shows only four client computing devices, any number of client computing devices may be supported.

[0369] Client devices may include various types of computing systems, such as portable handheld devices, general purpose computers such as personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computing devices may include various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh®, UNIX® or UNIX-like operating systems, Linux® or Linux-like operating systems, e.g., Google Chrome® OS, including various mobile operating systems (e.g., Microsoft Windows Mobile®, iOS®, Windows Phone®, Android®, BlackBerry®, Palm OS®). Portable handheld devices may include cellular phones, smartphones (e.g., iPhone®), tablets (e.g., iPad®), personal digital assistants (PDAs), etc. Wearable devices may include Google Glass® head-mounted displays and other devices. The gaming systems may include various handheld gaming devices, Internet-enabled gaming devices (e.g., Microsoft Xbox® gaming consoles with or without Kinect® gesture input devices, Sony PlayStation® systems, various gaming systems offered by Nintendo®, etc.) The client devices may be capable of running a wide variety of applications, such as various Internet-related applications, communication applications (e.g., email applications, short message service (SMS) applications), and may use a variety of communication protocols.

[0370] Network 2110 may be any type of network known to those skilled in the art capable of supporting data communications using any of a variety of available protocols, including, but not limited to, TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internet Packet Exchange), AppleTalk, etc. By way of example only, network 2110 may include a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., a wireless network operating under any of the Institute of Electrical and Electronics Engineers (IEEE) 802.11 protocol suite, Bluetooth, and / or any other wireless protocol), and / or any combination of these and / or other networks.

[0371] The servers 2112 may be comprised of one or more general purpose computers, dedicated server computers (including, by way of example, PC (personal computer) servers, UNIX servers, mid-range servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or other suitable configurations and / or combinations. The servers 2112 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization, such as one or more flexible pools of logical storage that can be virtualized to maintain virtual storage for the servers. In various embodiments, the servers 2112 may be adapted to run one or more services or software applications that provide the functionality described in the disclosure above.

[0372] The computing systems in the servers 2112 may run one or more operating systems, including any of the operating systems listed above, as well as commercially available server operating systems. The servers 2112 may also run any of a variety of other server and / or mid-tier applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, JAVA servers, database servers, and the like. Exemplary database servers include, but are not limited to, those commercially available from Oracle®, Microsoft®, Sybase®, IBM® (International Business Machines), and the like.

[0373] In some implementations, the server 2112 may include one or more applications for parsing and consolidating data feeds and / or event updates received from users of the client computing devices 2102, 2104, 2106, and 2108. By way of example, the data feeds and / or event updates may include, but are not limited to, Twitter® feeds, Facebook® updates, or real-time updates received from one or more third party sources and continuous data streams that may include real-time events related to sensor data applications, financial stock ticker boards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc. The server 2112 may also include one or more applications for displaying the data feeds and / or real-time events via one or more display devices of the client computing devices 2102, 2104, 2106, and 2108.

[0374] The distributed system 2100 may also include one or more data repositories 2114, 2116. In certain embodiments, these data repositories may be used to store data and other information. For example, one or more of the data repositories 2114, 2116 may be used to store information such as new strings of data that match regular expressions generated by the system. The data repositories 2114, 2116 may be in a variety of locations. For example, the data repository used by the server 2112 may be local to the server 2112 or may be remote from the server 2112 and communicate with the server 2112 via a network-based or dedicated connection. The data repositories 2114, 2116 may be of different types. In certain embodiments, the data repository used by the server 2112 may be a database, for example, a relational database such as those provided by Oracle Corporation and other manufacturers. One or more of these databases may be adapted to allow data to be stored, updated, and retrieved from the database in response to SQL-formatted commands.

[0375] In particular embodiments, one or more of the data repositories 2114, 2116 may be used by an application to store application data. The data repositories used by an application may be of various types, such as, for example, a key-value store repository, an object store repository, or a general-purpose storage repository supported by a file system.

[0376] In certain embodiments, the functionality described in this disclosure may be provided as a service via a cloud environment. FIG. 22 is a simplified block diagram of a cloud-based system environment that may provide various services as cloud services, according to a particular example. In the example shown in FIG. 22, a cloud infrastructure system 2202 may provide one or more cloud services that users may request using one or more client computing devices 2204, 2206, and 2208. The cloud infrastructure system 2202 may include one or more computers and / or servers, which may include those previously described with respect to the server 2112. The computers in the cloud infrastructure system 2202 may be organized as general purpose computers, dedicated server computers, server farms, server clusters, or any other suitable arrangement and / or combination.

[0377] The network 2210 may facilitate communication and exchange of data between the clients 2204, 2206, and 2208 and the cloud infrastructure system 2202. The network 2210 may include one or more networks. The networks may be of the same or different types. The network 2210 may support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0378] The example shown in Figure 22 is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be understood that in other examples, cloud infrastructure system 2202 may have more or fewer components than those shown in Figure 22, may combine two or more components, or may have components in a different configuration or arrangement. For example, while Figure 22 shows three client computing devices, in alternative examples, any number of client computing devices may be supported.

[0379] The term cloud services is generally used to refer to services made available to users on demand via a communication network such as the Internet by a service provider's system (e.g., cloud infrastructure system 2202). Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from the customer's own on-premise servers and systems. The cloud service provider's systems are managed by the cloud service provider. Thus, customers can use cloud services provided by the cloud service provider without having to purchase separate licenses, support, or hardware and software resources for the services. For example, the cloud service provider's system may host applications, and users may order and use the applications on demand via the Internet without having to purchase infrastructure resources to run the applications. Cloud services are designed to provide easy and scalable access to applications, resources, and services. Several providers offer cloud services. For example, several cloud services, such as middleware services, database services, and Java cloud services, are offered by Oracle Corporation of Redwood Shores, California.

[0380] In particular embodiments, cloud infrastructure system 2202 may provide one or more cloud services using a variety of models, such as a Software as a Service (SaaS) model, a Platform as a Service (PaaS) model, an Infrastructure as a Service (IaaS) model, etc., including hybrid service models. Cloud infrastructure system 2202 may include a suite of applications, middleware, databases, and other resources that enable the provisioning of various cloud services.

[0381] The SaaS model allows applications or software to be delivered as a service to customers over a communications network such as the Internet without the customer having to purchase the hardware or software for the underlying application. For example, the SaaS model may be used to allow customers to access on-demand applications hosted by the cloud infrastructure system 2202. Examples of SaaS services offered by Oracle Corporation include, but are not limited to, various services for human resource / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, and social applications.

[0382] The IaaS model is commonly used to provide flexible computing and storage capabilities by providing infrastructure resources (e.g., servers, storage, hardware and networking resources) to customers as cloud services. Various IaaS services are offered by Oracle Corporation.

[0383] The PaaS model is generally used to provide platform and environment resources as a service that enable customers to develop, run, and manage applications and services without having to procure, build, or manage the environment resources. Examples of PaaS services provided by Oracle Corporation (registered trademark) include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), data management cloud services, various application development solution services, etc.

[0384] Cloud services are typically provided on an on-demand, self-service basis, on a subscription basis, and in a flexible, scalable, reliable, highly available, and secure manner. For example, a customer may order one or more services provided by cloud infrastructure system 2202 via a subscription order. Cloud infrastructure system 2202 then provides the services requested in the customer's subscription order by performing processing. Cloud infrastructure system 2202 may be configured to provide one or more cloud services.

[0385] Cloud infrastructure system 2202 may provide cloud services through a variety of deployment models. In a public cloud model, cloud infrastructure system 2202 may be owned by a third-party cloud service provider and cloud services are provided to general public customers. The customers may be individuals or businesses. In a private cloud model, cloud infrastructure system 2202 may function within an organization (e.g., within a corporate organization) and services are provided to customers within the organization. For example, the customers may be various departments of a company, such as human resources, payroll, etc., or may be individuals within a company. In a community cloud model, cloud infrastructure system 2202 and the services provided may be shared among various organizations within an associated community. Various other models may be used, including hybrids of the above models.

[0386] Client computing devices 2204, 2206, and 2208 may be of different types (e.g., devices 2102, 2104, 2106, and 2108 shown in FIG. 21) and may be capable of operating one or more client applications. Users may use the client devices to interact with cloud infrastructure system 2202, such as to request services provided by cloud infrastructure system 2202.

[0387] In some embodiments, the processing performed by cloud infrastructure system 2202 to provide management-related services may include big data analytics. This analytics may include using, analyzing, and processing large data sets to detect and visualize various trends, behaviors, relationships, and the like within the data. This analytics may be performed by one or more processors, possibly processing the data in parallel, running simulations with the data, and the like. For example, big data analytics may be performed by cloud infrastructure system 2202 to determine regular expressions in an automated manner. The data used for this analytics may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)).

[0388] 22, cloud infrastructure system 2202 may include infrastructure resources 2230 that are utilized to facilitate the provision of various cloud services offered by cloud infrastructure system 2202. Infrastructure resources 2230 may include, for example, processing resources, storage or memory resources, networking resources, etc.

[0389] In certain embodiments, to facilitate efficient provisioning of these resources to support the various cloud services offered by cloud infrastructure system 2202 for different customers, resources may be organized into resource sets or resource modules (also referred to as "pods"). Each resource module or pod may include a pre-integrated and optimized combination of one or more types of resources. In certain embodiments, different pods may be pre-provisioned for different types of cloud services. For example, a first set of pods may be provisioned for database services, a second set of pods may be provisioned for Java services, etc., which may include a different combination of resources than the pods in the first set of pods. For some services, resources allocated to provision these services may be shared between the services.

[0390] Cloud infrastructure system 2202 itself may use services 2232 internally that are shared by different components of cloud infrastructure system 2202 and that facilitate provisioning of services by cloud infrastructure system 2202. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelist services, high availability, backup and recovery services, services enabling cloud support, email services, notification services, file transfer services, etc.

[0391] Cloud infrastructure system 2202 may include multiple subsystems. These subsystems may be implemented in software, or hardware, or a combination thereof. As shown in FIG. 22, the subsystems may include a user interface subsystem 2212 that allows users or customers of cloud infrastructure system 2202 to interact with cloud infrastructure system 2202. User interface subsystem 2212 may include a variety of different interfaces, such as a web interface 2214, an online store interface 2216 through which cloud services offered by cloud infrastructure system 2202 are advertised and available for purchase by consumers, and other interfaces 2218. For example, a customer may use a client device to request (service request 2234) one or more services offered by cloud infrastructure system 2202 using one or more of interfaces 2214, 2216, and 2218. For example, a customer may access an online store, browse cloud services offered by cloud infrastructure system 2202, and place a subscription order for one or more services offered by cloud infrastructure system 2202 that the customer wishes to subscribe to. The service request may include information identifying the customer and one or more services the customer wishes to subscribe to. For example, a customer may place an order for a subscription to an automated regular expression generation related service provided by cloud infrastructure system 2202.

[0392] 22, cloud infrastructure system 2202 may include an order management subsystem (OMS) 2220 configured to process new orders. As part of this processing, OMS 2220 may be configured to create an account for the customer if not already created, receive billing and / or account information from the customer that will be used to charge the customer for providing the requested services to the customer, verify the customer information, and once verified, reserve the order for the customer, and prepare the order for provisioning by coordinating various workflows.

[0393] Upon proper validation, the OMS 2220 may invoke an order provisioning subsystem (OPS) 2224 configured to provision resources for the order, including processing, memory, and networking resources. Provisioning may include allocating resources for the order and configuring the resources to facilitate the service requested by the customer order. The manner in which resources are provisioned for the order and the type of resources provisioned may depend on the type of cloud service ordered by the customer. For example, following a workflow, the OPS 2224 may be configured to determine the specific cloud service being requested and identify the number of pods that may be pre-configured for the specific cloud service. The number of pods allocated for an order may depend on the size / amount / level / scope of the service requested. For example, the number of pods to allocate may be determined based on the number of users the service should support, the duration for which the service is requested, etc. The allocated pods may then be customized to the specific requesting customer to provide the requested service.

[0394] Cloud infrastructure system 2202 may send a response or notification 2244 to the requesting customer to indicate when the requested service will be available for use. In some examples, information (e.g., a link) may be sent to the customer that enables the customer to begin using and availing the benefits of the requested service. In certain embodiments, for a customer requesting a regular expression auto-generation related service, the response may include instructions that, when executed, cause the display of a user interface.

[0395] Cloud infrastructure system 2202 may provide services to multiple customers. For each customer, cloud infrastructure system 2202 is responsible for managing information associated with one or more subscription orders received from the customer, maintaining customer data associated with the orders, and providing the requested services to the customer. Cloud infrastructure system 2202 may also collect usage statistics regarding the customer's use of the subscribed services. For example, statistics may be collected on the amount of storage used, the amount of data transferred, the number of users, and the amount of system uptime and system downtime, etc. This usage information may be used to bill the customer. Billing may be on a monthly basis, for example.

[0396] Cloud infrastructure system 2202 may provide services to multiple customers in parallel. Cloud infrastructure system 2202 may store information about these customers, possibly including copyright information. In certain embodiments, cloud infrastructure system 2202 includes an identity management subsystem (IMS) 2228 configured to manage information about customers and to separate the managed information so that information about one customer is not accessed from information about another customer. IMS 2228 may be configured to provide various security-related services, such as identity services, information access management, authentication and authorization services, services for managing customer identities and roles and associated capabilities, etc.

[0397] 23 illustrates an example of a computer system 2300. In some embodiments, the computer system 2300 may be used to implement any of the systems described above. As shown in FIG. 23, the computer system 2300 includes various subsystems, including a processing subsystem 2304 that communicates with several other subsystems via a bus subsystem 2302. These other subsystems may include a processing acceleration unit 2306, an I / O subsystem 2308, a storage subsystem 2318, and a communication subsystem 2324. The storage subsystem 2318 may include non-transitory computer readable storage media, including a storage medium 2322 and a system memory 2310.

[0398] Bus subsystem 2302 provides a mechanism for allowing the various components and subsystems of computer system 2300 to communicate with each other as intended. Although bus subsystem 2302 is shown diagrammatically as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 2302 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, a local bus, etc., using any of a variety of bus architectures. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, a Peripheral Component Interconnect (PCI) bus, which may be implemented as a mezzanine bus manufactured in accordance with the IEEE P1386.1 standard, and the like.

[0399] The processing subsystem 2304 controls the operation of the computer system 2300 and may include one or more processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). The processors may include single-core or multi-core processors. The processing resources of the computer system 2300 may be organized into one or more processing units 2332, 2334, etc. The processing units may include one or more processors, one or more cores from the same or different processors, combinations of cores and processors, or other combinations of cores and processors. In some embodiments, the processing subsystem 2304 may include one or more dedicated co-processors, such as graphics processors, digital signal processors (DSPs), etc. In some embodiments, some or all of the processing units of the processing subsystem 2304 may use customized circuitry, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs).

[0400] In some embodiments, the processing units in the processing subsystem 2304 may execute instructions stored in the system memory 2310 or the computer readable storage medium 2322. In various examples, the processing units may execute various program or code instructions and maintain multiple programs or processes running simultaneously. At any given time, some or all of the program code to be executed may reside in the system memory 2310 and / or the computer readable storage medium 2322, potentially including one or more storage devices. Through appropriate programming, the processing subsystem 2304 may provide the various functions described above. In examples where the computer system 2300 is running one or more virtual machines, one or more processing units may be assigned to each virtual machine.

[0401] In certain embodiments, a processing acceleration unit 2306 may optionally be provided to accelerate the overall processing performed by the computer system 2300, to perform customized processing, or to offload some of the processing performed by the processing subsystem 2304.

[0402] The I / O subsystem 2308 can include devices and mechanisms for inputting information to the computer system 2300 and / or outputting information from or through the computer system 2300. In general, use of the term "input device" is intended to include all conceivable types of devices and mechanisms for inputting information to the computer system 2300. User interface input devices may include, for example, keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, voice input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices, such as Microsoft Kinect® motion sensors that allow a user to control and interact with the input device, Microsoft Xbox® 360 game controllers, devices that provide an interface for receiving input using gestures and voice commands. The user interface input devices may also include eye gesture recognition devices, such as a Google Glass® blink detector, that detects eye movements from the user (e.g., "blinking" while taking a picture and / or making a menu selection) and translates the eye gestures as input to the input device (e.g., Google Glass®). The user interface input devices may also include a voice recognition sensing device that allows the user to interact with a voice recognition system (e.g., the Siri® navigator) via voice commands.

[0403] Other examples of user interface input devices may include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, game pads and graphic tablets, as well as audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers 3D scanners, 3D printers, laser range finders, and eye-tracking devices. User interface input devices may also include medical imaging input devices, such as, for example, computed tomography, magnetic resonance imaging, position emission tomography, and medical ultrasound devices. User interface input devices may also include audio input devices, such as, for example, MIDI keyboards, digital musical instruments, and the like.

[0404] In general, use of the term output device is intended to include all conceivable types of devices and mechanisms for outputting information from computer system 2300 to a user or to another computer. User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices, etc. Display subsystems may be flat panel devices such as those using cathode ray tubes (CRTs), liquid crystal displays (LCDs) or plasma displays, projection devices, touch screens, etc. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / visual information, such as monitors, printers, speakers, headphones, automobile navigation systems, plotters, audio output devices, and modems.

[0405] The storage subsystem 2318 provides a repository or data store for storing information and data used by the computer system 2300. The storage subsystem 2318 provides a tangible, non-transitory computer readable storage medium for storing basic programming and data constructs that provide some example functionality. Software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 2304, provide the functionality described above may be stored in the storage subsystem 2318. The software may be executed by one or more processing units of the processing subsystem 2304. The storage subsystem 2318 may also provide a repository for storing data used in accordance with the teachings of the present disclosure.

[0406] The storage subsystem 2318 may include one or more non-transitory memory devices, including volatile and non-volatile memory devices. As shown in FIG. 23, the storage subsystem 2318 includes a system memory 2310 and a computer-readable storage medium 2322. The system memory 2310 may include several memories, including a volatile main random access memory (RAM) for storing instructions and data during program execution, and a non-volatile read-only memory (ROM) or flash memory in which fixed instructions are stored. In some implementations, a basic input / output system (BIOS), containing the basic routines that help transfer information between elements within the computer system 2300, such as during start-up, may typically be stored in a ROM. Typically, the RAM contains data and / or program modules currently being operated on and executed by the processing subsystem 2304. In some implementations, the system memory 2310 may include several different types of memories, such as static random access memory (SRAM), dynamic random access memory (DRAM), and the like.

[0407] 23, system memory 2310 may load running application programs 2312, program data 2314, and operating system 2316, which may include various applications such as a web browser, a middle tier application, a relational database management system (RDBMS), etc. By way of example, operating system 2316 may include Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or various versions of mobile operating systems such as iOS®, Windows® Phone, Android® OS, BlackBerry® OS, Palm® OS operating systems, etc.

[0408] The computer readable storage medium 2322 can store programming and data configurations that provide some example functionality. The computer readable storage medium 2322 can provide storage of computer readable instructions, data structures, program modules, and other data for the computer system 2300. Software (programs, code modules, instructions) that provide the above functionality when executed by the processing subsystem 2304 may be stored in the storage subsystem 2318. As an example, the computer readable storage medium 2322 may include non-volatile memory such as a hard disk drive, a magnetic disk drive, a CD ROM, a DVD, an optical disk drive such as a Blu-Ray® disk, or other optical media. The computer readable storage medium 2322 may include, but is not limited to, a Zip® drive, a flash memory card, a Universal Serial Bus (USB) flash drive, a Secure Digital (SD) card, a DVD disk, a digital video tape, and the like. The computer readable storage medium 2322 may also include solid state drives (SSDs) based on non-volatile memory such as flash memory-based SSDs, enterprise flash drives, solid state ROM, etc., SSDs based on volatile memory such as solid state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs.

[0409] In particular embodiments, storage subsystem 2318 may also include a computer readable storage medium reader 2320 that may be further connected to a computer readable storage medium 2322. Reader 2320 may be configured to receive and read data from a memory device such as a disk, a flash drive, or the like.

[0410] In certain embodiments, computer system 2300 may support virtualization techniques, including but not limited to virtualization of processing and memory resources. For example, computer system 2300 may provide support for running one or more virtual machines. In certain embodiments, computer system 2300 may execute a program, such as a hypervisor, that facilitates configuration and management of virtual machines. Each virtual machine may be assigned memory, computing (e.g., processors, cores), I / O, and networking resources. Each virtual machine typically runs independently from other virtual machines. A virtual machine typically runs its own operating system, which may be the same or different from the operating systems run by other virtual machines executed by computer system 2300. Thus, potentially multiple operating systems may be run simultaneously by computer system 2300.

[0411] The communications subsystem 2324 provides an interface to other computer systems and networks. The communications subsystem 2324 serves as an interface for sending and receiving data between other systems and the computer system 2300. For example, the communications subsystem 2324 may enable the computer system 2300 to establish a communications channel over the Internet to one or more client devices to send information to and receive information from the one or more client devices.

[0412] The communications subsystem 2324 may support both wired and / or wireless communication protocols. In an embodiment, the communications subsystem 2324 may include a radio frequency (RF) transceiver component for accessing wireless voice and / or data networks (e.g., using cellular telephone technology, advanced data network technologies such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 802.XX family of standards, or other mobile communications technologies, or any combination thereof), a global positioning system (GPS) receiver component, and / or other components. In some embodiments, the communications subsystem 2324 may provide a wired network connection (e.g., Ethernet) in addition to or in place of a wireless interface.

[0413] The communications subsystem 2324 may receive and transmit data in a variety of formats. In some embodiments, the communications subsystem 2324 may receive incoming communications in the form of structured and / or unstructured data feeds 2326, event streams 2328, event updates 2330, etc., in addition to other formats. For example, the communications subsystem 2324 may be configured to receive (or transmit) data feeds 2326 in real time from users of social media networks and / or other communications services, such as web feeds, such as Twitter® feeds, Facebook® updates, Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third party sources.

[0414] In particular embodiments, the communications subsystem 2324 may be configured to receive data in the form of a continuous data stream, which may include an event stream 2328 of real-time events and / or event updates 2330, which may be continuous or infinite in nature without a clear end. Examples of applications that generate continuous data may include, for example, sensor data applications, financial stock ticker boards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, and the like.

[0415] The communications subsystem 2324 may be configured to communicate data from the computer system 2300 to other computer systems or networks. This data may be communicated in a variety of different formats, such as structured and / or unstructured data feeds 2326, event streams 2328, event updates 2330, etc., to one or more databases that may be in communication with one or more streaming data source computers coupled to the computer system 2300.

[0416] The computer system 2300 may be one of a variety of types, including a handheld portable device (e.g., an iPhone® cellular phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head-mounted display), a personal computer, a workstation, a mainframe, a kiosk, a server rack, or other data processing system. Due to the ever-changing nature of computers and networks, the description of the computer system 2300 shown in FIG. 23 is intended only as a specific example. Many other configurations are possible having more or fewer components than the system shown in FIG. 23. Those skilled in the art will recognize other aspects and / or methods for implementing the various examples based on the disclosure and teachings herein.

[0417] Although specific examples have been described, various variations, modifications, alternative configurations, and equivalents are possible. The examples are not limited to operation in a particular data processing environment, but may freely operate in multiple data processing environments. Moreover, while the examples have been described using a particular sequence of transactions and steps, it should be apparent to one skilled in the art that this is not intended to be limiting. Although some flow charts describe operations as a sequential process, many of these operations may be performed in parallel or simultaneously. In addition, the order of operations may be re-specified. A process may have additional steps not included in the figures. Various features and aspects of the above examples may be used individually or together.

[0418] Additionally, while particular examples have been described using particular combinations of hardware and software, it should be understood that other combinations of hardware and software are possible. Particular examples may be implemented exclusively in hardware, exclusively in software, or using a combination thereof. The various processes described herein may be implemented on the same processor or on different processors in any combination.

[0419] Where a device, system, component or module is described as being configured to perform a particular operation or function, such configuration may be achieved, for example, by designing an electronic circuit to perform the operation, by programming a programmable electronic circuit (such as a microprocessor) to perform the operation, by executing, for example, computer instructions or code, or a processor or core programmed to execute code or instructions stored in a non-transitory memory medium or any combination thereof. Processes may communicate using a variety of techniques, including, but not limited to, conventional techniques for inter-process communication, and different pairs of processes may use different techniques, and the same pair of processes may use different techniques at different times.

[0420] In this disclosure, specific details are provided to ensure that the examples are thoroughly understood. However, the examples may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary detail so as not to obscure the examples. This specification provides only illustrative examples and is not intended to limit the scope, applicability, or configuration of other examples. Rather, the above description of the examples provides those skilled in the art with an enabling description for implementing various examples. Various changes are possible within the functionality and configuration of the elements.

[0421] Accordingly, the specification and drawings are to be regarded in an illustrative rather than restrictive sense. It will be apparent, however, that additions, subtractions, deletions, and other modifications and changes may be made thereto without departing from the broader spirit and scope as set forth in the claims. Thus, while specific examples have been described, they are not intended to be limiting. Various modifications and equivalents are within the scope of the appended claims.

[0422] In the above specification, aspects of the disclosure are described with reference to specific examples thereof, but those skilled in the art will recognize that the disclosure is not limited thereto. Various features and aspects of the above disclosure may be used individually or together. Moreover, the examples may be utilized in various environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. Thus, the specification and drawings should be regarded as illustrative rather than restrictive.

[0423] In the above description, the methods have been described in a particular order for illustrative purposes. It should be understood that in alternative examples, the methods may be performed in an order different from that described. It should also be understood that the above methods may be performed by hardware components or embodied in a sequence of machine-executable instructions that, when used, may cause a machine, such as a general-purpose or special-purpose processor or logic circuitry programmed with such instructions, to perform the method. These machine-executable instructions may be stored on one or more machine-readable media, such as a CD-ROM or other type of optical disk, a floppy disk, a ROM, a RAM, an EPROM, an EEPROM, a magnetic or optical card, a flash memory, or other type of machine-readable medium suitable for storing electronic instructions. Alternatively, these methods may be performed by a combination of hardware and software.

[0424] Where a component is described as being configured to perform particular operations, such configuration may be achieved by, for example, designing electronic circuitry or other hardware to perform the particular operations, programming a programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform the particular operations, or any combination thereof.

[0425] Although illustrative examples of the present application have been described in detail herein, it is to be understood that the concepts of the present invention may be variously embodied and employed in other forms, and the claims are intended to be construed to include such modifications except insofar as limited by the prior art.

[0426] Where a component is described as being "configured to" perform particular operations, such configuration may be achieved by, for example, designing electronic circuitry or other hardware to perform the particular operations, by programming a programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform the particular operations, or any combination thereof.< / stuff> < / stufff> < / h1>

Claims

1. 1. A method for generating a regular expression, comprising the steps of: The method is implemented by a computer having a processor executing a program stored in a storage medium, displaying the data set in a tabular format within a user interface, the displayed tabular data set comprising a plurality of data cells, each of the plurality of data cells comprising one or more character sequences, the method further comprising: accepting a first selection via the user interface selecting a character sequence from a first data cell of the plurality of data cells; generating a first regular expression for the character sequence selected by the first selection; identifying one or more data cells of the plurality of data cells that includes a character sequence that matches the first regular expression; accepting commands via the user interface for operations to be applied to character sequences in the one or more data cells of the plurality of data cells that match the first regular expression; executing a command for said manipulation on character sequences in said one or more data cells of said plurality of data cells that match said first regular expression; The method, wherein generating the first regular expression includes generating a regular expression of one or more category codes corresponding to one or more characters in the character sequence selected by the first selection.

2. The method of claim 1 , further comprising displaying the generated first regular expression in the user interface.

3. The method of claim 1 or 2, wherein accepting the first selection comprises highlighting the selected character sequence in the first data cell.

4. The method further comprises: The method of claim 1 , further comprising highlighting character sequences in the one or more data cells that match once the first regular expression has been generated.

5. The tabular dataset includes a plurality of columns, and the operation includes an extract operation, the extract operation comprising:

5. The method of claim 1, further comprising extracting character sequences in the one or more data cells of the plurality of data cells that match the first regular expression into a newly created column in the tabular dataset.

6. The tabular dataset includes a plurality of columns, and the operation includes a split operation, the split operation comprising: splitting each character sequence in the one or more data cells of the plurality of data cells that matches the first regular expression into a plurality of character sequences based on one or more delimiters in the character sequence; and updating the tabular dataset to add the plurality of character sequences obtained by the division as a plurality of columns.

7. The operation includes an obfuscation operation, the obfuscation operation comprising: replacing character sequences in the one or more data cells that match the first regular expression with a symbol sequence for obfuscation; and displaying the replaced symbol sequence in the one or more data cells.

8. The operation includes a substitution operation, the substitution operation being accepting one or more replacement characters via the user interface; replacing characters of character sequences in the one or more data cells that match the first regular expression with the one or more replacement characters; and displaying the replaced character sequence in the one or more data cells.

9. A method according to any one of claims 1 to 8, wherein the operation comprises a deletion operation, the deletion operation comprising replacing character sequences in the one or more data cells that match the first regular expression with an empty character string.

10. the tabular data set includes a plurality of rows of data cells, the first data cell and the one or more data cells being located in a plurality of different rows; The operation includes a row filtering operation, the row filtering operation comprising: filtering the plurality of rows of data cells of the tabular data set by excluding the one or more rows of data cells from the plurality of rows; and displaying the filtered rows of data cells.

11. The method further comprises: receiving, via the user interface, a second selection that selects a portion of the character sequence selected by the first selection of the first data cell; identifying one or more data cells of the plurality of data cells that include a character sequence that matches the portion of the character sequence selected by the second selection; and generating a second regular expression, based on the portion of the character sequence selected by the second selection, the second regular expression being composed of one or more category codes corresponding to one or more characters in the portion of the character sequence.

12. The method of claim 11, wherein accepting the second selection includes highlighting a portion of the selected character sequence within the first data cell.

13. 1. A regular expression generator server computer, comprising: A processor; Memory, and a storage medium coupled to said processor, said storage medium storing instructions executable by said processor to implement the method of any one of claims 1 to 12.

14. A program having instructions for causing one or more processors to carry out a method according to any one of claims 1 to 12.

Images