Classification of data using aggregated information from multiple classification modules
By employing multiple classification modules to aggregate confidence levels, the method improves data classification accuracy, addressing misclassification issues and ensuring regulatory compliance.
Patent Information
- Application Number
- JP2023114805
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2019-08-15
- Filing Date
- 2023-07-13
- Publication Date
- 2024-11-11
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing computing systems face challenges in accurately classifying data portions, particularly sensitive data, leading to improper storage and processing, which can result in non-compliance with regulations and potential breaches of privacy.
A method involving multiple classification modules (dictionary, regex, and machine learning) to analyze data streams, aggregating their confidence levels to determine a final classification, ensuring accurate identification of data types, including sensitive information.
Enhances the accuracy of data classification, reducing the risk of misclassification and ensuring compliance with data protection regulations by effectively identifying and managing sensitive data.
Smart Images

Figure 2024160041000001_ABST
Abstract
Description
[Technical field]
[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Patent Application No. 16 / 542,016, filed August 15, 2019, which is incorporated by reference in its entirety.
[0002] The present disclosure relates to data classification, and in particular to classifying data using aggregated classification information generated from multiple classification modules. [Background technology]
[0003] A computing system can receive and process various types of data. For example, a computing device can process a data stream representing various characteristics of a digital document. The computing system can process the data stream to identify characteristics contained in the data stream, such as name, address, business related information, and personal information.
[0004] In many cases, a computing system may classify portions of data contained in a data stream to identify the nature of the data. As an example, a first portion of the data stream may be classified as an individual's name and a second portion of the data stream may be classified as the individual's personal identifier (e.g., a government-issued identification number).
[0005] The computing system may take an action based on the classification of the data portion (e.g., store the data portion, delete the data portion, send the data portion to an external device). In some cases, some classifications represent personal or sensitive data. Upon determining that the classification of a data portion includes personal / sensitive data, the action taken with respect to the data portion may be based on the nature of the personal / sensitive data. For example, data classified as sensitive data may be stored in a particular portion of memory or deleted to protect the sensitive nature of the data. Summary of the Invention [Problem to be solved by the invention]
[0006] The present invention has been made to solve the problems in the conventional techniques described above. [Means for solving the problem]
[0007] The present invention includes a method performed by a system for classifying data, the method including: obtaining a data stream by each classification module in a set of classification modules; examining a first data portion in the data stream to identify characteristics of the first data portion; comparing the identified characteristics by each classification module in the set of classification modules to determine a proposed classification and a confidence value, wherein each proposed classification type in the classification list matches the characteristics of the first data portion having a first similarity exceeding a first similarity threshold; aggregating the proposed classifications and the confidence values by an aggregation module for each classification type; generating a final classification based on the aggregated proposed classifications and the confidence values by the aggregation module; and transmitting the final classifications to an external device for subsequent operations. [Brief description of the drawings]
[0008] Various features and characteristics of the technology will become apparent to those skilled in the art upon review of the Detailed Description in conjunction with the drawings, in which: Embodiments of the technology are illustrated by way of example, and not by way of limitation, in the drawings, in which like reference numerals may refer to similar elements. [Figure 1] 1 illustrates a block diagram of a production architecture for data classification in accordance with various embodiments. [Diagram 2] 1 illustrates a block diagram of a classification module ensemble in accordance with various embodiments. [Diagram 3] 1 illustrates a block diagram of a machine learning module in accordance with various embodiments. [Figure 4] FIG. 1 is a graphical representation of clusters of trained classification data in accordance with various embodiments. [Diagram 5]1 illustrates a block diagram of a potential data classification table according to various embodiments. [Figure 6] 1 illustrates a flowchart for determining a final classification at an aggregator in accordance with various embodiments. [Figure 7] 1 illustrates a block diagram of a data classification generation method according to various embodiments. [Figure 8] FIG. 1 is a block diagram illustrating an example of a processing system in which at least some of the operations described herein may be performed. The drawings depict various embodiments for illustrative purposes only. Those skilled in the art will recognize that alternative embodiments may be employed without departing from the principles of the technology. Thus, while the drawings depict specific embodiments, various modifications may be applicable to the technology. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] The embodiments presented below represent the information necessary for one skilled in the art to practice the embodiments and illustrate the best mode of practicing the embodiments. Upon reading the following description in light of the accompanying figures, one skilled in the art will understand the concepts disclosed and recognize applications of these concepts not specifically addressed herein. These concepts and applications are encompassed within the scope of this disclosure and the appended claims.
[0010] Embodiments may be described with reference to particular computer programs, system configurations, networks, etc. However, those skilled in the art will recognize that these features are equally applicable to other computer program types, system configurations, network types, etc. For example, the term "Wi-Fi network" may be used to describe a network, but related embodiments may be deployed in other types of networks.
[0011] Additionally, the disclosed techniques may be embodied using dedicated hardware (e.g., circuitry), programmable circuitry that is appropriately programmed with software and / or firmware, or a combination of dedicated hardware and programmable circuitry.Thus, embodiments may include a machine-readable medium having instructions that can be used to program a computing device (e.g., a base station or a networked computer server) to examine video content generated by an electronic device, identify elements contained in the video content, apply a classification model to determine an appropriate action, and perform the appropriate action.
[0012] term The terminology used herein is for the purpose of describing the embodiments only and is not intended to limit the scope of the disclosure. Where the context allows, words using the singular or plural form may also include the plural or singular form respectively.
[0013] As used herein, unless otherwise specified, words such as "processing," "computing," "calculating," "determining," "displaying," "generating," and the like refer to the operations and processing of a computer or similar electronic computing device that manipulates or converts data represented as physical (electronic) quantities in the computer's memory or registers into other data that is also represented as physical quantities in the computer's memory, registers, or other such storage medium, transmission, or display device.
[0014] As used herein, terms such as "connected," "coupled," and the like, may refer to either a direct or indirect connection or coupling between two or more elements. The coupling or connection between the elements may be physical, logical, or a combination thereof.
[0015] Reference to "an embodiment" or "one embodiment" means that a particular feature, function, structure, or characteristic being described is included in at least one embodiment. Appearances of such phrases do not necessarily refer to the same embodiment, and do not necessarily include alternative embodiments that are mutually exclusive of each other.
[0016] Unless the context clearly requires otherwise, the words "comprise" and "comprising" are to be construed in their inclusive sense (i.e., in the sense of "including but not limited to") and not in their exclusive or exhaustive sense.
[0017] The term "based on" is also to be construed in an inclusive sense, rather than an exclusive or exhaustive sense. Thus, unless otherwise noted, the term "based on" is intended to mean "based at least in part on."
[0018] The term "module" refers broadly to a software component, a hardware component, and / or a firmware component. A module is generally a functional component that can generate useful data or other output based on specified input. A module may be self-contained. A computer program may contain one or more modules. Thus, a computer program may contain multiple modules responsible for completing various tasks or a single module responsible for completing multiple tasks.
[0019] The word "or," when used in reference to a list of multiple items, is intended to include all of the following interpretations: any of the items in the list, all of the items in the list, any combination of the items in the list.
[0020] The sequence of steps performed in any of the processes described herein is exemplary. However, steps may be performed in various sequences and combinations, unless physically possible. For example, steps may be added to or removed from the processes described herein. Similarly, steps may be substituted or rearranged. Thus, the process descriptions are intended to be non-limiting.
[0021] overview A computing system can receive and process one or more data streams (e.g., structured data) to identify characteristics of the data and to perform actions based on the characteristics of the data. For example, a computing system can retrieve a database and identify various characteristics of database columns contained in the database (e.g., name, address, phone number). The characteristics of the database can be included as part of the data stream retrieved by the computing system.
[0022] In many cases, a computing system can ingest and process multiple data streams, which can be computationally resource intensive. To identify the nature of the data portions contained within each data portion, the computing system can identify appropriate data classifications that indicate the nature of the data. For example, a computing device can examine a database column of structured data to classify a first digitized document portion as text related to a person's name and a second digitized document portion as text related to a country. The classifications associated with the data portions can be used to identify the nature of the data by examining the classifications rather than processing the associated data, making the process less computationally resource intensive.
[0023] An action may be taken on the data stream portion based on the classification of the data stream portion. Examples of actions that may be taken on the data stream portion may include storing the data in a particular memory location, sending the data to an external device, deleting the data, etc.
[0024] In many cases, the classification of a data portion may indicate that the data portion contains confidential / sensitive data. For example, the classification may indicate that the data contains personally identifiable information (PII) that may be indicative of an individual. Based on such classification, the computing system may take action to protect the confidentiality of the data (e.g., store the data in a particular memory portion).
[0025] Additionally, various municipalities may have laws or regulations that include specific rules related to the processing, dissemination, and storage of certain types of data. For example, a law may include a rule that indicates that all personally identifiable information (PII) associated with an individual is to be identified and retained so that the individual may request access to the PII or request deletion of such information. In this event, the computing system may take appropriate action including identifying such information that may identify data classified as PII for a particular individual. For example, all classified data including PII indicative of a first individual may be stored in a first memory portion, and upon request by the first individual, the data may be deleted or transmitted to an external device.
[0026] However, in many cases, the classification associated with a data portion may be inaccurate. For example, a computing system may examine a data portion containing the text "Christian" and classify it as "religion," when the correct classification is "name." Such improper classification may result in improper storage and processing of the data. Furthermore, improper classification of data may result in non-compliance with various laws / regulations in some municipalities. For example, if data representing PII is improperly classified as non-sensitive data, the data representing PII may be stored in an inappropriate portion of memory or may be accessible by unauthorized entities in violation of, for example, the laws / regulations in the municipality.
[0027] System Overview The present embodiment relates to classifying data portions using aggregated classification information received from multiple classification modules. The multiple classification modules may use a variety of techniques (e.g., dictionaries, regular expression (Regex) pattern matching, neural networks) to examine the received data and determine a proposed classification of the data and a confidence level for each proposed classification. The variable techniques employed by the classification modules may provide one or more proposed classifications with varying confidence levels for each proposed classification.
[0028] Additionally, an aggregation module (or "aggregator") can receive and combine the proposed classifications and confidences and examine the aggregated information to determine with a high degree of certainty a final classification of the data representing the classifications. Based on the final classification determined by the aggregator, further actions (e.g., storing the data, transmitting the data to a client device, encrypting the data) can be performed.
[0029] 1 illustrates a block diagram of a production architecture 100 for classifying data, according to various embodiments. The production architecture 100 can take various types of structured data (e.g., columns of a table contained in a database, a name, an address) and determine a classification for that portion of data.
[0030] The controller 102 may include a computing device (e.g., a computer, a server, a series of interconnected servers) configured to obtain one or more streams of structured data representing various types of information retrieved from the external device 104. For example, the external device 104 may include a device configured to obtain structured data, such as a server. In some embodiments, the controller 102 may receive the data streams (e.g., columns of structured data) from the database 112. In alternative embodiments, the external device 104 may include any device configured to capture or receive a data stream (e.g., a sensor, an image scanner). In some embodiments, one or more external devices 104 may be connected to the controller 102 via a gateway.
[0031] A data stream may include various types of data (e.g., addresses, names). A data stream can contain multiple data portions, such as, for example, database columns contained in a database. As an example, a data stream may include information (e.g., data, metadata) associated with a digitized document, where a first data portion includes an individual's social security number and a second data portion includes terms and conditions included in the document.
[0032] The controller 102 may be in communication with a broker 106. The broker 106 may include a computing device configured to transfer information between the controller 102 and components of the production architecture 100 (e.g., classification modules 108, aggregator 110, etc.). For example, the broker 106 may transfer structured data from the controller 102 to the collection of classification modules 108. The broker 106 may receive proposed classifications from the classification modules 108a-n and forward the proposed classifications to the aggregator 110. The broker 106 may receive final classifications from the aggregator 110 and forward the final classifications to the controller 102. In some embodiments, the production architecture 100 includes multiple brokers 106, each of which transfers information between the controller 102, the collection of classification modules (e.g., classification modules 108a-n), and the aggregator (e.g., aggregator 110).
[0033] The collection of classification modules 108 may include multiple modules configured to examine received data and determine a proposed classification for that data. Classification modules 108a-n utilize various techniques (e.g., dictionary matching, Regex pattern matching) to associate received data with a proposed classification with a particular confidence level indicating the confidence in the accuracy of the proposed classification.
[0034] The proposed classifications and confidence levels generated from each of the collection of classification modules 108 may be forwarded to the aggregator 110 via the broker 106. The aggregator 110 may examine and aggregate the proposed classifications to determine a final classification, which may include an identified data classification based on a combination of multiple classification techniques.
[0035] The controller 102 can receive the final classification from the broker 106 and forward the final classification to the external device 104. The external device 104 can be configured to perform a subsequent action based on the final classification (e.g., storing the data in a first location in a memory of either the controller 102 or the database 112, transmitting the data to the external device 104, updating the database 112).
[0036] Any of the classification module 108, aggregator 110, database 112, etc. may be executed on one or more computing devices, such as, for example, a series of interconnected servers.
[0037] 2 illustrates a block diagram of a collection of classification modules 200 according to various embodiments. As noted above, multiple classification modules (e.g., 202, 204, 206) in the collection of classification modules 200 may examine data and determine a proposed classification based on characteristics of the data.
[0038] 2, the collection of classification modules 200 may include a dictionary module 202, a regular expression (Regex) module 204, and a machine learning module 206. In some embodiments, each classification module 202, 204, and 206 may receive data from a broker simultaneously. In other embodiments, data may be forwarded to the classification modules 202, 204, and 206 in a particular sequence.
[0039] As shown in FIG. 2, each of the collection of classification modules 200 may have access to a classification list 208. The classification list 208 may include a list of all potential classifications for the data. Examples of potential classifications listed in the classification list 208 may include personal information (e.g., name, address, social security number, political information), location information (e.g., country, city, state), technical information (e.g., Internet Protocol (IP) address, Media Access Control (MAC) address), etc. The classification list 208 is discussed in more detail with respect to FIG. 5. The classification list 208 may be predetermined or, in some embodiments, the classification list 208 may include a hierarchy of classification subtypes that provide a more detailed classification for the data.
[0040] The collection of classification modules 200 may include a dictionary module 202. The dictionary module 202 may compare text contained in the received data to one or more dictionaries 212a-b. The dictionaries 212a, 212b may include a list or database of words / phrases in sentences / paragraphs and associated definitions or common examples of the words / phrases. The dictionary module 202 may identify one or more suggested classifications based on a comparison of the identified definitions of the text contained in the data to the list of categories 208.
[0041] The dictionary module 202 may identify characteristics of the received data to identify text contained in the data. The identified characteristics may be compared to dictionary lists. For example, the identified characteristics of the word "Christian" may match multiple lists in the dictionary 212a, one list related to religion and another list related to a given first or last name. The dictionary module 202 may compare the identified characteristics to the matching dictionary lists to identify similarities that may be used to determine a confidence level for each match between the characteristics and the dictionary lists. The matching dictionary lists may be compared to the classification lists 208 to identify classifications that represent the suggested classifications provided by the dictionary module 202.
[0042] The dictionary module 202 may include a confidence level for the proposed classification indicating an estimated accuracy or similarity to the proposed classification. In a first example, if the text identified in the received text matches only one instance in the dictionary 212a, the dictionary module 202 may generate a similar proposed classification with high confidence. In a second example, if the text contains multiple potential definitions (e.g., the word "Christian" as relating to either religion or name), the dictionary module 202 may identify multiple matches and, as a result, generate a low confidence level in the proposed classification for the word. The first line of data may define a minimum number of single-character edits (e.g., Levenshtein distance) between the received word and the dictionary definition of the word to identify the word as a matched word. As an example, if the text contains a state code, a matched word for the state code is found in the dictionary, but there may be a spelling difference with the first or last name.
[0043] In some embodiments, the dictionary module 202 may include a hierarchy of dictionaries. A dictionary hierarchy may include multiple dictionaries organized into a taxonomy or hierarchy. As an example, the dictionary hierarchy includes a high-level dictionary 212a and a low-level dictionary 212b. In this example, the received text is first matched to an entry in the high-level dictionary 212a (e.g., matching the text with the word "Countries"). Based on the match in the high-level dictionary 212a, the dictionary module 202 may check a low-level dictionary 212b (e.g., a European country-specific dictionary) to identify a more specific definition for the text.
[0044] The collection of classification modules 200 may include a regular expression (Regex) module 204. The Regex module 204 may include a set of Regex patterns 214 (i.e., character sequences that define a search pattern). The Regex module 204 may compare the Regex patterns 214 to the received text to identify Regex patterns 214 that match the text with a similarity that exceeds a threshold similarity. The matching Regex patterns 214 may be compared to a classification list 208 to identify a proposed classification and a confidence level.
[0045] In some embodiments, a collection of regular expressions may be linked to a single data type. The Regex module 204 may present several samples of a given entry (any Regex linked to this type) that match a given type. The Regex module 204 may present all matching types above a threshold similarity, along with a confidence level. For example, the classification module may be retrained to improve the accuracy of dictionary lookups and regex matches.
[0046] The set of classification modules 200 may include a machine learning module 206. The machine learning module 206 may comprise a neural network to determine a proposed classification for the received data. For example, the machine learning module 206 may identify a set of rules associated with various information that may increase the accuracy and confidence level of the proposed classification for the received data. For example, an example may include the machine learning module 206 identifying whether a particular arrangement of digits is a telephone number that includes a country code designating the United States. The machine learning module 206 may generally implement this information as a rule and utilize this information when examining subsequent portions of the data. In this example, based on the rules included in the machine learning module 206, the machine learning module 206 may identify that the second set of received information is a telephone number that includes a country code designating the United States. The machine learning module 206 is described in more detail with respect to FIG. 3.
[0047] The classification module 200 may generate a proposed classification and a confidence level that represents an estimated confidence that the proposed classification accurately reflects the received text. The confidence level may include a value in a range of values that represents a quantified confidence in the accuracy of the proposed classification. For example, the confidence level may be a number ranging from 0 to 100, with 100 indicating the highest confidence.
[0048] The confidence level may be based at least in part on some degree of similarity between the received text and the proposed classification. In some embodiments, the confidence level may be based at least in part on the number of classifications that the classification module matches with the received text, since a larger number of potentially accurate classifications indicates a lower confidence level for the proposed classification. In some embodiments, the classification module may generate a number of proposed classifications and a confidence level associated with each proposed classification.
[0049] The classification model may determine a confidence level that estimates the certainty of a proposed classification based on previous classifications made. The confidence level (or percentage) may be based on training data. In some embodiments, for column data, the classification module generates a proposed classification for every data entity in the data column, which may be combined to generate an overall classification for the entire column.
[0050] For example, a dictionary lookup may generate a 90% confidence level for a data entity indicating that 90 samples out of 100 identify the proposed classification for the received data. As another example, a regex matcher may generate a 90% confidence level indicating that the data matches the Regex pattern in 90% of the generated samples.
[0051] In some embodiments, the machine learning module 206 may generate a confidence level for the proposed classification that may differ from the confidence levels of other classification modules (e.g., dictionary module 202, Regex module 204). By way of example, the machine learning module may generate a 90% confidence level indicating that the received data is consistent with the proposed classification 90% of the time based on information learned by the machine learning module.
[0052] The machine learning module 206 may use the training data to take advantage of context and provide more insight into the incoming text when determining a proposed classification. For example, the machine learning module may use the training data to identify rule sets and context information to modify any proposed classifications and confidence levels. The training data allows the machine learning module to eliminate some potential classifications that are inappropriate classifications of the incoming data. Thus, eliminating the potential classifications may increase the confidence level of the proposed classifications.
[0053] A block diagram of the machine learning module 306 according to various embodiments is illustrated in Figure 3. As described above, the machine learning module 306 utilizes training data in conjunction with classification techniques (e.g., dictionaries, Regex) to determine proposed classifications.
[0054] The machine learning module 306 may generate proposed classifications based on data samples 308 representing sample data from iterations of proposed classifications generated for received data. In some embodiments, the machine learning module 306 may be trained based on previously generated data classifications. Examples of data samples 308 may include previously generated classifications for multiple email addresses, common first names, religions, health conditions, etc. The proposed classifications may be sent to client devices and feedback related to the data samples may be received from the clients and the feedback may be integrated into the training data. In some embodiments, a training set may be generated for each client / customer.
[0055] Character embeddings can be used to learn similarities of sample data characteristics, including features based on data metadata. The machine learning module 306 can form character embeddings 310. Character embeddings 310 can include modified data samples that can be converted from the data samples into one or more numerical matrices. In some embodiments, an unsupervised learning algorithm can provide a shallow neural network that is trained to reconstruct the linguistic context of the text. In some embodiments, performing multiple iterations of character embeddings can facilitate the generation of clusters in the numerical matrix. Clusters represent groupings of classifications and data samples with similar characteristics, and clusters can be used to identify similarities between a given classification and a previous classification iteration. An example of a graphical representation of a numerical matrix identifying clusters is shown with respect to FIG. 4.
[0056] The machine learning module 306 may include a neural net 312. The neural net 312 may have multiple inputs to receive both the data samples 308 and the contextual data 314.
[0057] The context data 314 may be utilized to improve the accuracy of the data classification. As an example, a set of samples may be identified as ['F','M','M','F','F','M','M','F']. From the samples, no conclusion can be drawn as to what the classification type is. It is possible that the sample relates to 'gender', but the sample may also relate to, for example, a product status field. The context data may provide other information such that there are only two distinct values for the field, and other fields in the table may include 'customer_name', 'customer_family_status', etc. Thus, the confidence level of the proposed classification that the sample data is for 'gender' is based on the context data. Conversely, if the context data includes 10 distinct values for the field and other fields in the table include 'product_number', 'product_status_change_date', etc., then it is more likely that the proposed classification is not 'gender', but rather a classification related to project status.
[0058] Neural nets 312 may be used in either computer vision or natural language processing (NLP) processes. The neural nets may include either convolutional neural nets (CNNs) or recurrent neural nets (RNNs). Neural nets may be used to identify characteristics or insights related to sample data and / or character embeddings based on information introduced by the neural net.
[0059] The data samples and the context data may be merged in a merge layer 316 and forwarded to a dropout layer 318 and a dense layer 320, which contain fully connected neural net layers. The dropout layer 318 may avoid overfitting by stopping some features during the learning process, so that features that stand out in the training set are not given too much weight.
[0060] The fully-connected neural net layers (e.g., Neural Net 312, Merge 316, Dropout 318, Dense 320) may utilize training data. The training data may include characteristics / insights identified by the fully-connected neural net layers from the sample data and character embeddings. In some embodiments, the training data may be represented as a set of rules that may be utilized by the machine learning module 306 to refine the proposed classification for the received data.
[0061] In some embodiments, the machine learning module may identify features based on column names included in the context data. Further iterations of training the machine learning module may include adding features to the training process while receiving feedback from the client.
[0062] The machine learning module 306 may evaluate the fully connected neural net layers to generate an output 322. Evaluation of the neural net may include determining one or more accuracy metrics to identify the training and testing accuracy of the learning process. The training accuracy metrics may indicate a level of success in using the training data to correct the data samples and character embeddings in identifying a proposed classification for the received text. The output may be sent to an external device requesting feedback on the accuracy of the output. Any feedback received may be incorporated into the training data.
[0063] In some embodiments, the machine learning module 306 may utilize the training data and character embeddings to identify various contextual data contained in the received data. The contextual data may represent contextual information related to a particular data stream. For example, the contextual data for a data stream may indicate that the data stream generally relates to digitized contracts. Further, the contextual data in this example may be utilized to provide context to the classification of data in the data stream. For this example, it may be identified that a classification of "Religion" is less likely than a contract-related classification such as "Name" because contracts are generally less likely to contain information related to religion.
[0064] 4 is a graphical representation of clusters of trained classification data according to various embodiments. As described above, character embeddings can be used to convert the received data and classification proposals into one or more numerical matrices. As an example, multiple iterations of the classification proposals and associated data are converted into a numerical matrix, and characteristics of the classification proposals and data are represented numerically in the matrix. Data can be converted from text into vectors that can be introduced into the numerical matrix.
[0065] The character embeddings formed by the machine learning module are included in the machine learning submodule, and the character embeddings may be converted to numerical vectors. The character embeddings may include both transformations of the text and relationships between the text and other properties. The clusters may represent a graphical representation of the relationships of the character embeddings when converted to numerical vectors. As an example, the letter 'a' may be converted to a numerical vector. Clustering may be used to determine that the letter 'a' is closer to another character (e.g., the letter 'b') than to another character (e.g., the sign '-').
[0066] Additionally, clustering may be utilized with respect to the entire data set. In other words, data relating to each class (e.g., name, address, IP address) may be converted to a numerical vector and examined to determine clustered data. As an example, a data set relating to classes of "name" and "gender" may be converted to a numerical vector. The numerical vector may be examined to determine insights related to the data classes. In this example, the numerical vector may be examined to determine that the "name" and "gender" classifications are related to each other in a "personal information" cluster, which represents a class related to personal information.
[0067] In the graphical representation of the numeric matrix, each instance of the transformed classification data may be represented as a point 402. The location of the point 402 in the graphical representation is based on the transformed characteristic (e.g., classification type, text letter, text word, type of text character (e.g., Greek letter, English capital letter, number)) such that different characteristics are assigned different values in the vectors contained in the numeric matrix.
[0068] In some embodiments, when multiple iterations of the proposed classifications and data are converted into a numerical matrix, multiple clusters (e.g., clusters 404a-e) may be formed. A cluster may represent several points 402 of the numerical matrix that contain multiple similarities in characteristics. For example, a numerical matrix of character embeddings may include a cluster of English lowercase letters, a cluster of numbers, a cluster of English uppercase letters, and a cluster of Greek letters that may be identified. Either the machine learning module or the aggregator may convert the data into a matrix to identify the clusters in the matrix.
[0069] In some embodiments, the machine learning module may convert any of the proposed classifications, the received data, the sample data, the character embeddings, and the training data into a numerical matrix. The numerical matrix may include multiple iterations of the proposed classifications and associated data generated by the machine learning module. In some embodiments, the proposed classifications and associated data are converted into vectors that are populated into a numerical matrix, and the vectors may be represented as points in a graphical representation of the numerical matrix. The machine learning module may identify clusters that represent common characteristics of the various proposed classifications and / or the received data.
[0070] As an example, a first cluster may be identified for a numeric matrix representing data including numbers and classifications commonly associated with numbers (e.g., government issued identifiers, phone numbers). Using this information, the machine learning module may identify whether the particular classification and data is included in the first cluster when converting the information into a numeric matrix. The machine learning module may identify that the particular classification and data is included in the first cluster that may be used to identify that the correct classification is likely to be a classification commonly associated with numbers. This may increase the accuracy and confidence level in identifying classifications for the received data.
[0071] 5 illustrates a block diagram of a table of potential data classifications 500 according to various embodiments. As mentioned above, the list of classifications 500 may include a database or list that includes multiple types of classifications 502 that may be associated with a piece of data. Examples of classifications include any of the following: name, state, country, city, gender, race, occupation, job type, date, day of the week, month, year, language, email, phone number, address, International Bank Account Number (IBAN), Globally Unique Identifier (GUID) / Universally Unique Identifier (UUID), Internet Protocol (IP) address, National Drug Code (NDC) code, Media Access Control (MAC) address, Social Security Number (SSN), health condition, political affiliation, religion, credit card number, etc.
[0072] In some embodiments, the list of categories may include common terms or characteristics associated with each category. For example, a "state" category may have a state (e.g., California) associated with it that makes matching this category and the incoming text relatively easy.
[0073] In some embodiments, the classification may include a hierarchy of classifications or subtypes 504. For example, a classification of "country" may refer to a sublist of countries (e.g., official country name, country code, European countries), with more detailed classifications within each sub-sublist (e.g., Italy, Germany). The hierarchy of classifications may more accurately identify the nature of the incoming text because it includes more appropriate subclasses for each classification.
[0074] In some embodiments, each classification may indicate whether information of this classification should be treated in a particular manner by law or regulation 506. For example, a regulation 506 may relate to the processing and retention of certain types of data to protect the confidentiality of individuals identified in this information. In this example, several types of information (e.g., name, email, gender, religion) may be subject to such a regulation, and the regulation may require the storage of such data so that all relevant data can be identified and retrieved based on a request to retrieve the information.
[0075] In some embodiments, each category may indicate whether the category includes personally identifiable information (PII) 508. PII 508 may include information that may be indicative of an individual, as represented by an "x" in table 500. Examples of information that may include PII may include a name, an email address, an IP address, etc.
[0076] In some embodiments, each classification may indicate whether the classification is subject to regulation 510, such as the European Union's General Data Protection Regulation.
[0077] To protect the privacy of the PII, certain actions may be taken by the external device on data that contains PII or is classified with a classification covered by the GDPR, such as, for example, storing the data in a particular portion of at least one memory, deleting the data, making the data accessible to users with appropriate access rights, shielding the data, tagging the data so that it cannot be used for certain processes, etc.
[0078] Each classification may include a tag or identifier that indicates whether the classification type is subject to the regulation, as represented by an "x" in table 500. The tags may be stored in a label master, where a hierarchy of labels may be stored that may indicate a set of labels related to PII or GDPR. The label master may utilize metadata management for the labels or data classes.
[0079] In some embodiments, the list of categories may be associated with a language. As an example, for a "State" category, subtypes include "New York" and the French translation of New York.
[0080] If the data is classified as sensitive data, the external device may follow various rules or regulations and utilize a workflow to send and store a table of sensitive data.
[0081] Aggregation device overview 6 illustrates a flow chart for determining a final classification by an aggregator, according to various embodiments. The aggregator can receive a proposed classification and a confidence level from each classification module in a collection of classification modules (block 602). Based on the proposed classification and the confidence values, the aggregator can determine a final classification that represents an accurate classification of the received data.
[0082] The aggregator may aggregate the proposed classifications and confidence values received from the collection of classification modules (block 604). In some embodiments, the proposed classifications may be aggregated by classification type. As an example, if a received portion of text contains the word "Christian", the classification modules may generate multiple proposed classifications. In this example, a first proposed classification may include a classification for "Religion" and a second proposed classification may include a classification for "Name". The aggregator may aggregate each proposed classification type, such as, for example, an aggregation of all proposed classifications for "Religion" and all proposed classifications for "Name" and their associated confidence levels.
[0083] The aggregator may utilize machine learning to generate the final classification, in other words, the aggregator may increase the accuracy of the final classification based on information retained from previous iterations of generating the final classification of the data.
[0084] The aggregator may examine the aggregated classification suggestions and confidence levels to determine a final classification, block 606. An aggregator that utilizes multiple classification suggestions to determine a final classification may increase the accuracy of the classification of the received data.
[0085] As an example, if a received portion of text contains the word “California,” each classification module may identify a proposed classification of “State” with a high level of confidence. The aggregator may examine these proposed classifications and determine that the final classification is “State.”
[0086] As an example, the aggregator may aggregate multiple classification suggestions for received text portions that contain the word "Christian." A first aggregated classification type may include a classification suggestion of "Religion" at 40% confidence, and a second classification type may include multiple instances of a classification suggestion of "Name" at both 60% and 50% confidence. The confidence levels of each classification suggestion type may be averaged for each classification type. In this example, the first type "Religion" may have an overall confidence of 40% and the second type "Name" may include an overall confidence of 55% on average. The aggregator may determine a final classification of "Name" based on a determination of the largest aggregated average confidence level for each classification type.
[0087] In some embodiments, the aggregator can utilize training data to aid in determining a final classification for a data portion. The training data can include information or insights identified during a previous iteration of the final classification determination that can be used to improve the accuracy of a subsequent final classification determination. In some embodiments, the training data can include a set of rules or patterns that indicate commonalities between a previous iteration of the final classification determination and a current final classification determination. The aggregator can utilize the rules / patterns contained in the training data when determining the final classification.
[0088] For example, the training data may indicate that if a pattern of received text includes a series of numbers followed by one or more words, then the classification is relatively likely to be an address. As another example, the training data may indicate that given first or last names are often adjacent to one another, so that if a prior final classification of a data stream is "first name," then an adjacent piece of data from the data stream is also likely to be classified as "first name."
[0089] As described above, for each iteration of a final classification determination for the data, the aggregator may convert the final classification, confidence level, and characteristics of the received data into one or more numerical matrices. The numerical matrices may include numerical representations of the final data classification iteration characteristics. Each converted final classification may be represented by a point in the graphical representation of the numerical matrix, and the distance between the points may represent the relative similarity between the final classification iteration characteristics. For example, points that are close together may include a high degree of similarity in either the final classification type or characteristics of the data portions. The aggregator may identify clusters of points in the numerical matrix that have similar characteristics, for example, as shown in FIG. 4.
[0090] In some embodiments, the aggregator may utilize the clustering information to identify insights related to various classifications. For example, multiple iterations of various classifications (e.g., "name", "address", "IP address") may be converted into numerical vectors. In this example, the clustering information may be utilized in identifying a final classification of the data with a high confidence value based on the numerical vectors.
[0091] The final classification determined by the aggregator may be output for subsequent action (block 608). In some embodiments, the final classification is sent to a broker and forwarded to a controller, where the controller forwards the final classification to an external device that performs a subsequent action on the data based on the final classification. One such action that may be performed is storing the received data in a particular portion of at least one memory based on the classification. For example, the received data may be stored in a first portion of the memory based on an identification of the classification of the data as being a non-sensitive classification type.
[0092] In some embodiments, the received data may be stored in a particular portion of memory based on a determination that the final classification of the received data indicates that the nature of the received text is sensitive. This data may include PII or data that is within the scope of laws / regulations in a particular locality. For example, a classification indicating the received data may include an individual's social security number, which indicates an individual and is sensitive in nature. Based on such a determination, the received data may be stored in a portion of memory that includes all sensitive data relating to the individual. Thus, based on the request, the sensitive data about the individual may be retrieved, deleted, sent to the individual, etc. In some embodiments, the classification and the received text may be associated with a database, and the database may be used to identify the data.
[0093] In some embodiments, the final classification and the received data may be transmitted to an external device with a request for feedback related to the final classification (block 610). The feedback request may indicate a request to have the final classification checked for accuracy with respect to the relevant portion of the data classified in the classification. The controller may receive the feedback information from the external device. For example, the feedback information may include an indication of whether the final classification is an accurate or inaccurate representation of the data portion. Based on the received feedback, the external device may perform actions such as storing the final classification, requesting the aggregator to revise the final classification, updating training data for the aggregator, etc.
[0094] In some embodiments, the training data utilized by the aggregator to determine the final classification may be updated (block 612). The training data may be updated based on either determining clustering information from the numerical matrix of the final classification or receiving feedback from an external device related to the accuracy of the final classification.
[0095] For example, if feedback is received indicating that the final classification for the word "Christian" was a classification of "religion" and the correct classification is "name," then the final classification may be revised and the training data may be updated by the aggregator to include this change. In this example, the training data may be updated to include the feedback of the received data and the correct classification. Further in this example, if a second instance of text containing "Christian" similar to the first portion is identified, then the training data may be used to identify that the final classification is likely to be "name" rather than "religion" based on the updated training data.
[0096] In some embodiments, the proposed classifications may be represented by an aggregated table, which may be a representation of the aggregated proposed classifications aggregated by an aggregator, which may determine a final classification based on the content contained in any of the example aggregated tables.
[0097] In some embodiments, the aggregator may perform retraining of the aggregate to aggregate the proposed classifications and determine a final classification. In many cases, classifiers may have relatively high or relatively low performance based on the subset of labels used. The aggregator may diversify the coefficients used for the labels so that the classifications have relatively high performance and make decisions about classifications that can be visualized as a table of coefficients per label for each classifier.
[0098] The confidence levels depicted in the aggregation table may be utilized to determine a final classification. For example, the first data portion may be associated with various classifications (e.g., "Name", "Mail", "Date", "Country") and each classification module may provide a confidence level for that classification. The confidence levels for each classification may be combined or examined to reduce the number of potential classifications or to identify a final classification.
[0099] Each column may represent a different class. A prediction vector may be presented as feedback. In some embodiments, the prediction vector may include "1" for accepted classifications and "0" for rejected classifications. The classification result vector may include a vector formed from each classification output. This input data format for training may be presented as a sparse vector (depending on the application, the classifiers used may be represented as categorical values (1) or true / false subvectors (2)).
[0100] In the aggregate table, a number of potential approaches to retraining the aggregates can be identified. The first approach is to perform a simple convolution of the results from the various classifiers with the coefficients. Another approach can use a variety of approaches to create the result data.
[0101] The use of multiple approaches can minimize the error for each classification using coefficients. A threshold can be selected and the coefficients can be based on a confusion matrix. A long vector can be included as an input for the neural net to minimize the error. For each classification, a training set can be generated by transforming and projecting the classifier vector and the prediction vector.
[0102] For example, for the classification of "Name", the first column is the identifier ID and the second column ML Result indicates the proposed value for the "Name" class from the machine learning classification module. The third column may contain a dictionary result indicating the proposed value for the "Name" class from the dictionary classification module. The fourth column includes "WAS EXPECTED?" which may present a value of "1" to indicate that the incoming data was labeled as the "Name" class and a value of "0" if not.
[0103] In some embodiments, the aggregator may utilize a neural network to aid in the aggregation and determination of the final classification. The neural network may be used to perform a convolution of the given results. The input may include a long vector created from the received classification vectors received by the classification module. The output may include a probability vector for each classification. The aggregator may determine the final classification by examining the output vector for each classification.
[0104] In some embodiments, the data received upon request may not be stored. Based on this, the system may receive and analyze feedback information received from an external device. The controller may receive data from a database. Results may be updated if the classifier is changed. Feedback may be overwritten, the system may check all given feedback for an identifier and aggregate the feedback, with the last feedback overwriting the previous input. The collected data may be used to create a condensed representation in post-processing, including a classification vector and a prediction vector.
[0105] Evaluation Metrics The evaluation metrics may be used to train or retrain components included in the production architecture. During the training process, the evaluation metrics may be used to quantify the quality of the generated classifications. In other words, the evaluation metrics may be used to identify the overall accuracy of the generated classifications.
[0106] The evaluation metric may be based on either classification accuracy or performance. For example, the evaluation metric may represent the speed of the classification module (e.g., how many classification suggestions can be generated in a given amount of time).
[0107] The evaluation metrics may include a confusion matrix that indicates how often the aggregator distinguishes between correct and incorrect classifications in the final classification. The confusion matrix may include Boolean values (true / false) that divide the results into four types: True Positive (TP) - the "true" class is properly classified, True Negative (TN) - the "false" class is properly classified, False Negative (FN) - the "true" class is classified as "false", and False Positive (FP) - the "false" class is classified as "true".
[0108] Accuracy may be a metric that indicates how often the aggregator makes the correct choice between a true being a positive and a false being a negative. With respect to a confusion matrix, accuracy may be defined as a function of the confusion matrix.
[0109] False positives may be different from false negatives. In some embodiments, the system may terminate by presenting only some values above a certain threshold, and present only those deemed most appropriate. Even if the system terminates without presenting a matching class, the system may strive for the lowest possible false positive rate.
[0110] The classification results may be described using either precision, recall, or F1 score. Precision may indicate how often the aggregator determines the correct (or presented as positive) classification. Recall may indicate how often the aggregator presents a positive classification out of all the presentations. In some cases, precision may increase and recall may decrease. This may be the result of fewer failures in the "true" classification, but ultimately the "true" classification is presented less frequently. On the other hand, the system may present more results, but with more incorrect classifications among them. Thus, recall may increase and precision may decrease.
[0111] The F1 score may comprise a harmonic mean of precision and recall that may complete the picture and select a "balance" point between presenting only the correct classification and presenting all the correct classifications.
[0112] In some embodiments, the base metric may be precision, with a focus on low false positive rates, but the decision is not based solely on precision, as the system gives accurate answers and recall is at least partially utilized in making the decision.
[0113] In some embodiments, one of two approaches may be implemented by the aggregator: A first approach may involve maximizing precision without allowing recall to fall below some threshold, and a second approach may involve calculating a weighting metric that may promote precision.
[0114] In some embodiments, multiple inputs can be sent with a proposed probability that can be used as a way to classify the results in a given order. The proposed probability can also give information indicating which model is expected to be more likely. The classification to which a given example should be assigned can be compared to the expected result. The evaluation script can presume that the N top results are positive and the rest are negative. The only exception can be when more than N results have the same probability, such as when all are considered positive. For each model and probability, each label can define a threshold above which the label is presumed to be "positive".
[0115] In some embodiments, the aggregator may make flexible decisions based on the results. It is possible to separate a "leader group" from all the results. A leader group may be a group of top results that are relatively close to each other and relatively far from the rest of the results. Thus, determining and finding a "leader group" may include clustering, calculating standard deviations for and between groups, etc.
[0116] The system may include a data flow. The data flow may include two clusters, a production cluster in which the model is included, and a data cluster. The customer device may send sample / profiling data to the ML platform through a secure connection. The ML platform gateway may forward the request to a controller of the data classification model. Information about the request may be stored in at least one memory. The controller may send the data to a message broker where it may be processed by a subcomponent. A database may hold a list of requests / rows to be processed. Once the results are retrieved, the customer device records the results and all tracked requests are also removed from the database.
[0117] In some embodiments, the feedback may include the customer accepting or rejecting the final classification provided by the aggregator. The feedback may be presented to the system via an external device associated with the customer. The sample data / profiling may be sent to the ML platform along with additional feedback from the customer. The controller may transfer the feedback data to the database. The system periodically offloads data from the database to at least one storage module, which may be copied to the storage module of the data cluster. In some embodiments, retention of the feedback data may be performed in the data cluster based on the feedback stored in the storage module for each customer. The system also maintains sufficient traceability of the classification model so that customer feedback may be incorporated and processed for use in the training data.
[0118] FIG. 7 illustrates a block diagram of a method for generating a classification of data, according to various embodiments.
[0119] Each classification module in the set of classification modules may examine a first data portion in the data stream to identify a set of features for the first data portion (block 702). The set of features may include features of the first data portion, such as text, objects, location of the first data portion in a digitized document, etc. The set of features may be determined using one or more techniques, such as dictionary matching, regular expression (Regex) pattern matching, use of training data, etc.
[0120] In some embodiments, the classification set includes a learning module. The learning module may include at least one of a dictionary, a regular expression list, and a training module. The training module may be configured to examine the proposed classification, the confidence value, and the final classification. The training module may determine an accuracy of the proposed classification of the first data portion and a number of differences between the proposed classification and the final classification. The determined accuracy and number of differences may be included in training data, and the training data may be utilized in subsequent iterations of data classification to determine subsequent proposed data classifications.
[0121] In some embodiments, examining the first data portion to identify the set of characteristics includes identifying at least one entry in a dictionary associated with any classification module in the set of classification modules that includes a similarity between the first data portion and at least one entry that exceeds a threshold similarity, the set of characteristics including at least one entry in the dictionary that includes a similarity between the first data portion and at least one entry that exceeds a threshold similarity.
[0122] In some embodiments, examining the first data portion to identify the set of characteristics includes identifying at least one pattern included in a regular expression list associated with any classification module included in the set of classification modules that includes some difference between the first data portion and the at least one pattern that is less than a threshold difference, wherein the set of characteristics includes at least one pattern included in the regular expression list that includes some difference that is less than the threshold difference.
[0123] The identified feature sets are compared to the classification list to determine a proposed classification and a confidence value (block 704). The proposed classification may identify a classification listed in the classification list that includes a similarity with the feature set of the first data portion that exceeds a threshold similarity. The confidence value may include a value indicative of a probability of the proposed classification accurately describing the first data portion. The proposed classification and the confidence value may be transmitted to an aggregation module.
[0124] The aggregation module may aggregate the proposed classifications and confidence values determined by each classification module in the collection of classification modules (block 706).
[0125] An aggregation module may generate a final classification based on the aggregated classification proposals and confidence values of the classification modules in the collection of classification modules (block 708).
[0126] In some embodiments, generating the final classification includes identifying each classification type included in the received proposed classification, where each classification type is included in the classification list. The aggregation module may compare each identified classification type with a confidence value to identify an overall confidence for each classification type. The aggregator may determine the final classification by identifying the identified classification type with the highest overall confidence.
[0127] In some embodiments, the method may include determining a final classification confidence indicating a confidence between the first data portion and the final classification, and the aggregator may add the final classification confidence to an aggregator training data set configured to be utilized in subsequent iterations in determining a final classification of the data.
[0128] In some embodiments, the method includes sending a request for feedback to an external device, where the request for feedback may include any of the final classification, the first data portion, or the final classification accuracy. A response is received from the external device including the feedback information, and the feedback information is added to the aggregator training data set.
[0129] In some embodiments, the method may include converting the final classification and the first data portion into at least one vector that is introduced into a numerical matrix that identifies quantified characteristics of the final classification and the first data portion, where clusters may be identified in the numerical matrix, the clusters representing the final classification and repetitions of the data that have common characteristics, and information relating to the clusters is added to the aggregator training data set.
[0130] The aggregation module may transmit the final classification to the external device to perform a subsequent action based on the final classification of the first data portion (block 710). In some embodiments, the aggregation module may transmit the final classification to a broker, which may forward the final classification to a controller, which may forward the final classification to the external device. The subsequent action may include storing the first data portion in a first location of at least one memory associated with the data including the final classification, transmitting the first data portion to the external device, deleting the first data portion, and storing the first data portion in a second location of at least one memory associated with the data including the data indicative of the individual.
[0131] In some embodiments, the final classification may be examined to determine whether it includes an identifier indicating that the final classification includes information indicative of an individual. Information indicative of an individual may include any information identified by PII, GDPR, or laws / regulations imposed by a locality. The external device may be configured to identify an action subset representing an action on the first data portion that holds the information indicative of an individual. The action subset may include, for example, storing the first data portion in a particular memory portion that includes other information indicative of an individual, updating a database to identify the first data portion, deleting the first data portion, transmitting the first data portion to the external device, etc. The external device may perform a subsequent action including one of the action subset based on a determination that the final classification includes the identifier.
[0132] Processing system example Figure 8 is a block diagram illustrating an example of a processing system 800 in which at least some of the operations described herein may be performed. As shown in Figure 8, the processing system 800 may include one or more central processing units ("processors") 802, a main memory 806, a non-volatile memory 810, a network adapter 812 (e.g., a network interface), a video display 818, input / output devices 820, control devices 822 (e.g., a keyboard and pointing device), a drive unit 824 including a storage medium 826, and a signal generating device 830 communicatively coupled to a bus 816. The bus 816 is illustrated as an abstraction representing one or more separate physical buses, point-to-point connections, or both connected by appropriate bridges, adapters, or controllers. Thus, the bus 816 may include, for example, a system bus, a Peripheral Component Interconnect (PCI) bus or a PCI Express bus, a HyperTransport or Industry Standard Architecture (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus (USB), an IIC (I2C) bus, or the Institute of Electrical and Electronic Engineers (IEEE) standard 1394 bus, also known as "Firewire."
[0133] In various embodiments, the processing system 800 operates as part of a user device, although the processing system 800 may be connected (e.g., wired or wirelessly) to the user device. In a networked deployment, the processing system 800 may operate in the capacity of a server or a client machine in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.
[0134] Processing system 800 may be a server computer, a client computer, a personal computer, a tablet, a laptop computer, a personal digital assistant (PDA), a mobile phone, a processor, a web appliance, a network router, switch or bridge, a console, a handheld console, a gaming device, a music player, a network-connected ("smart") television, a television-connected device, or other portable device or machine capable of executing a set of instructions (sequential or otherwise) that specify operations to be performed by processing system 800.
[0135] Although main memory 806, non-volatile memory 810, and storage medium 826 (also referred to as "machine-readable medium") are shown as single media, the terms "machine-readable medium" and "storage medium" should be interpreted to include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) that store one or more sets of instructions 828. The terms "machine-readable medium" and "storage medium" should also be interpreted to include any medium capable of storing, encoding, or transporting a set of instructions for execution by a computing system and causing a computing system to implement one or more methodologies of the presently disclosed embodiments.
[0136] Generally, the routines executed to perform the disclosed embodiments may be executed as part of an operating system or a specific application, component, program, object, module, or sequence of instructions referred to as a “computer program.” A computer program typically comprises one or more instructions (e.g., instructions 804, 808, 828) that are configured at various times in various memory and storage devices of a computer and that, when interpreted and executed by one or more processing units or processors 802, cause the processing system 800 to perform operations and implement elements associated with various aspects of the disclosure.
[0137] Additionally, while embodiments are described in the context of full-featured computers and computer systems, those skilled in the art will recognize that various embodiments may be distributed as program products in various forms, and that the disclosure applies equally regardless of the particular type of machine or computer-readable medium used to actually perform the distribution. For example, the techniques described herein may be performed using a virtual machine or cloud computing services.
[0138] Further examples of machine-readable storage media, machine-readable media, or computer-readable (storage) media include, but are not limited to, recordable type media such as volatile and non-volatile memory devices 810, floppy and other removable disks, hard disk drives, optical disks (e.g., Compact Disk Read Only Memories (CD ROMS), Digital Versatile Disks (DVDs), and transmission type media such as digital and analog communications links.
[0139] Network adapter 812 enables processing system 800 to interface data on network 814 with entities external to processing system 800 through any known and / or convenient communications protocol supported by processing system 800 and the external entities. Network adapter 812 may include one or more of a network adapter card, a wireless network interface card, a router, an access point, a wireless router, a switch, a multi-layer switch, a protocol converter, a gateway, a bridge, a bridge router, a hub, a digital media receiver, and / or a repeater.
[0140] Network adapter 812 may, in some embodiments, include a firewall that can govern and / or manage permissions to access / proxy data on a computer network to track variable trust levels among various machines and / or applications. A firewall may be any number of modules having any combination of hardware and / or software components that can enforce a set of predefined access rights between machines and applications, machines and / or specific sets of applications, to regulate, for example, the flow of traffic and resources shared among these variable entities. A firewall may additionally manage and / or have access to access control lists that detail permissions, including rights to access and manipulate objects by individuals, machines and / or applications, and the circumstances under which the permission rights are valid.
[0141] As noted above, the techniques introduced herein may be performed, for example, by dedicated hardwired (i.e., non-programmable) circuitry as a whole, or by programmable circuitry (e.g., one or more microprocessors) programmed with software and / or firmware, or by any combination of such forms. The dedicated circuitry may be in the form of, for example, one or more application specific integrated circuits (ASICs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), etc.
[0142] From the foregoing, it will be appreciated that, although specific embodiments of the invention have been described herein for purposes of illustration, various modifications may be made without deviating from the scope of the invention. Accordingly, the invention is not limited except as by the appended claims. [Explanation of symbols]
[0143] 100 Production Architecture 102 Control device 104 External Devices 106 Broker 108, 108a, 108b, …, 108n Classification Module 110 Aggregation Device 112 Database 200 Classification Module 202 Dictionary Module 204 Regular Expression Module 206 Machine Learning Module 208 Classification List 212a, 212b, …, 212n Dictionary 214a, 214b, …, 214n Regular Expression Patterns 306 Machine Learning Module 308 Data Samples 310 Character embedding 312 Neural Networks 314 Context Data 316 Merge Layer 318 Dropout Layer 320 Dense Layer 322 Output 402 points 404a, 404b, 404c, 404d, 404e Cluster 500 classifications 800 Processing Systems 802 Processor 804 command 806 Main Memory 808 command 810 Non-volatile memory 812 Network Adapter 814 Network 816 Bus 818 Video Display 820 Input / Output Devices 822 Control Device 824 Drive Unit 826 Storage medium 828 command 830 Signal Generating Device
Claims
1. 1. A method implemented by a system for classifying data, comprising: obtaining a data stream by each classification module in a set of classification modules; examining a first data portion included in the data stream to identify a characteristic of the first data portion; comparing the identified characteristics with a classification list by each classification module of the set of classification modules to determine a proposed classification and a confidence value, each proposed classification identifying a classification type included in the classification list that matches the characteristics of the first data portion having a first similarity that exceeds a first similarity threshold; aggregating, by an aggregation module, the proposed classifications and the confidence values for each classification type; generating, by the aggregation module, a final classification based on the aggregated proposed classifications and the confidence values; transmitting the final classification to an external device for subsequent operation; The method includes:
2. The method of claim 1 , wherein the data stream includes data structured in tables contained in a database, each portion of the data stream representing a column of a table contained in the database.
3. The set of classification modules: a dictionary module configured to compare the identified characteristics of the first data portion with entries contained in at least one dictionary; a regular expression module configured to compare the identified characteristics of the first data portion to regular expression patterns included in at least one regular expression pattern list; a learning module configured to compare the identified characteristics of the first data portion with a training data set; The method of claim 1 , comprising:
4. converting each data sequence and each of said classification schemes to at least one numerical vector identifying a quantified characteristic of each data sequence and each of said classification schemes; The method of claim 2 further comprising:
5. The learning module: generating a first proposed classification and a first confidence value based on a comparison of the characteristics of the first data portion and the classification list to the training data set; determining at least one rule corresponding to the characteristics of the first data portion and the generated first proposed classification; introducing at least one rule into said training data set; The method of claim 2 , configured as follows:
6. The generating of the final classification further comprises: comparing the aggregated proposed classifications and the aggregated confidence values for each classification type to a set of rules utilized by the aggregation module to determine a first classification; determining at least one rule corresponding to a common characteristic between the aggregated proposed classification and the final classification; introducing at least one rule into said rule set; The method of claim 1 , comprising:
7. examining the final classification to determine whether the final classification includes information indicative of an individual; adding a tag to the first data portion based on a determination that the final classification includes information indicative of the individual; and The method of claim 1 further comprising:
8. The method of claim 1 , wherein the set of classification modules and the aggregation module forward information to the external device via a broker.
9. 2. The method of claim 1, wherein the subsequent action performed by the external device includes any of storing the first data portion in a first location of at least one memory, transmitting the first data portion to an external device, deleting the first data portion, and storing the first data portion in a second location of at least one memory associated with data identified as indicative of an individual.
10. sending a feedback request to an external device, the feedback request including either the final classification or the first data portion; receiving a response from the external device including feedback information, the feedback information being added to an aggregator training data set utilized by the aggregation module to generate a subsequent final classification; The method of claim 1 further comprising:
11. converting said final classification and said first data portion into at least one numerical vector representative of a quantified characteristic of said final classification and said first data portion; The method of claim 1 further comprising:
12. A set of classification modules including a dictionary module, a regular expression module, and a learning module, each classification module comprising: Obtaining a first data set structured into data columns in a database; identifying a characteristic of a first data string of the database that is included in the first data set; determining a first proposed classification from the classification list and a first confidence value, the first proposed classification identifying a classification in the classification list that matches the identified characteristic of the first string; A set of classification modules, 1. An aggregation module, comprising: receiving the proposed classifications and the confidence values determined by each of the set of classification modules; aggregating the proposed classifications and the confidence values by classification type; determining a final classification based on the aggregated proposed classifications and the confidence values; transmitting the final classification to an external device configured to perform a subsequent operation on the first data set based on the final classification. an aggregation module configured to A system comprising:
13. a broker connected to the set of classification modules and the aggregation module, transmitting the first data set to each of the set of classification modules; forwarding the proposed classifications from the set of classification modules to the aggregation module; receiving the final classification from the aggregation module; a broker configured to a controller connected to the broker and to the external device, the controller configured to transfer the final classification from the broker to the external device; The system of claim 12 further comprising:
14. The dictionary module: identifying entries in at least one dictionary included in the dictionary module that match the characteristic of the first data string; comparing each classification in the list of classifications to the identified entry to determine a first classification that has a similarity with the entry that exceeds a similarity threshold; wherein the proposed classifications of the dictionary module include the first classification; The regular expression module: identifying a pattern in at least one regular expression pattern list included in the regular expression module that matches the characteristic of the first data string; comparing the identified pattern to each classification in the list of classifications to determine a second classification that has a similarity to the pattern that exceeds the similarity threshold. wherein the classification proposal of the regular expression module includes the second classification. The system of claim 12.
15. The learning module: generating the proposed classification and the confidence value based on a comparison of characteristics of the first data string and the classification list to a training data set associated with the learning module; determining at least one rule corresponding to the characteristics of the first data sequence and the generated proposed classification; introducing at least one rule into said training data set; The system of claim 12 configured to:
16. The aggregation module further comprises: examining the final classification to determine whether the final classification relates to a classification that includes information indicative of an individual; adding tags to said final classification; It is structured as follows: the external device is configured to perform one of a subset of subsequent actions on the first data set based on receipt of the final classification including the tag. The system of claim 12.
17. 1. A method implemented by an aggregation module for classifying data, comprising: receiving from each of a set of classification modules a proposed classification and a confidence value for a data sequence included in the first structured data set, each proposed classification indicating a classification included in a list of classifications and each confidence value indicating a degree of certainty of the proposed classification as related to a characteristic of the data sequence; aggregating the received classification suggestions and confidence values based on each classification type listed in the classification list; comparing the aggregated proposed classifications to identify a final classification; transmitting the final classification to an external device configured to perform a subsequent operation on the sequence of data based on the final classification; The method includes:
18. 1. A dictionary module, comprising: identifying entries in at least one dictionary included in a dictionary module that match the characteristics of the data string; comparing each classification in the list of classifications to the identified entry to determine a first classification having a similarity with the entry that exceeds a similarity threshold, and the proposed classifications in the dictionary module include the first classification; A dictionary module configured as follows: A regular expression module, comprising: identifying patterns in at least one regular expression pattern list included in a regular expression module that match the characteristic of the data string; comparing the identified pattern to each classification in the classification list to determine a second classification having a similarity to the pattern that exceeds the similarity threshold, and the proposed classification of the regular expression module includes the second classification. a regular expression module configured as follows: A learning module comprising: generating a third proposed classification based on a comparison of the characteristics of the data string and the classification list to a training data set associated with a learning module; determining at least one rule corresponding to the characteristics of the data sequence and the generated proposed classification; introducing said at least one rule into said training data set; A learning module configured to: The method of claim 17 , wherein the set of classification modules comprises:
19. sending a feedback request to the external device, the feedback request including the final classification and the data string; receiving a response from the external device including feedback information; 20. The method of claim 17, further comprising:
20. introducing the received feedback information into any of a set of training data that is configured to be utilized by the set of classification modules through a retraining process to improve subsequent proposed classifications of other data portions; 20. The method of claim 19, further comprising:
