Method and system for word edit distance embedding

Abstract

A system for classifying words in a batch of words may include at least one memory device storing instructions that cause at least one processor to: use a neural network (NN) to create a dictionary vector for each of a plurality of dictionary words, store each dictionary vector with a classification indicator corresponding to the associated dictionary word, and use the NN to create a word vector for each word in the batch of words for classification. A closest matching dictionary vector for each word vector is found, and the classification indicator of the closest matching dictionary vector for each word vector is reported.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 008,552, filed April 10, 2020, the disclosure of which is incorporated herein by reference in its entirety.

[0002] This patent application relates to text similarity computation, and in particular to word edit distance embedding. [Background technology]

[0003] Companies that handle personal information, whether it be employee information, medical data, or financial information, are required to protect that information and limit its use. Additionally, all data collected is to be anonymized and stored only for as long as necessary. Governments around the world have enacted regulations, such as the European Union's General Data Protection Regulation (GDPR), in an effort to help protect personal information. Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention has been made to solve the problems in the conventional techniques described above. [Means for solving the problem]

[0005] The systems and methods described herein may be better understood by reference to the following Detailed Description in conjunction with the accompanying drawings, in which like reference numbers indicate identical or functionally similar elements.

[0006] Each heading provided in this specification is merely for convenience and does not necessarily affect the scope of the embodiments. Moreover, the drawings are not necessarily drawn to scale. For example, the dimensions of some of the elements in the drawings may be enlarged or reduced to help improve understanding of the embodiments. Furthermore, the technology of the present disclosure is applicable to various modifications and alternative forms, and specific embodiments are shown by way of example in the drawings and will be described in detail below. However, it is not intended to unnecessarily limit the described embodiments. Rather, the embodiments are intended to encompass all modifications, combinations, equivalents, and alternatives falling within the scope of the present disclosure. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a flow diagram illustrating a method of operation of a processor-based word edit distance embedding system in accordance with some implementations of the present technology. [Diagram 2] FIG. 1 is a flow diagram illustrating a method for training a neural network to classify words, according to some embodiments of the present technology. [Diagram 3] 1 illustrates a two-dimensional matrix for encoding words according to some embodiments of the present technology. [Figure 4] FIG. 1 is a schematic diagram of a training architecture having a twin convolutional neural network, in accordance with some embodiments of the present technology. [Diagram 5] FIG. 2 is a flow diagram illustrating a method for setting up a word vector index for use in classifying words according to some implementations of the present technology. [Figure 6] 1 is a chart showing dictionary word embeddings and associated classes and thresholds. [Figure 7] FIG. 2 is a flow diagram illustrating a method for classifying words in accordance with some implementations of the present technology. [Figure 8] 1 is a block diagram illustrating an overview of an apparatus in which some embodiments can operate. [Figure 9] FIG. 1 is a block diagram illustrating an overview of an environment in which some embodiments can operate. [Figure 10] FIG. 1 is a block diagram illustrating components that can be used in a system employing techniques of the present disclosure in some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Various embodiments of the system and method introduced above will now be described in detail. In the following description, specific details are provided to provide a thorough understanding and practical description of these embodiments. However, as will be appreciated by those skilled in the art, the methods and techniques described herein can be practiced without many of these details. As will also be appreciated by those skilled in the art, the present technology may include many other features not described in detail herein. Moreover, in the following, some well-known structures or functions may not be shown or described in detail to avoid unnecessarily obscuring important descriptions.

[0009] The terms used below should be given their broadest reasonable interpretation, even if they are used in conjunction with the detailed description of some specific examples of embodiments. Indeed, some terms may be emphasized more below, but if there is any term intended to be given some restrictive interpretation, it is expressly and specifically defined in this section to be so interpreted.

[0010] A method and system for classifying unknown words using a novel neural network and similarity search architecture is disclosed. In the context of privacy protection, it is desirable to monitor whether personal information is being transmitted within a network and / or outside of a network. The techniques of this disclosure make it possible to monitor those transmissions by securely anonymizing the words in the transmissions and comparing them to words in various class dictionaries (e.g., country, name, ethnicity, language, religion, gender, marital status, etc.) to identify the class of information being transmitted. Advantageously, the techniques of this disclosure securely anonymize and classify words without ever storing the original words.

[0011] In some embodiments, a neural network is trained and retrained such that the distance between pairs of vectors in a word-based dictionary dataset approximates an edit-distance-based metric between corresponding words. Training involves creating vectors for every dictionary word with the trained neural network and storing those vectors in a specialized structure. A matching algorithm then creates vectors for every word in a batch process. This involves finding the nearest match for each word and applying a score based on the distance to the nearest dictionary match to determine the classification of each word.

[0012] FIG. 1 is a high-level flow diagram illustrating a method 100 of operation of a processor-based word edit distance embedding system for classifying a batch of data (e.g., words) according to some implementations of the present technology. The method may include step 102 of training a neural network (NN) with pairs of words such that the distance between corresponding vectors is close to an edit distance-based metric between the words. At step 104, vectors for a selection of dictionary words are created by the trained NN. The resulting vectors may be stored at step 106. Then, at step 108, vectors are created for all words in the batch of words to be classified. At step 110, the closest match for each word in the batch is determined. In some embodiments, at step 112, a score based on the distance to the associated closest match may be applied to each word in the batch.

[0013] FIG. 2 illustrates a method 200 for training a neural network to classify words according to some implementations of the present technology. In some embodiments, the neural network may be a convolutional neural network (CNN). Training the CNN may include, for example, encoding each of a plurality of training words into a matrix form (FIG. 3) at step 202. The training words may include a dictionary of words that belong to a particular class (e.g., country, name, ethnicity, language, religion, gender, marital status, etc.). The training dictionary may include a base dictionary, a full English dictionary, and a non-word dictionary (which includes data that does not resemble any data of a dictionary-like type).

[0014] In step 204, CNN and CNN Twin (FIG. 4) are used to create training vector pairs for multiple pairs of encoded training words. The pairs may be randomly selected from the training dictionary for training when the word edit distance is large, and may be created by adding noise to the words for training when the word edit distance is small.

[0015] A similarity metric (SM) is calculated for each of the plurality of pairs of the plurality of training words at step 206. The SM may be calculated based on an edit distance (ED) (e.g., Levenshtein ED) as follows: SM(word1,word2)=1-ED(word1,word2) / max(length(word1),length(word2)) In some embodiments, the SM may be based on an encoded form of the word.

[0016] In step 208, the similarity metric and the cosine similarity are compared for each pair of training words, and the CNN is adjusted (e.g., weights are adjusted) based on this comparison to drive the cosine similarity between the pairs to match the similarity metric. Using the cosine similarity between vectors instead of Euclidean distance reduces computation time and improves computation accuracy. Once the CNN is trained, in step 210, the trained CNN is stored. In some embodiments, the cosine similarity may be calculated by dividing the dot product of a pair of vectors by the product of their Euclidean norms.

[0017] Using the above similarity metric rather than the traditional Levenshtein ED improves retrieval. This can be illustrated with two examples: two two-letter codes, PL and PS, and two words, Poland and Roland. Both of these pairs have ED=1. However, the value 1 has a much larger effect on the first pair, since it changes half of the words. Therefore, if the edit distance error is around "1", using ED is much less accurate for short words than for long words. In contrast, the similarity metric of the present disclosure is less accurate for long pairs (especially those with a large edit distance), but this is acceptable in the system of the present disclosure, since we search for words within a certain range, and the exact distance value is not very important as long as it is not too large. In some embodiments, the minimum similarity threshold for accepting a match is 0.7.

[0018] FIG. 3 illustrates a two-dimensional matrix 220 for encoding a word (e.g., “Poland”) according to some implementations of the present technology. The two-dimensional matrix of the present disclosure is similar to that described in “Levenshtein Space Deep Embedding for Query-by-string Word Spotting” by Lluis Gomez, Marcal Rusinol, and Dimosthenis Karatzas, International Conference on Document Analysis and Recognition (ICDAR) 2017, which is incorporated by reference in its entirety. The matrix 220 includes a first dimension 222 that includes 26 English letters and 4 special characters, and a second dimension 224 that includes character positions within the string. The number of character positions may be limited to, for example, 50, resulting in a 30×50 matrix. Note that, as outlined below, matrix 220 is capable of encoding English and non-English characters, as well as the "null" character.

[0019] The characters in the data set may be grouped as follows: 1 English letter - English letters are standardized

number

[0020] FIG. 4 is a schematic diagram of a training architecture 300 with a twin convolutional neural network according to some implementations of the present technology. The twin CNNs 302(1) and 302(2) have the same structure and the same weights. The input to the twin CNNs is the same 30×50 matrix 304, e.g., the matrix described above with respect to FIG. 3. In some embodiments, the CNN applies a convolutional layer 308 with 64 kernels 306 of size 30×3, and a fully connected layer 310 with 128 output neurons. For each pair of vectors corresponding to word 1 and word 2, a cosine similarity 312 is calculated and compared to the similarity metric between the encoded words 1 and 2 to derive a “loss” 316. The CNN is driven to learn an optimal transformation of words (e.g., strings) into vector form such that the cosine similarity distance between the vectors is comparable to the similarity metric of their original words. In other words, the loss 316 is driven towards zero.

[0021] FIG. 5 is a flow diagram illustrating a method 400 for setting up a word vector index for use in classifying words according to some implementations of the present technology. After the CNN has been trained as described above with respect to FIGS. 2-4, a plurality of dictionary words are encoded in a matrix format (e.g., the matrix described above with respect to FIG. 3) at step 402. In some embodiments, the dictionary words used to train the CNN may include the dictionary words used for classification. At step 404, a dictionary vector for each of the plurality of encoded dictionary words is created using the trained CNN. At step 406, the resulting dictionary vectors are stored with classes corresponding to the associated dictionary words. At step 408, the dictionary word vectors may be indexed for efficient searching. In some embodiments, the vector indexing is performed according to the Facebook Artificial Intelligence Similarity Search (FAISS) technique. FIG. 6 illustrates an example data structure 420 for dictionary word vector embeddings 426 and associated classes 428 and thresholds 430. For example, a country dictionary 422 includes names of various countries, and a name dictionary 424 includes various first names. Each country and first name corresponds to a vector embedding 426. The class code 428 is "0" for the country and "1" for the name. The data structure 420 contains a mapping 430 of class codes and class thresholds to each vector embedding. In some embodiments, the thresholds for each class may be determined empirically to minimize false positives.

[0022] FIG. 7 is a flow diagram illustrating a method 500 for classifying words according to some implementations of the present technology. Once the CNN is trained and the dictionary word index is prepared, the words of interest in the batch of words may be classified to determine whether the words in the batch correspond to any sensitive class of data (e.g., personal information). In step 502, the batch of words is received for classification but is not stored. In step 504, the received words are encoded into a matrix format. This can be done, for example, using the technique described above with respect to FIG. 3. In step 506, a word vector for each of the encoded batch words is created using the trained CNN. In step 508, the closest match and score for each resulting word vector in the batch is found in the index of dictionary word vectors. This is done by searching for nearest neighbors (e.g., using the FAISS library). FAISS is a library for efficient similarity search and clustering of dense vectors. It includes algorithms for searching in a collection of vectors of any size. In step 510, the average results for each class corresponding to the batch of words are reported.

[0023] Step 510 can be illustrated with an example batch containing two words, Holland and Cuba. For each word, the four closest matches and corresponding scores are found as follows: Holland - Holland, Poland, Roland, Holland Cuba - Cuba, Coby, Curt, Aruba

[0024] Each match is associated with a class and a retrieved score as follows: Holand - Holand(Last name - 1.0), Poland(Country - 0.8), Roland(First name - 0.8), Holland(Country - 0.8) Cuba - Cuba(country - 1.0), Coby(first name - 0.5), Curt(first name - 0.5), Aruba(country - 0.6)

[0025] For each word in the batch, the best score from each class is selected as follows: Holand - Last name: 1.0, Country: 0.8, First name: 0.8 Cuba - Country: 1.0, First name: 0.5

[0026] The following class breaks are given: Country:0.8 Last Name: 0.8 First Name: 0.7

[0027] The above thresholds are applied to the highest score in each class. If a score is equal to or greater than the corresponding threshold, it is left as is, and if it is less than the threshold, it is treated as 0, as follows: Holand - Last name: 1.0, Country: 0.8, First name: 0.8 Cuba - Country: 1.0, First Name: 0

[0028] The results for each class are then averaged as follows: Last name: (1+0) / 2=0.5 Country:(0.8+1) / 2=0.9 First name: (0.8+0) / 2=0.4

[0029] And the results are reported as follows: Country:0.9 Last Name: 0.5 First Name: 0.4

[0030] The reported results can indicate to the user that a particular classification of data (e.g., country, last name, and first name) is likely to be transmitted on the system, without storing or exposing the original data. The right system

[0031] The techniques disclosed herein may be embodied as dedicated hardware (e.g., circuitry), or as programmable circuitry appropriately programmed with software and / or firmware, or as a combination of dedicated and programmable circuitry. Thus, embodiments may include a machine-readable medium having instructions stored thereon that may be used to cause a computer, microprocessor, processor, and / or microcontroller (or other electronic device) to perform a process. The machine-readable medium may include, but is not limited to, an optical disk, a compact disk read-only memory (CD-ROM), a magneto-optical disk ROM, a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic or optical card, a flash memory, or other type of medium / machine-readable medium suitable for storing electronic instructions.

[0032] Some embodiments are discussed in detail below with reference to the drawings. FIG. 8 is a block diagram showing an overview of devices in which some embodiments of the disclosed technology can operate. These devices may include hardware components of a device 700 that determines a risk score and associated pricing and / or risk ratio. The device 700 may include one or more input devices 720 that provide input to a CPU (processor) 710 to inform actions. Actions are typically mediated by a hardware controller that interprets signals received from the input devices and communicates information to the CPU 710 using a communication protocol. The input devices 720 may include, for example, a mouse, a keyboard, a touch screen, an infrared sensor, a touch pad, a wearable input device, a camera-based or image-based input device, a microphone, or other user input device.

[0033] CPU 710 may be a single processing unit or multiple processing units within a single device or distributed across multiple devices. CPU 710 may be coupled to other hardware devices, for example using a bus such as a PCI bus or a SCSI bus. CPU 710 may communicate with a hardware controller for the device (e.g., for display 730). Display 730 may be used to display text and graphics. In some embodiments, display 730 provides visual feedback of graphics and text to a user. In some implementations, display 730 includes an input device as part of the display. This is the case, for example, when the input device is a touch screen or includes a line-of-sight monitoring system. In some implementations, the display and the input device are separate. Examples of display devices include an LCD display screen, an LED display screen, a projection display, a holographic display, or an augmented reality display (e.g., a head-up display device or a head-mounted device). Other I / O devices 740 may also be coupled to the processor, such as a network card, a video card, an audio card, a USB, Firewire or other external device, a camera, a printer, speakers, a CD-ROM drive, a DVD drive, a disk drive, or a Blu-Ray device.

[0034] In some implementations, the apparatus 700 also includes a communication device capable of communicating wirelessly or wired with a network node. The communication device is capable of communicating with another device or server over a network (e.g., using TCP / IP protocols). The apparatus 700 may utilize the communication device to distribute operations across multiple network devices.

[0035] The CPU 710 can access memory 750. Memory may include one or more of a variety of hardware devices for volatile and non-volatile storage, and may include both read-only and writeable memory. For example, memory may include random access memory (RAM), CPU registers, read-only memory (ROM), and writeable non-volatile memory (e.g., flash memory, hard drives, floppy disks, CDs, DVDs, magnetic storage devices, tape drives, device buffers, etc.). Memory is not a propagating signal separate from the underlying hardware. Thus, memory is non-transitory. Memory 750 may include program memory 760, which stores programs and software, such as an operating system 762, a word edit distance embedding platform 764, and other application programs 766. Memory 750 may also include data memory 770, which may include database information, etc., that may be provided to the program memory 760 or any element of the device 700.

[0036] Some implementations are operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that may be suitable for use with the present technology include, but are not limited to, personal computers, server computers, handheld or laptop devices, cellular telephones, mobile phones, wearable electronics, gaming consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, etc.

[0037] 9 is a block diagram illustrating an overview of an environment 800 in which some implementations of the disclosed technology can operate. The environment 800 can include one or more client computing devices 805A-D, examples of which can include device 700. The client computing devices 805 can operate in a networked environment using logical connections with one or more remote computers (e.g., a server computing device 810) via a network 830.

[0038] In some implementations, server computing device 810 may be an edge server that receives client requests and coordinates with other servers (e.g., servers 820A-C) to fulfill those requests. Server computing devices 810 and 820 may include computing systems such as device 700. Although each server computing device 810 and 820 is logically depicted as a single server, each server computing device may be a distributed computing environment that encompasses multiple computing devices at the same physical location or at multiple geographically distinct physical locations. In some implementations, each server computing device 820 corresponds to a group of servers.

[0039] Each of the client computing device 805 and server computing devices 810 and 820 may act as a server or client to other server / client devices. The server 810 may connect to a database 815. Each of the servers 820A-C may connect to a corresponding database 825A-C. As described above, each server 820 may represent a group of servers, each of which may share a database or each may have its own database. The databases 815 and 825 may warehouse (e.g., store) information. Although the databases 815 and 825 are shown as a logical single unit, each may be a distributed computing environment that includes multiple computing devices, or each may be in a corresponding server, or may be in the same physical location or multiple geographically different physical locations.

[0040] Network 830 may be a local area network (LAN) or a wide area network (WAN), but may also be other wired or wireless networks. Network 830 may be the Internet or any other public or private network. Client computing devices 805 may be connected to network 830 via a network interface (e.g., by wired or wireless communication). Although the connections between server 810 and server 820 are shown as separate connections, these connections may be any type of local, wide area, wired, or wireless network, including network 830 or a separate public or private network.

[0041] 10 is a block diagram illustrating components 900 that may be used in a system employing techniques of the present disclosure in some embodiments. Components 900 include hardware 902, general-purpose software 920, and special-purpose components 940. As discussed above, a system implementing techniques of the present disclosure may use a variety of hardware including a processing unit 904 (e.g., a CPU, GPU, APU, etc.), working memory 906, storage memory 908, and input / output devices 910. Components 900 may be implemented in a client computing device, such as client computing device 805, or in a server computing device, such as server computing device 810 or 820.

[0042] General purpose software 920 may include various applications, including an operating system 922, local programs 924, and a basic input / output system (BIOS) 926. Special purpose components 940 may be subcomponents of general purpose software applications 920 (e.g., local programs 924). Special purpose components 940 may include a preparation module 944, a training module 946, a matching module 948, and components (e.g., interface 942) that can be used to transfer data and control the special purpose components. In some implementations, components 900 may be in a computing system distributed across multiple computing devices or may be an interface to a server-based application that executes one or more of the special purpose components 940.

[0043] As will be appreciated by those skilled in the art, the components illustrated in Figures 8-10 and the flow diagrams described above may be modified in various ways. For example, the order of logic may be rearranged, substeps may be performed in parallel, illustrated logic may be omitted, other logic may be included, etc. In some implementations, one or more of the components described above may perform one or more of the processes described herein.

[0044] Although certain embodiments have been shown in the drawings and described in detail by way of example, other embodiments are possible. For example, in some embodiments, a system for classifying words in a batch of words may include at least one memory device storing instructions that cause at least one processor to: use a neural network (NN) to create a dictionary vector for each of a plurality of dictionary words, store each dictionary vector with a classification indicator corresponding to the associated dictionary word, and use the NN to create a word vector for each word in the batch of words for classification. A closest matching dictionary vector for each word vector is found, and a classification indicator of the closest matching dictionary vector for each word vector is reported.

[0045] In some embodiments, the dictionary words and the words in the batch are encoded into a matrix of length 30 by length 50. In some embodiments, the NN is a convolutional neural network. In some embodiments, each word vector is created without storing the corresponding words in the batch. In some embodiments, each dictionary vector and corresponding classification indicator is indexed to facilitate similarity search. In some embodiments, the system may further include training the NN, which includes calculating a similarity metric for a plurality of pairs of a plurality of training words, using the NN and the twin of the NN to create a pair of training vectors for each of the plurality of pairs of training words, calculating a cosine similarity between the training vectors of each pair, comparing the similarity metric and the cosine similarity for each pair of training words, and adjusting the NN based on the comparison. In some embodiments, the training words include dictionary words. In some embodiments, the similarity metric is equal to one less the Levenshtein edit distance divided by the length of the longest word in the pair. In some embodiments, the cosine similarity is calculated by dividing the dot product of the pair of training vectors by the product of their Euclidean norms. In some embodiments, each of the one or more closest matching dictionary vectors has an associated score, and reporting the classification indicators of the one or more closest matching dictionary vectors for each word vector in the batch includes selecting, for each word in the batch, a highest score for each classification indicator, and averaging the selected highest scores for each classification indicator.

[0046] In another representative embodiment, a system for classifying words in a batch of words may include at least one memory device storing instructions that cause at least one processor to train a convolutional neural network (CNN); encode each of a plurality of dictionary words into a matrix form; use the trained CNN to create a dictionary vector for each of a plurality of encoded dictionary words; store each dictionary vector with a classification indicator corresponding to an associated dictionary word; encode each word in the batch of words into a matrix form for classification; use the trained CNN to create a word vector for each encoded word in the batch; find a closest matching dictionary vector for each word vector; and report a classification indicator of the closest matching dictionary vector for each word vector in the batch. Training the CNN may include encoding each of a plurality of training words into a matrix form, calculating a similarity metric for a plurality of pairs of the plurality of training words, using the CNN and a twin of the CNN to create a pair of training vectors for each pair of the encoded training words corresponding to the plurality of pairs, calculating a cosine similarity between each pair of training vectors, comparing the similarity metric and the cosine similarity for each pair of training words and adjusting the CNN based on the comparison, and storing the trained CNN.

[0047] In another representative embodiment, a system for classifying words in a batch of words may include at least one memory device storing instructions that cause at least one processor to: train a neural network (NN); use the trained NN to create a dictionary vector for each of a plurality of dictionary words; store each dictionary vector with a classification indicator corresponding to an associated dictionary word; use the trained NN to create a word vector for each word in the batch of words for classification; find a closest matching dictionary vector for each word vector; and report a classification indicator of the closest matching dictionary vector for each word vector in the batch.

[0048] Training the NN may include calculating a similarity metric for a plurality of pairs of a plurality of training words (the similarity metric is equal to one less the Levenshtein edit distance divided by the length of the longest word in the pair); using the NN and the twin of the NN to create a pair of training vectors for each of the plurality of pairs of training words; calculating a cosine similarity between the training vectors of each pair; comparing the similarity metric and the cosine similarity for each pair of training words; and adjusting the NN based on the comparison.

[0049] The following examples provide further embodiments of the present technology. Working Example Example 1 1. A system for classifying words in a batch of words, comprising: At least one memory device, creating a dictionary vector for each of a plurality of dictionary words using a neural network (NN); storing each dictionary vector with a classification indicator corresponding to said associated dictionary word; using the NN to create a word vector for each word in a batch of words for classification; finding one or more closest matching dictionary vectors for each word vector; reporting the classification indicator of the one or more closest matching dictionary vectors for each word vector in the batch; said at least one memory device storing instructions for causing at least one processor to carry out A system including: Example 2 2. The system of example 1, wherein the dictionary words and the words in the batch are encoded into matrices of length 30 by length 50. Example 3 3. The system of claim 1 or 2, wherein the NN is a convolutional neural network. Example 4 4. The system of any one of embodiments 1-3, wherein each word vector is created without storing the corresponding words in the batch. Example 5 The system of any one of embodiments 1 to 4, wherein each dictionary vector and corresponding classification indicator is indexed to facilitate similarity searching. Example 6 Each of the one or more closest matching dictionary vectors has an associated score, and the step of reporting the classification indicator of the one or more closest matching dictionary vectors for each word vector in the batch comprises: selecting, for each word in the batch, the highest score for each classification indicator; averaging the selected best scores for each classification indicator; Including, The system according to any one of the first to fifth embodiments. Example 7 Training the NN, calculating a similarity metric for a plurality of pairs of a plurality of training words; generating a pair of training vectors for each of the plurality of pairs of training words using the NN and a twin of the NN; calculating the cosine similarity between each pair of training vectors; comparing the similarity metric and the cosine similarity for each pair of training words and adjusting the NN based on the comparison; training the NN, The system according to any one of the first to sixth embodiments. Example 8 8. The system of example 7, wherein the training words include the dictionary words. Example 9 9. The system of any one of examples 7 to 8, wherein the similarity metric is equal to one less the Levenshtein edit distance divided by the length of the longest word in the pair. Example 10 10. The system of any one of claims 7 to 9, wherein the cosine similarity is calculated by dividing the dot product of the pair of training vectors by the product of their Euclidean norms. Example 11 1. A system for classifying words in a batch of words, comprising: At least one memory device, Training a convolutional neural network (CNN), encoding each of a plurality of training words into a matrix form; calculating a similarity metric for a plurality of pairs of the plurality of training words; creating a pair of training vectors for each pair of encoded training words corresponding to the plurality of pairs using the CNN and a twin of the CNN; calculating the cosine similarity between each pair of training vectors; comparing the similarity metric and the cosine similarity for each pair of training words and adjusting the CNN based on the comparison; storing the trained CNN; training the CNN, encoding each of a plurality of dictionary words into a matrix form; creating a dictionary vector for each of the plurality of encoded dictionary words using the trained CNN; storing each dictionary vector with a classification indicator corresponding to said associated dictionary word; encoding each word in the batch of words into a matrix form for classification; creating a word vector for each encoded word in the batch using the trained CNN; finding one or more closest matching dictionary vectors for each word vector; reporting the classification indicator of the one or more closest matching dictionary vectors for each word vector in the batch; said at least one memory device storing instructions for causing at least one processor to carry out A system including: Example 12 12. The system of example 11, wherein the training words include the dictionary words. Example 13 13. The system of example 11 or 12, wherein the similarity metric is equal to one less the Levenshtein edit distance divided by the length of the longest word in the pair. Example 14 14. The system of any one of claims 11 to 13, wherein the cosine similarity is calculated by dividing the dot product of the pair of training vectors by the product of their Euclidean norms. Example 15 15. The system of any one of claims 11 to 14, wherein the training words, the dictionary words, and the words in the batch are encoded into a matrix of length 30 characters by length 50. Example 16 16. The system of any one of embodiments 11 to 15, wherein each word vector is created without storing the corresponding words in the batch. Example 17 The system of any one of embodiments 11 to 16, wherein each dictionary vector and corresponding classification indicator is indexed to facilitate similarity searching. Example 18 Each of the one or more closest matching dictionary vectors has an associated score, and the step of reporting the classification indicator of the one or more closest matching dictionary vectors for each word vector in the batch comprises: selecting, for each word in the batch, the highest score for each classification indicator; averaging the selected best scores for each classification indicator; Including, The system according to any one of Examples 11 to 17. Example 19 1. A system for classifying words in a batch of words, comprising: At least one memory device, Training a neural network (NN), calculating a similarity metric for a plurality of pairs of a plurality of training words, the similarity metric being equal to one less the Levenshtein edit distance divided by the length of the longest word in the pair; generating a pair of training vectors for each of the plurality of pairs of training words using the NN and a twin of the NN; calculating the cosine similarity between each pair of training vectors; comparing the similarity metric and the cosine similarity for each pair of training words and adjusting the NN based on the comparison; training the NN, creating a dictionary vector for each of a plurality of dictionary words using the trained NN; storing each dictionary vector with a classification indicator corresponding to said associated dictionary word; using the trained NN to create a word vector for each word in a batch of words for classification; finding one or more closest matching dictionary vectors for each word vector; reporting the classification indicator of the one or more closest matching dictionary vectors for each word vector in the batch; said at least one memory device storing instructions for causing at least one processor to carry out A system including: Example 20 20. The system of example 19, wherein the cosine similarity is calculated by dividing the dot product of the pair of training vectors by the product of their Euclidean norms. Example 21 Each of the one or more closest matching dictionary vectors has an associated score, and the step of reporting the classification indicator of the one or more closest matching dictionary vectors for each word vector in the batch comprises: selecting, for each word in the batch, the highest score for each classification indicator; averaging the selected best scores for each classification indicator; Including, The system described in Example 19 or 20. Example 22 The system of any one of Examples 19 to 21, wherein the training words, the dictionary words, and the words in the batch are encoded into a matrix of length 30 by length 50, and then corresponding vectors are created by the NN. Example 23 23. The system of any one of Examples 19 to 22, wherein the NN is a convolutional neural network. Example 24 25. The system of any one of embodiments 19-24, wherein each word vector is created without storing the corresponding words in the batch. Example 25 The system of any one of Examples 19 to 24, wherein each dictionary vector and corresponding classification indicator is indexed to facilitate similarity searching. Example 26 26. The system of any one of Examples 19 to 25, wherein the training words include the dictionary words. Note

[0050] The above description and drawings are illustrative and should not be construed as limiting. Numerous specific details have been set forth to provide a thorough understanding of the present disclosure. However, in some cases, well-known details have not been set forth to avoid obscuring the description. Furthermore, various modifications may be made without departing from the scope of the embodiments.

[0051] References herein to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described with respect to that embodiment is included in at least one embodiment of the present disclosure. The appearances of the phrase "in one embodiment" in various places in this specification do not necessarily all refer to the same embodiment, nor do they refer to separate or alternative embodiments that are mutually exclusive with respect to other embodiments. Moreover, various features described may be exhibited by some embodiments and not by other embodiments. Similarly, various requirements described may be requirements for some embodiments and not for other embodiments.

[0052] The terms used herein generally have their ordinary meaning in the art, within the context of this disclosure, and in the specific context in which each term is used. It is to be understood that the same thing may be said in more than one way. Thus, alternative phrases or synonyms may be used for any one or more of the terms described herein, and no special meaning is given to the term, whether or not it is detailed or discussed in this specification. Synonyms are given for some terms. The description of one or more synonyms does not exclude the use of other synonyms. The use of examples anywhere in this specification, including examples of any terms described herein, is merely illustrative and is not intended to further limit the scope and meaning of the disclosure or of any exemplary term. Similarly, this disclosure is not limited to the various embodiments provided herein. Unless otherwise defined, the meaning of all technical and scientific terms used herein is the same as that commonly understood by one of ordinary skill in the art to which this disclosure belongs. In case of conflict, this document, including definitions, will control. [Appendix 1] 1. A system for classifying words in a batch of words, comprising: At least one memory device, creating a dictionary vector for each of a plurality of dictionary words using a neural network (NN); storing each dictionary vector with a classification indicator corresponding to said associated dictionary word; using the NN to create a word vector for each word in a batch of words for classification; finding one or more closest matching dictionary vectors for each word vector; reporting the classification indicator of the one or more closest matching dictionary vectors for each word vector in the batch; said at least one memory device storing instructions for causing at least one processor to carry out A system including: [Appendix 2] 2. The system of claim 1, wherein the dictionary words and the words in the batch are encoded into matrices of length 30 characters by length 50. [Appendix 3] 2. The system of claim 1, wherein the NN is a convolutional neural network. [Appendix 4] 2. The system of claim 1, wherein each word vector is created without storing the corresponding word in the batch. [Appendix 5] 2. The system of claim 1, wherein each dictionary vector and corresponding classification indicator is indexed to facilitate similarity searching. [Appendix 6] Each of the one or more closest matching dictionary vectors has an associated score, and the step of reporting the classification indicator of the one or more closest matching dictionary vectors for each word vector in the batch comprises: selecting, for each word in the batch, the highest score for each classification indicator; averaging the selected best scores for each classification indicator; Including, 2. The system of claim 1. [Appendix 7] Training the NN, calculating a similarity metric for a plurality of pairs of a plurality of training words; generating a pair of training vectors for each of the plurality of pairs of training words using the NN and a twin of the NN; calculating the cosine similarity between each pair of training vectors; comparing the similarity metric and the cosine similarity for each pair of training words and adjusting the NN based on the comparison; training the NN, 2. The system of claim 1. [Appendix 8] 8. The system of claim 7, wherein the training words include the dictionary words. [Appendix 9] 8. The system of claim 7, wherein the similarity metric is equal to one less a Levenshtein edit distance divided by the length of the longest word in the pair. [Appendix 10] 8. The system of claim 7, wherein the cosine similarity is calculated by dividing the dot product of the pair of training vectors by the product of their Euclidean norms. [Appendix 11] 1. A system for classifying words in a batch of words, comprising: At least one memory device, Training a convolutional neural network (CNN), encoding each of a plurality of training words into a matrix form; calculating a similarity metric for a plurality of pairs of the plurality of training words; creating a pair of training vectors for each pair of encoded training words corresponding to the plurality of pairs using the CNN and a twin of the CNN; calculating the cosine similarity between each pair of training vectors; comparing the similarity metric and the cosine similarity for each pair of training words and adjusting the CNN based on the comparison; storing the trained CNN; training the CNN, encoding each of a plurality of dictionary words into a matrix form; creating a dictionary vector for each of the plurality of encoded dictionary words using the trained CNN; storing each dictionary vector with a classification indicator corresponding to said associated dictionary word; encoding each word in the batch of words into a matrix form for classification; creating a word vector for each encoded word in the batch using the trained CNN; finding one or more closest matching dictionary vectors for each word vector; reporting the classification indicator of the one or more closest matching dictionary vectors for each word vector in the batch; said at least one memory device storing instructions for causing at least one processor to carry out A system including: [Appendix 12] 12. The system of claim 11, wherein the training words include the dictionary words. [Appendix 13] 12. The system of claim 11, wherein the similarity metric is equal to one less Levenshtein edit distance divided by the length of the longest word in the pair. [Appendix 14] 12. The system of claim 11, wherein the cosine similarity is calculated by dividing the dot product of the pair of training vectors by the product of their Euclidean norms. [Appendix 15] 12. The system of claim 11, wherein the training words, the dictionary words, and the words in the batch are encoded into matrices of length 30 characters by length 50. [Appendix 16] 12. The system of claim 11, wherein each word vector is created without storing the corresponding word in the batch. [Appendix 17] 12. The system of claim 11, wherein each dictionary vector and corresponding classification indicator is indexed to facilitate similarity searching. [Appendix 18] Each of the one or more closest matching dictionary vectors has an associated score, and the step of reporting the classification indicator of the one or more closest matching dictionary vectors for each word vector in the batch comprises: selecting, for each word in the batch, the highest score for each classification indicator; averaging the selected best scores for each classification indicator; Including, 12. The system of claim 11. [Appendix 19] 1. A system for classifying words in a batch of words, comprising: At least one memory device, Training a neural network (NN), calculating a similarity metric for a plurality of pairs of a plurality of training words, the similarity metric being equal to one less the Levenshtein edit distance divided by the length of the longest word in the pair; generating a pair of training vectors for each of the plurality of pairs of training words using the NN and a twin of the NN; calculating the cosine similarity between each pair of training vectors; comparing the similarity metric and the cosine similarity for each pair of training words and adjusting the NN based on the comparison; training the NN, creating a dictionary vector for each of a plurality of dictionary words using the trained NN; storing each dictionary vector with a classification indicator corresponding to said associated dictionary word; using the trained NN to create a word vector for each word in a batch of words for classification; finding one or more closest matching dictionary vectors for each word vector; reporting the classification indicator of the one or more closest matching dictionary vectors for each word vector in the batch; said at least one memory device storing instructions for causing at least one processor to carry out A system including: [Appendix 20] 20. The system of claim 19, wherein the cosine similarity is calculated by dividing the dot product of the pair of training vectors by the product of their Euclidean norms. [Appendix 21] Each of the one or more closest matching dictionary vectors has an associated score, and the step of reporting the classification indicator of the one or more closest matching dictionary vectors for each word vector in the batch comprises: selecting, for each word in the batch, the highest score for each classification indicator; averaging the selected best scores for each classification indicator; Including, 20. The system of claim 19. [Appendix 22] 20. The system of claim 19, wherein the training words, the dictionary words, and the words in the batch are encoded into matrices of length 30 by length 50, and then corresponding vectors are created by the NN. [Appendix 23] 23. The system of claim 22, wherein the NN is a convolutional neural network. [Appendix 24] 24. The system of claim 23, wherein each word vector is created without storing the corresponding word in the batch. [Appendix 25] 25. The system of claim 24, wherein each dictionary vector and corresponding classification indicator is indexed to facilitate similarity searching. [Appendix 26] 26. The system of claim 25, wherein the training words include the dictionary words.

Claims

1. 1. A system for classifying words in a batch of words, comprising: At least one memory device, creating a dictionary vector for each of a plurality of dictionary words using a neural network (NN); storing each dictionary vector with a classification indicator corresponding to said associated dictionary word; using the neural network to create a word vector for each word in a batch of words for classification; finding one or more closest matching dictionary vectors for each word vector; reporting the classification indicators of the one or more closest matching dictionary vectors for each word vector in the batch; the at least one memory device storing instructions for causing at least one processor to perform Training the neural network, calculating a similarity metric for a plurality of pairs of a plurality of training words; generating a pair of training vectors for each of said plurality of pairs of training words using said NN and a twin of said NN; calculating the cosine similarity between each pair of training vectors; comparing the similarity metric and the cosine similarity for each pair of training words and adjusting the NN based on the comparison; training the NN, system.

2. The system of claim 1 , wherein the dictionary words and the words in the batch are encoded into a matrix of length 30 characters by length 50.

3. The system of claim 1 , wherein the NN is a convolutional neural network.

4. The system of claim 1 , wherein each word vector is created without storing the corresponding word in the batch.

5. The system of claim 1 , wherein each dictionary vector and corresponding classification indicator is indexed to facilitate similarity searching.

6. Each of the one or more closest matching dictionary vectors has an associated score, and the step of reporting the classification indicator of the one or more closest matching dictionary vectors for each word vector in the batch comprises: selecting, for each word in the batch, the highest score for each classification indicator; averaging the selected best scores for each classification indicator; Including, The system of claim 1 .

7. The system of claim 1, wherein the training words include the dictionary words.

8. The system of claim 1, wherein the similarity metric is equal to one less the Levenshtein edit distance divided by the length of the longest word in the pair.

9. The system of claim 1, wherein the cosine similarity is calculated by dividing the dot product of the pair of training vectors by the product of their Euclidean norms.

10. A system for classifying words in a batch of words, comprising: At least one memory device, Training a convolutional neural network (CNN), encoding each of a plurality of training words into a matrix form; calculating a similarity metric for a plurality of pairs of the plurality of training words; creating a pair of training vectors for each pair of encoded training words corresponding to the plurality of pairs using the CNN and a twin of the CNN; calculating the cosine similarity between each pair of training vectors; comparing the similarity metric and the cosine similarity for each pair of training words and adjusting the CNN based on the comparison; storing the trained CNN; training the CNN, encoding each of a plurality of dictionary words into a matrix form; creating a dictionary vector for each of the plurality of encoded dictionary words using the trained CNN; storing each dictionary vector with a classification indicator corresponding to said associated dictionary word; encoding each word in the batch of words into a matrix form for classification; using the trained CNN to create a word vector for each encoded word in the batch; finding one or more closest matching dictionary vectors for each word vector; reporting the classification indicators of the one or more closest matching dictionary vectors for each word vector in the batch; said at least one memory device storing instructions for causing at least one processor to carry out A system including:

11. The system of claim 10, wherein the training words include the dictionary words.

12. The system of claim 10, wherein the similarity metric is equal to one less the Levenshtein edit distance divided by the length of the longest word in the pair.

13. The system of claim 10, wherein the cosine similarity is calculated by dividing the dot product of the pair of training vectors by the product of their Euclidean norms.

14. The one or more closest matching dictionary vectors each have an associated score, and the step of reporting the classification indicator of the one or more closest matching dictionary vectors for each word vector in the batch comprises: selecting, for each word in the batch, the highest score for each classification indicator; averaging the selected best scores for each classification indicator; Including, The system of claim 10.

Images