A system for classifying words in a batch of words

Abstract

A system for classifying words in a batch of words may include at least one storage device storing instructions for causing at least one processor to create a dictionary vector for each of a plurality of dictionary words using a neural network (NN), store each dictionary vector and a classification indicator corresponding to the associated dictionary word, and create a word vector for each word in a batch of words for classification using the NN, find the dictionary vector that best matches each word vector, and report the classification indicator of the best matching dictionary vector for each word vector in the batch.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 008,552, filed on April 10, 2020, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This patent application relates to text similarity calculation, and more specifically, to word edit distance embedding. Background Art

[0004] Businesses that handle personal information, whether it's employee information, medical data, or financial information, need to protect that information and limit its use. Additionally, any data collected should be anonymized and stored only as long as necessary. Governments have enacted regulations, such as the European Union's General Data Protection Regulation (GDPR), to help protect personal information. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The systems and methods described herein may be better understood by referring to the following detailed description in conjunction with the accompanying drawings, wherein like reference numerals indicate identical or functionally similar elements:

[0006] Figure 1 is a flowchart of a method of operation of a processor-based word edit distance embedding system according to some embodiments of the present technology;

[0007] Figure 2 is a flow chart illustrating a method for training a neural network for classifying words according to some embodiments of the present technology;

[0008] Figure 3 shows a two-dimensional matrix for encoding words according to some embodiments of the present technology;

[0009] Figure 4 is a schematic diagram of a training architecture with a Siamese convolutional neural network according to some embodiments of the present technology;

[0010] Figure 5 is a flowchart illustrating a method of setting a word vector index for classifying a word according to some embodiments of the present technology;

[0011] Figure 6 is a graph showing dictionary word embeddings and associated categories and thresholds;

[0012] Figure 7 is a flow chart illustrating a method for classifying words according to some embodiments of the present technology;

[0013] Figure 8 is a block diagram showing an overview of a device in which some embodiments may operate;

[0014] Fig. 9 is a block diagram showing an overview of an environment in which some embodiments may operate; and

[0015] Fig.10 is a block diagram illustrating components that, in some implementations, may be used in a system employing the disclosed techniques.

[0016] The headings provided herein are for convenience only and do not necessarily affect the scope of the embodiments. In addition, the drawings are not necessarily drawn to scale. For example, the sizes of some elements in the drawings may be enlarged or reduced to help improve the understanding of the embodiments. In addition, although the disclosed technology is suitable for a variety of modifications and alternative forms, specific embodiments have been shown in the drawings by way of example and are described in detail below. However, it is not intended to unnecessarily limit the described embodiments. In contrast, the embodiments are intended to cover all modifications, combinations, equivalents, and alternatives that fall within the scope of the present disclosure. DETAILED DESCRIPTION

[0017] Multiple examples of the above-mentioned systems and methods will now be described in further detail. The following description provides specific details so as to fully understand and implement the description of these examples. However, it will be appreciated by those skilled in the relevant art that the processes and techniques discussed herein can be practiced without many of these details. Similarly, it will be appreciated by those skilled in the relevant art that the present technology may include many other features not described in detail herein. In addition, some well-known structures or functions may not be shown or described in detail below to avoid unnecessarily obscuring the relevant description.

[0018] The terms used below should be interpreted in their broadest reasonable manner, even if it is used in conjunction with a detailed description of some specific examples of embodiments. In fact, some terms may even be emphasized below; however, any term intended to be interpreted in any limited manner will be clearly and specifically defined in this section.

[0019] Disclosed herein are methods and systems for classifying unknown words using a novel neural network and similarity search architecture. In the context of protecting personal information, it is desirable to monitor whether personal information is transmitted within and / or outside a network. The disclosed technology enables these transmissions to be monitored by securely anonymizing the words in transmission and comparing them to words in a variety of category dictionaries (e.g., country, name, race, language, religion, gender, marital status, etc.) to determine the category of information being transmitted. Beneficially, the disclosed technology securely anonymizes and classifies words without ever saving the original words.

[0020] In some embodiments, a neural network is trained and retrained so that the distance between pairs of vectors based on a word dictionary data set approximates a metric based on edit distance between corresponding words. Training includes creating vectors for all dictionary words using the trained neural network and storing the vectors in a specific structure. A matching algorithm then creates vectors for all words in a batch, including finding the closest neighbor match for each word and applying a score based on the distance to the dictionary's closest match to determine the classification of each word.

[0021] Figure 1 1 is a high-level flow chart of a method 100 of operation of a processor-based word edit distance embedding system for classifying batches of data (e.g., words) in accordance with some embodiments of the present technology. The method may include training a neural network (NN) with multiple pairs of words at step 102 so that the distance between corresponding vectors approaches an edit distance-based metric between the words. At step 104, vectors for a selection of dictionary words are created using the trained NN. The resulting vectors may be stored at step 106. Next, vectors are created for all words in a batch of words to be classified at step 108. The closest match for each word in the batch is determined at step 110. In some embodiments, at step 112, a score may be applied to each word in the batch based on the distance to the associated closest match.

[0022] Figure 2 A method 200 for training a neural network for classifying words according to some embodiments of the present technology is shown. In some embodiments, the neural network may be a convolutional neural network (CNN). For example, training the CNN may include encoding each of a plurality of training words into a matrix form ( Figure 3 ). The training words may include a dictionary of words belonging to a particular category (e.g., country, name, race, language, religion, gender, marital status, etc.). The training dictionary may include a base dictionary, a complete English dictionary, and a non-word dictionary containing data that is not similar to any type of data from similar dictionaries.

[0023] At step 204, CNN and CNN twins are used to create training vector pairs for multiple pairs of encoded training words ( Figure 4 ). Pairs can be randomly selected from the training dictionary to train for large word edit distances, and pairs can be created by adding noise to the words to train for small word edit distances.

[0024] At step 206, a similarity measure (SM) is calculated for each of the plurality of pairs of training words of the plurality of training words. The SM may be calculated based on an edit distance (ED) (eg, Levenshtein ED) as follows:

[0025] SM(word1,word2)=1-ED(word1,word2) / max(length(word1),length(word2))

[0026] In some embodiments, the SM may be based on the encoded form of the word.

[0027] At step 208, the similarity metric and cosine similarity of each pair of training words are compared, and the CNN is adjusted (e.g., weights are adjusted) based on the comparison to drive the cosine similarity between the pairs to match the similarity metric. Using cosine similarity instead of Euclidean distance between vectors can improve computation time and accuracy. Once the CNN is trained, it is stored at step 210. In some embodiments, cosine similarity can be calculated by dividing the dot product of the vector pair by the product of their Euclidean norms.

[0028] Using the above mentioned similarity metric instead of the traditional Levenshtein ED improves the search. This can be illustrated with two examples: two 2-letter codes - PL and PS, and two words - Poland and Roland. Both pairs have ED = 1. But the value 1 has a much greater impact on the first pair, because it changes half of the word. Therefore, when the error in the edit distance is about "1", the accuracy of using ED for short words is much lower than that for long words. In contrast, the disclosed similarity metric loses precision on longer pairs (especially on pairs with large edit distances), which is acceptable in the disclosed system because it searches for words within some range, and if it is too large, the exact distance value is not so important. In some embodiments, the minimum similarity threshold for accepting a match is 0.7.

[0029] Figure 3 A two-dimensional matrix 220 for encoding a word (e.g., "Poland") is shown according to some embodiments of the present technology. The disclosed two-dimensional matrix is ​​similar to the matrix described by Lluís Gómez, and Dimosthenis Karatzas, LSDE: Levenshtein Space Deep Embedding for Query-by-string Word Spotting, International Conference on Document Analysis and Recognition (ICDAR) 2017, which is incorporated herein by reference in its entirety. The matrix 220 includes a first dimension 222 including 26 alphabetic characters and four special characters and a second dimension 224 including character positions in the string. The number of character positions can be limited to, for example, 50 positions, thereby generating a 30×50 matrix. It should be noted that the matrix 220 can encode both alphabetic and non-alphabetic characters, as well as an “empty” character briefly described below.

[0030] Each character in the dataset can be grouped as follows:

[0031] 1. Alphabetic characters - Alphabetic characters are normalized (eg, é = e), resulting in the 26 basic alphabetic characters.

[0032] 2. Numbers - Create a "bag" for all numeric characters, represented by "#".

[0033] 3. Separation characters - Create a "bag" for all separation characters, represented by "-" (e.g., '', '-', '_', '.', ',', ':', ' / '). However, if any of these characters appear more than a threshold minimum number of times in the dataset, they are considered independent characters.

[0034] 4. "Empty" position, represented by "*".

[0035] 5. Characters that appear in the dataset less than the threshold minimum number of times will be excluded and represented by “?”.

[0036] Figure 4 30 is a schematic diagram of a training architecture 300 with twin convolutional neural networks according to some embodiments of the present technology. Twin CNNs 302(1) and 302(2) have the same structure and the same weights. The input to the twin CNNs is the same 30×50 matrix 304, as described above with respect to Figure 3Those described. In some embodiments, the CNN applies a convolutional layer 308 with 64 kernels 306 of size 30×3 and a fully connected layer 310 with 128 output neurons. The cosine similarity 312 corresponding to each vector pair of word1 and word2 is calculated and compared with the similarity measure between the encoded word1 and word2 to obtain a "loss" 316. The CNN is driven to learn the optimal conversion of words (e.g., strings) to vector form so that the cosine similarity distance between vectors is equal to the similarity measure of their original words. In other words, the loss 316 is approached to zero.

[0037] Figure 5 is a flowchart showing a method 400 of setting a word vector index for classifying a word according to some embodiments of the present technology. Figure 2-4 After training the CNN, at step 402, multiple dictionary words are encoded into a matrix form, such as the above Figure 3 The matrix described. In some embodiments, the dictionary words used to train the CNN may include dictionary words for classification. At step 404, a dictionary vector for each of a plurality of encoded dictionary words is created using the trained CNN. At step 406, the resulting dictionary vectors and the categories corresponding to the associated dictionary words are stored. At step 408, the dictionary word vectors may be indexed for efficient searching. In some embodiments, the vectors are indexed according to Facebook Artificial Intelligence Similarity Search (FAISS) technology. Figure 6 An example data structure 420 of dictionary word vector embeddings 426 and associated categories 428 and thresholds 430 is shown. For example, a country dictionary 422 includes the names of multiple countries, and a name dictionary 424 includes multiple given names. Each individual country and given name corresponds to a vector embedding 426. The category code 428 for countries is "0" and the category code 428 for names is "1". The data structure 420 includes a mapping 430 of category codes and category thresholds to each vector embedding. In some embodiments, the threshold for each category can be determined experimentally to minimize false positives.

[0038] Figure 7 is a flow chart illustrating a method 500 for classifying words according to some embodiments of the present technology. Once the CNN is trained and the dictionary word index is prepared, the words in a batch of words of interest can be classified to determine whether the words in the batch fall into any sensitive data categories, such as personal information. At step 502, a batch of words for classification is received but not stored. At step 504, for example, using the reference above Figure 3The described technology encodes the received words into a matrix form. At step 506, a word vector for each of a plurality of encoded batches of words is created using a trained CNN. At step 508, the best match and score for each resulting word vector in the batch is found in the index of the dictionary word vector by searching for the closest neighbor using, for example, the FAISS library. FAISS is a library for efficient similarity search and dense vector clustering. It contains algorithms for searching vector sets of any size. At step 510, the average results for each category corresponding to the batch of words are reported.

[0039] Step 510 can be illustrated with an example batch of words containing two words: Holand and Cuba. For each word, the four best matches and corresponding scores are obtained as follows:

[0040] Holand-Holand,Poland,Roland,Holland

[0041] Cuba-Cuba,Coby,Curt,Aruba

[0042] Each match is associated with a category and an achieved score as follows:

[0043] Holand-Holand (surname -1.0), Poland (country -0.8), Roland (first name -0.8), Holland (country -0.8)

[0044] Cuba-Cuba (country-1.0), Coby (name-0.5), Curt (name-0.5), Aruba (country-0.6)

[0045] For each word in the batch, select the highest score for each category, as follows:

[0046] Holand - Surname: 1.0, Country: 0.8, First Name: 0.8

[0047] Cuba - Country: 1.0, Name: 0.5

[0048] Given the following class thresholds:

[0049] Country: 0.8

[0050] Last name: 0.8

[0051] Name: 0.7

[0052] The above thresholds are applied to the highest score for each category separately. If the score is greater than or equal to the corresponding threshold, the score is retained, and if it is less than the threshold, the score is considered to be 0, as follows:

[0053] Holand - Surname: 1.0, Country: 0.8, First Name: 0.8

[0054] Cuba-Country: 1.0, Name: 0

[0055] Next, the results for each category are averaged as follows:

[0056] Last name: (1+0) / 2=0.5

[0057] Country: (0.8+1) / 2=0.9

[0058] Name: (0.8+0) / 2=0.4

[0059] And, report the results as follows:

[0060] Country: 0.9

[0061] Last name: 0.5

[0062] Name: 0.4

[0063] Without storing or exposing the underlying data, the reported results can indicate to the user the likelihood that certain categories of data (eg, country, last name, and first name) are being transmitted on the system.

[0064] Suitable system

[0065] The technology disclosed herein may be embodied as dedicated hardware (e.g., circuits), programmable circuits suitably programmed with software and / or firmware, or a combination of dedicated circuits and programmable circuits. Thus, embodiments may include a machine-readable medium having instructions stored thereon, which instructions may be used to cause a computer, a microprocessor, a processor, and / or a microcontroller (or other electronic device) to perform a process. The machine-readable medium may include, but is not limited to, an optical disk, a compact disk read-only memory (CD-ROM), a magneto-optical disk, a ROM, a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic card or an optical card, a flash memory, or other types of media / machine-readable media suitable for storing electronic instructions.

[0066] Several embodiments are discussed in more detail below with reference to the accompanying drawings. Figure 8is a block diagram illustrating an overview of a device in which some embodiments of the disclosed technology may operate. The device may include hardware components of a device 700 for determining a risk score and associated pricing and / or risk ratio. The device 700 may include one or more input devices 720 that provide input to a CPU (processor) 710 to notify it of an action. The action is typically mediated by a hardware controller that interprets the signals received from the input device and transmits the information to the CPU 710 using a communication protocol. The input device 720 includes, for example, a mouse, a keyboard, a touch screen, an infrared sensor, a touchpad, a wearable input device, a camera-based or image-based input device, a microphone, or other user input device.

[0067] CPU 710 can be a single processing unit or multiple processing units in a device or distributed in multiple devices. For example, CPU 710 can be coupled to other hardware devices using a bus, such as a PCI bus or a SCSI bus. CPU 710 can communicate with a hardware controller of a device (e.g., display 730). Display 730 can be used to display text and graphics. In some examples, display 730 provides graphical and textual visual feedback to the user. In some embodiments, display 730 includes an input device as part of the display, such as when the input device is a touch screen or is equipped with an eye direction monitoring system. In some embodiments, the display is separate from the input device. Examples of display devices are: LCD display screens; LED display screens; projection, holographic, or augmented reality displays (e.g., head-up display devices or head-mounted devices); and so on. Other I / O devices 740 can also be coupled to the processor, such as a network card, video card, sound card, USB, FireWire or other external devices, cameras, printers, speakers, CD-ROM drives, DVD drives, disk drives, or Blu-ray devices.

[0068] In some implementations, the device 700 also includes a communication device capable of wireless or wired communication with a network node. The communication device can communicate with another device or server via a network using, for example, the TCP / IP protocol. The device 700 can use the communication device to distribute operations to multiple network devices.

[0069] CPU 710 may have access to memory 750. Memory includes one or more of a variety of hardware devices for volatile and non-volatile storage, and may include read-only and writeable memory. For example, memory may include random access memory (RAM), CPU registers, read-only memory (ROM), and writeable non-volatile memory such as flash memory, hard drives, floppy disks, CDs, DVDs, magnetic storage devices, tape drives, device buffers, and the like. Memory is not a propagating signal that is separate from the underlying hardware; therefore, memory is non-temporary. Memory 750 may include program memory 760, which stores programs and software, such as an operating system 762, a word edit distance embedding platform 764, and other applications 766. Memory 750 may also include data storage 770, which may include database information, etc. that may be provided to program memory 760 or any element of device 700.

[0070] Some embodiments may operate with many other general or special purpose computing system environments or configurations. Examples of known computing systems, environments, and / or configurations that may be suitable for use with the present technology include, but are not limited to, personal computers, server computers, handheld or laptop devices, cellular phones, mobile phones, wearable electronics, game consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like.

[0071] Fig. 9 8 is a block diagram illustrating an overview of an environment 800 in which some embodiments of the disclosed technology may operate. The environment 800 may include one or more client computing devices 805A-D, examples of which may include the device 700. The client computing device 805 may operate in a networked environment connected to one or more remote computers (e.g., a server computing device 810) through a network 830 using logical connections.

[0072] In some embodiments, server computing device 810 may be an edge server that receives client requests and coordinates the implementation of these requests through other servers (e.g., servers 820A-C). Server computing devices 810 and 820 may include computing systems, such as device 700. Although each server computing device 810 and 820 is logically shown as a server, the server computing devices may each be a distributed computing environment that includes multiple computing devices located in the same or geographically different physical locations. In some embodiments, each server computing device 820 corresponds to a group of servers.

[0073] The client computing device 805 and the server computing devices 810 and 820 can each act as a server or client to other server / client devices. The server 810 can be connected to a database 815. The servers 820A-C can each be connected to a corresponding database 825A-C. As described above, each server 820 can correspond to a group of servers, and each of these servers can share a database or can have its own database. The databases 815 and 825 can store information (e.g., store). Although the databases 815 and 825 are logically displayed as a unit, the databases 815 and 825 can each be a distributed computing environment containing multiple computing devices, which can be located in their corresponding servers, or can be located in the same or geographically different physical locations.

[0074] The network 830 may be a local area network (LAN) or a wide area network (WAN), but may also be other wired or wireless networks. The network 830 may be the Internet or some other public or private network. The client computing device 805 may be connected to the network 830 via a network interface (e.g., via wired or wireless communication). Although the connections between the server 810 and the server 820 are shown as separate connections, these connections may be any type of local, wide area, wired, or wireless network, including the network 830 or a separate public or private network.

[0075] Fig.10 900 is a block diagram illustrating components that, in some implementations, may be used in a system employing the disclosed technology. Components 900 include hardware 902, general software 920, and application-specific components 940. As discussed above, a system implementing the disclosed technology may use a variety of hardware, including a processing unit 904 (e.g., a CPU, GPU, APU, etc.), a working memory 906, a storage memory 908, and input and output devices 910. Components 900 may be implemented in a client computing device such as client computing device 805 or on a server computing device such as server computing device 810 or 820.

[0076] The general software 920 may include a variety of applications, including an operating system 922, a local program 924, and a basic input and output system (BIOS) 926. The special components 940 may be subcomponents of the general software applications 920, such as the local program 924. The special components 940 may include a preparation module 944, a training module 946, a matching module 948, and components that can be used to transfer data and control the special components, such as an interface 942. In some embodiments, the component 900 may be located in a computing system distributed across multiple computing devices, or may be an interface to a server-based application that executes one or more special components 940.

[0077] Those skilled in the art will understand that the above Figure 8-10 The components shown in and in each of the flowcharts discussed above may be changed in a variety of ways. For example, the order of the logic may be rearranged, sub-steps may be performed in parallel, the shown logic may be omitted, other logic may be included, etc. In some embodiments, one or more of the components described above may perform one or more processes described herein.

[0078] Although specific embodiments have been shown by way of example in the drawings and described in detail above, other embodiments are possible. For example, in some embodiments, a system for classifying words in a batch of words may include at least one storage device storing instructions for causing at least one processor to create a dictionary vector for each word in a plurality of dictionary words using a neural network (NN), store each dictionary vector and a classification indicator corresponding to the associated dictionary word, and create a word vector for each word in a batch of words for classification using the NN. Find the best matching dictionary vector for each word vector, and report the classification indicator of the best matching dictionary vector for each word vector in the batch.

[0079] In some embodiments, the dictionary words and the words in the batch are encoded into a matrix of 30 characters by 50 length. In some embodiments, the NN is a convolutional neural network. In some embodiments, each word vector is created without storing the corresponding word in the batch. In some embodiments, each dictionary vector and the corresponding classification indicator are indexed to facilitate similarity search. In some embodiments, the system may also include training the NN, including: calculating a similarity metric for multiple pairs of training words for multiple training words; creating a pair of training vectors for each of the multiple pairs of training words using the NN and the twin of the NN; calculating the cosine similarity between each pair of training vectors; and comparing the similarity metric and the cosine similarity for each pair of training words, and adjusting the NN based on the comparison. In some embodiments, the training words include dictionary words. In some embodiments, the similarity metric is equal to 1 minus the Levenshtein edit distance divided by the length of the longest word in the pair. In some embodiments, the cosine similarity is calculated by dividing the dot product of the pair of training vectors by the product of their Euclidean norms. In some embodiments, each of the one or more best matching dictionary vectors has an associated score, and wherein reporting the classification indicator of the one or more best matching dictionary vectors for each word vector in the batch includes: for each word in the batch, selecting the highest score for each classification indicator; and averaging the selected highest scores for each classification indicator.

[0080] In another representative embodiment, a system for classifying words in a batch of words may include at least one storage device storing instructions for causing at least one processor to train a convolutional neural network (CNN); encode each of a plurality of dictionary words into a matrix form; create a dictionary vector for each of a plurality of encoded dictionary words using the trained CNN; store each dictionary vector and a classification indicator corresponding to the associated dictionary word; encode each word in a batch of words for classification into a matrix form; create a word vector for each encoded word in the batch using the trained CNN; find the best matching dictionary vector for each word vector; and report the classification indicator of the dictionary vector that best matches each word vector in the batch. Training the CNN may include: encoding each of a plurality of training words into a matrix form; calculating a similarity metric for a plurality of pairs of training words for the plurality of training words; creating a pair of training vectors for each pair of encoded training words corresponding to the plurality of pairs of training words using the CNN and a twin of the CNN; calculating a cosine similarity between each pair of training vectors; comparing the similarity metric and the cosine similarity for each pair of training words and adjusting the CNN based on the comparison; and storing the trained CNN;

[0081] In a further representative embodiment, a system for classifying words in a batch of words may include at least one storage device storing instructions for causing at least one processor to: train a neural network (NN); create a dictionary vector for each of a plurality of dictionary words using the trained NN; store each dictionary vector and a classification indicator corresponding to the associated dictionary word; create a word vector for each word in a batch of words for classification using the trained NN; find the best matching dictionary vector for each word vector; and report the classification indicator of the best matching dictionary vector for each word vector in the batch.

[0082] Training the NN may include calculating a similarity metric for multiple pairs of training words for the multiple training words, where the similarity metric is equal to 1 minus the Levenshtein edit distance divided by the length of the longest word in the pair; creating a pair of training vectors for each of the multiple pairs of training words using the NN and the twin of the NN; calculating a cosine similarity between each pair of training vectors; and comparing the similarity metric and the cosine similarity for each pair of training words and adjusting the NN based on the comparison.

[0083] The following examples provide additional embodiments of the present technology.

[0084] Example

[0085] 1. A system for classifying words in a batch of words, comprising:

[0086] at least one storage device storing instructions for causing at least one processor to:

[0087] creating a dictionary vector for each of a plurality of dictionary words using a neural network (NN);

[0088] storing each dictionary vector and a category indicator corresponding to an associated dictionary word;

[0089] Create word vectors for each word in a batch of words for classification using the NN;

[0090] For each word vector, find one or more best matching dictionary vectors; and

[0091] The classification indicator of the one or more best matching dictionary vectors is reported for each word vector in the batch.

[0092] 2. A system as described in Example 1, wherein the dictionary words and the words in the batch are encoded as a matrix of 30 characters × 50 length.

[0093] 3. A system as described in Example 1 or 2, wherein the NN is a convolutional neural network.

[0094] 4. A system as described in any of Examples 1 to 3, wherein each word vector is created without storing the corresponding word in the batch.

[0095] 5. A system as described in any of Examples 1 to 4, wherein each dictionary vector and corresponding category indicator is indexed to facilitate similarity searching.

[0096] 6. The system of any of Examples 1 to 5, wherein each of the one or more best matching dictionary vectors has an associated score, and wherein reporting the classification indicator of the one or more best matching dictionary vectors for each word vector in the batch comprises:

[0097] For each word in the batch, select the highest score for each category indicator; and

[0098] The selected highest scores for each category indicator are averaged.

[0099] 7. The system of any of Examples 1 to 6, further comprising training the NN, comprising:

[0100] calculating similarity measures for multiple pairs of training words for multiple training words;

[0101] Creating a pair of training vectors for each of the plurality of pairs of training words using the NN and the twin of the NN;

[0102] Calculate the cosine similarity between each pair of training vectors; and

[0103] The similarity measure and the cosine similarity of each pair of training words are compared, and the NN is adjusted based on the comparison.

[0104] 8. The system of Example 7, wherein the training words include the dictionary words.

[0105] 9. A system as described in example 7 or 8, wherein the similarity metric is equal to 1 minus the Levenshtein edit distance divided by the length of the longest word in the pair.

[0106] 10. A system as described in any of Examples 7 to 9, wherein the cosine similarity is calculated by dividing the dot product of the training vectors by the product of their Euclidean norms.

[0107] 11. A system for classifying words in a batch of words, comprising:

[0108] at least one storage device storing instructions for causing at least one processor to:

[0109] Training a convolutional neural network (CNN), including:

[0110] encoding each of a plurality of training words into a matrix form;

[0111] calculating similarity measures for pairs of training words of the plurality of training words;

[0112] Creating a pair of training vectors for each pair of encoded training words corresponding to the plurality of pairs of training words using the CNN and the twin of the CNN;

[0113] Calculate the cosine similarity between each pair of training vectors;

[0114] comparing the similarity metric and the cosine similarity for each pair of training words, and adjusting the CNN based on the comparison; and

[0115] Store the trained CNN;

[0116] encoding each of a plurality of dictionary words into a matrix form;

[0117] creating a dictionary vector for each of the plurality of encoded dictionary words using the trained CNN;

[0118] storing each dictionary vector and a category indicator corresponding to an associated dictionary word;

[0119] Encode each word in a batch of words for classification into a matrix form;

[0120] Using the trained CNN to create word vectors for each encoded word in the batch;

[0121] For each word vector, find one or more best matching dictionary vectors; and

[0122] The classification indicator of the one or more best matching dictionary vectors is reported for each word vector in the batch.

[0123] 12. The system of example 11, wherein the training words include the dictionary words.

[0124] 13. A system as described in example 11 or 12, wherein the similarity metric is equal to 1 minus the Levenshtein edit distance divided by the length of the longest word in the pair.

[0125] 14. A system as described in any of Examples 11 to 13, wherein the cosine similarity is calculated by dividing the dot product of the pair of training vectors by the product of their Euclidean norms.

[0126] 15. A system as described in any of Examples 11 to 14, wherein the training words, the dictionary words, and the words in the batch are encoded into a matrix of 30 characters×50 length.

[0127] 16. A system as described in any of Examples 11 to 15, wherein each word vector is created without storing the corresponding word in the batch.

[0128] 17. A system as described in any of Examples 11 to 16, wherein each dictionary vector and corresponding category indicator is indexed to facilitate similarity searching.

[0129] 18. The system of any of Examples 11 to 17, wherein each of the one or more best matching dictionary vectors has an associated score, and wherein reporting the classification indicator of the one or more best matching dictionary vectors for each word vector in the batch comprises:

[0130] For each word in the batch, select the highest score for each category indicator; and

[0131] The selected highest scores for each category indicator are averaged.

[0132] 19. A system for classifying words in a batch of words, comprising:

[0133] at least one storage device storing instructions for causing at least one processor to:

[0134] Training a neural network (NN), including:

[0135] calculating a similarity metric for a plurality of pairs of training words of a plurality of training words, wherein the similarity metric is equal to 1 minus the Levenshtein edit distance divided by the length of the longest word in the pair; creating a pair of training vectors for each of the plurality of pairs of training words using the NN and the twin of the NN;

[0136] Calculating a cosine similarity between each pair of training vectors; and comparing the similarity measure and the cosine similarity for each pair of training words, and adjusting the NN based on the comparison;

[0137] Creating a dictionary vector for each of a plurality of dictionary words using the trained NN;

[0138] storing each dictionary vector and a category indicator corresponding to an associated dictionary word;

[0139] Using the trained NN, creating a word vector for each word in a batch of words for classification;

[0140] For each word vector, find one or more best matching dictionary vectors; and

[0141] The classification indicator of the one or more best matching dictionary vectors is reported for each word vector in the batch.

[0142] 20. The system of example 19, wherein the cosine similarity is calculated by dividing the dot product of the pair of training vectors by the product of their Euclidean norms.

[0143] 21. The system of example 19 or 20, wherein each of the one or more best matching dictionary vectors has an associated score, and wherein reporting the classification indicator of the one or more best matching dictionary vectors for each word vector in the batch comprises:

[0144] For each word in the batch, select the highest score for each category indicator; and

[0145] The selected highest score for each category indicator is scored.

[0146] 22. A system as described in any of Examples 19 to 21, wherein the training words, the dictionary words, and the words in the batch are encoded into a matrix of length 30 characters x 50 before creating corresponding vectors using the NN.

[0147] 23. A system as described in any of Examples 19 to 22, wherein the NN is a convolutional neural network.

[0148] 24. A system as described in any of Examples 19 to 23, wherein each word vector is created without storing the corresponding word in the batch.

[0149] 25. A system as described in any of Examples 19 to 24, wherein each dictionary vector and corresponding category identifier is indexed to facilitate similarity searching.

[0150] 26. A system as described in any of Examples 19 to 25, wherein the training words include the dictionary words.

[0151] statement

[0152] The above description and accompanying drawings are illustrative and should not be construed as limiting. Many specific details are described to provide a thorough understanding of the present technology. However, in some cases, well-known details are not described to avoid obscuring the description. In addition, various modifications can be made without departing from the scope of the embodiments.

[0153] References in this specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present technology. The appearances of the phrase "in one embodiment" in different places in the specification do not necessarily all refer to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. In addition, various features are described that may be exhibited by some embodiments but not by other embodiments. Similarly, various requirements are described that may be specific to some embodiments but not to other embodiments.

[0154] The terms used in this specification generally have their ordinary meanings in the art, within the context of the present disclosure, and in the specific context in which each term is used. It is understood that the same thing can be narrated in a variety of ways. Therefore, alternative language and synonyms can be used for any one or more terms discussed herein, and whether the terms are elaborated or discussed in this article does not have any special meaning. Synonyms for some terms are provided. The recording of one or more synonyms does not exclude the use of other synonyms. Examples are used anywhere in this specification, including examples of any terms discussed herein, for illustration only, and are not intended to further limit the scope and meaning of this technology or any example terms. Similarly, this technology is not limited to the various embodiments given in this specification. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those of ordinary skill in the field to which this technology relates. In the event of a conflict, this document (including definitions) shall prevail.

Claims

1. A system for classifying words in a batch of words, comprising: at least one storage device storing instructions for causing at least one processor to: Creating a dictionary vector for each of a plurality of dictionary words using a neural network NN; storing each dictionary vector and a category indicator corresponding to an associated dictionary word; Using the NN to create word vectors for each word in a batch of words for classification; For each word vector, find one or more best matching dictionary vectors; and reporting the classification indicator of the one or more best matching dictionary vectors for each word vector in the batch of words, The system also includes training the NN, including: calculating similarity measures for multiple pairs of training words for multiple training words; Creating a pair of training vectors for each of the plurality of pairs of training words using the NN and the twin of the NN; Calculate the cosine similarity between each pair of training vectors; and The similarity measure and the cosine similarity of each pair of training words are compared, and the NN is adjusted based on the comparison to drive the NN to learn the conversion of training words into vector form so that the cosine similarity of each pair of training words is equal to the similarity measure.

2. The system of claim 1, wherein the dictionary words and the words in the batch of words are encoded as a matrix of 30 characters x 50 length.

3. The system of claim 1, wherein the NN is a convolutional neural network.

4. The system of claim 1, wherein each word vector is created without storing a corresponding word in the batch of words.

5. The system of claim 1, wherein each dictionary vector and corresponding category identifier is indexed to facilitate similarity searching.

6. The system of claim 1 , wherein each of the one or more best matching dictionary vectors has an associated score, and wherein reporting the classification indicator of the one or more best matching dictionary vectors for each word vector in the batch of words comprises: For each word in the batch, select the highest score for each category indicator; as well as The selected highest scores for each category indicator are averaged.

7. The system of claim 1, wherein the training words include the dictionary words.

8. The system of claim 1, wherein the similarity metric is equal to 1 minus the Levenshtein edit distance divided by the length of the longest word in the pair.

9. The system of claim 1, wherein the cosine similarity is calculated by dividing the dot product of the training vectors by the product of their Euclidean norms.

10. A system for classifying words in a batch of words, comprising: at least one storage device storing instructions for causing at least one processor to: Training a convolutional neural network (CNN), including: encoding each of a plurality of training words into a matrix form; calculating similarity measures for pairs of training words of the plurality of training words; Creating a pair of training vectors for each pair of encoded training words corresponding to the plurality of pairs of training words using the CNN and the twin of the CNN; Calculate the cosine similarity between each pair of training vectors; comparing the similarity measure and the cosine similarity of each pair of training words, and adjusting the CNN based on the comparison to drive the CNN to learn the conversion of training words into vector form so that the cosine similarity and the similarity measure of each pair of training words are equal; and Store the trained CNN; encoding each of a plurality of dictionary words into a matrix form; creating a dictionary vector for each of the plurality of encoded dictionary words using the trained CNN; storing each dictionary vector and a category indicator corresponding to an associated dictionary word; Encode each word in a batch of words for classification into a matrix form; Using the trained CNN to create a word vector for each encoded word in the batch of words; For each word vector, find one or more best matching dictionary vectors; and The classification indicator of the one or more best matching dictionary vectors is reported for each word vector in the batch of words. The system of claim 10 , wherein the training words include the dictionary words.

12. The system of claim 10, wherein the similarity metric is equal to 1 minus the Levenshtein edit distance divided by the length of the longest word in the pair.

13. The system of claim 10, wherein the cosine similarity is calculated by dividing the dot product of the pair of training vectors by the product of their Euclidean norms.

14. The system of claim 10, wherein the training words, the dictionary words, and the words in the batch of words are encoded into a matrix of 30 characters x 50 length.

15. The system of claim 10, wherein each word vector is created without storing the corresponding word in the batch of words.

16. The system of claim 10, wherein each dictionary vector and corresponding category identifier is indexed to facilitate similarity searching.

17. The system of claim 10, wherein each of the one or more best matching dictionary vectors has an associated score, and wherein reporting the classification indicator of the one or more best matching dictionary vectors for each word vector in the batch of words comprises: For each word in the batch, select the highest score for each category indicator; as well as The selected highest scores for each category indicator are averaged.

18. A system for classifying words in a batch of words, comprising: at least one storage device storing instructions for causing at least one processor to: Training of neural network NN, including: calculating a similarity metric for multiple pairs of training words for the multiple training words, wherein the similarity metric is equal to 1 minus the Levenshtein edit distance divided by the length of the longest word in the pair; Creating a pair of training vectors for each of the plurality of pairs of training words using the NN and the twin of the NN; Calculate the cosine similarity between each pair of training vectors; and comparing the similarity measure and the cosine similarity of each pair of training words, and adjusting the NN based on the comparison to drive the NN to learn the conversion of training words into vector form so that the cosine similarity and the similarity measure of each pair of training words are equal; Creating a dictionary vector for each of a plurality of dictionary words using the trained NN; storing each dictionary vector and a category indicator corresponding to an associated dictionary word; Using the trained NN, creating a word vector for each word in a batch of words for classification; For each word vector, find one or more best matching dictionary vectors; and The classification indicator of the one or more best matching dictionary vectors is reported for each word vector in the batch of words.

19. The system of claim 18, wherein the cosine similarity is calculated by dividing the dot product of the pair of training vectors by the product of their Euclidean norms.

20. The system of claim 18, wherein each of the one or more best matching dictionary vectors has an associated score, and wherein reporting the classification indicator of the one or more best matching dictionary vectors for each word vector in the batch of words comprises: For each word in the batch, select the highest score for each category indicator; as well as The selected highest score for each category indicator is scored.

21. The system of claim 18, wherein the training words, the dictionary words, and the words in the batch of words are encoded into a matrix of 30 characters x 50 length before creating corresponding vectors using the NN.

22. The system of claim 21, wherein the NN is a convolutional neural network.

23. The system of claim 22, wherein each word vector is created without storing the corresponding word in the batch of words.

24. The system of claim 23, wherein each dictionary vector and corresponding category identifier is indexed to facilitate similarity searches.

25. The system of claim 24, wherein the training words include the dictionary words.

Images