Method and apparatus for decentralized supervised learning in NLP applications

Abstract

To provide a method and a device for decentralized supervised learning in NLP application.SOLUTION: A method for training a neural network as a natural language processing (NLP) model inputs training data to a first architecture part of the neural network. The first architecture part is executed in a plurality of centralized clients. The training data is derived from text data private to the clients. A server does not have an access to the private text data. Further, the method derives, from the training data, weight matrices of numerical weights using the first architecture part and separated from the private text data, connects the weight matrices in a second architecture part of the neural network executed in the server to acquire a single connected weight matrix, and trains the NLP model in the second architecture part by using the connected weight matrix.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] Embodiments relate to methods and apparatus for decentralized teacher learning in NLP applications.

Background Art

[0002] Text mining is a computer-implemented process that uses natural language processing (NLP) to derive high-quality information from text, e.g., examining a collection of large written resources to generate new information. To discover relevant information, NLP techniques are used to transform text into data that can be used for further analysis. In addition to text mining, NLP techniques include named entity recognition, relation extraction, text categorization, and topic extraction.

[0003] To perform supervised learning of a neural network and train it as an NLP model, it is necessary to have access to a large amount of labeled training data, ideally datasets from multiple different sources. However, labeled data is often difficult, expensive, or time-consuming to obtain because they require the labor of experienced human annotators.

[0004] There may be cases where institutions using trained models desire to provide information from their databases for use as labeled training data in supervised learning. However, ensuring data privacy, security, and confidentiality is essential in most enterprises today. Many critical infrastructures in banking, healthcare, insurance, etc. require architectures where data, models, and resources are protected from external individuals and organizations, and thus information is often separated from external access by decentralizing components and modules such that typically no single node has complete system information.

[0005] Therefore, it is desirable to maintain the privacy of data on client computing devices in a decentralized environment while making that data available for natural language processing in supervised learning problems. [Overview of the project]

[0006] One embodiment according to the first aspect can provide a method for training a neural network as a natural language processing (NLP) model, the method comprising the steps of: inputting each set of annotated training data into a plurality of first architectural parts of a neural network, each first architectural part running on each of a plurality of distributed client computing devices communicating with a server computing device, each set of training data being derived from a set of text data that is private to the client computing device on which the first architectural part runs, the server computing device having no access to any of the private text data sets, and all sets of training data sharing a common encoding; using the first architectural part, deriving each weight matrix of numerical weights separated from the private text data sets from the annotated training data sets; in a second architectural part of a neural network running on a server computing device, concatenating the weight matrices received from the client computing devices to obtain a single concatenated weight matrix; and in the second architectural part, training an NLP model using the concatenated weight matrix.

[0007] One embodiment according to a second aspect can provide a device for training a neural network as a natural language processing (NLP) model, the device comprising a plurality of distributed client computing devices, each executing a plurality of first architectural parts of the neural network, each first architectural part receiving a set of annotated training data derived from a set of text data private to the client computing device on which the first architectural part is executed, and the entire set of training data sharing a common encoding, and a server computing device communicating with each of the plurality of client computing devices, the server computing device executing a second architectural part of the neural network, the server computing device having no access to any of the private sets of text data, the first architectural part derives a weight matrix for each numerical weight separated from the set of private text data from the set of annotated training data, the weight matrices received from the client computing devices are concatenated in the second architectural part to obtain a single concatenated weight matrix, and the NLP model is trained in the second architectural part using the concatenated weight matrix.

[0008] Each client computing device can preprocess a private set of text data to derive a set of training data for a common encoding.

[0009] Preprocessing a private set of text data may involve applying a systematization common to all client computing devices to the text data.

[0010] In preprocessing, text data can be mapped to a vector of real numbers using mapping parameters that are common to all client computing devices.

[0011] Preprocessing a private set of text data includes performing lexical organization on the set of text data within each client computing device to ensure that a common lexical organization is provided by the client computing device across all training data, and performing word embedding on each client computing device, where the text data is mapped to vectors of real numbers, using predefined common character-level representations and predefined common configuration parameters.

[0012] The embodiment provides a system architecture that applies decentralized learning across neural networks to handle different NLP tasks based on supervised learning strategies. In this way, it is possible to maintain an independent and distributed method of utilizing data from client computing devices while preserving the privacy and confidentiality of sensitive information.

[0013] The global performance of the trained model can potentially be improved by leveraging data from different client devices during training, while complete protection of sensitive information is provided through the proposed decentralized learning architecture, where each client's data is isolated from the server, model, and other clients. Human resource costs can also be reduced due to the reduced amount of manual annotation required to augment the input corpus.

[0014] The proposed method / system can be applied to any domain in the use cases of text mining applications. Furthermore, embodiments can be adapted to any type of natural language processing technique, such as relation extraction, text categorization, and topic extraction. [Brief explanation of the drawing]

[0015] Next, refer to the attached drawing as an example. [Figure 1]This is a flowchart of a method according to one embodiment. [Figure 2] The apparatus according to the embodiment is shown. [Figure 3] This document shows a system architecture according to one embodiment. [Figure 4] This shows the workflow of the vocabulary systematization process. [Figure 5] An example of a learning architecture according to one embodiment is shown. [Figure 6] This shows the architecture of the model estimation module. [Figure 7] This shows a snapshot of the text from the dataset. [Figure 8] This document describes the design of a neural network according to one embodiment. [Figure 9] This is a block diagram of a computing device suitable for performing a method according to one embodiment. [Modes for carrying out the invention]

[0016] This document describes an embodiment that solves the problem of generating high-quality centralized models in supervised learning problems for NLP applications, such as text mining, while maintaining confidentiality and privacy because the data does not need to be shared by the client with the server or other clients. The use of a decentralized architecture ensures that the data protection, privacy, and confidentiality of the originating system are maintained. That is, the model is trained on data derived from information in the originating system, but is completely invisible to the information in the originating system.

[0017] We propose a decentralized learning architecture that provides a distributed learning system where the neural network layer is separated between the client and the central server, and the data preprocessing and initial training layers are deployed on the client computing device (e.g., the customer's server). Subsequent learning steps are performed within the server once access to the original data is no longer required to continue the learning process. Although not discussed below, it may also be possible to apply an encryption algorithm initially to add an additional security layer to the architecture, further enhancing data anonymity and protection.

[0018] The flowchart in Figure 1 illustrates a method for training a neural network as an NLP model according to one embodiment. In this embodiment, the method includes, in step A, inputting each set of annotated training data into a plurality of first architectural parts A1 of the neural network, the first architectural parts A1 being executed only on each of the n distributed client computing devices 10 (client 1, client 2, ..., client n) communicating with a server computing device 20. Each set of training data is derived from a set of text data that is private to the client computing device 10 on which the first architectural part A1 is executed, i.e., the server computing device 20 has no access to any of the private text datasets. The entire set of training data shares a common encoding. The method further includes, in step B, using the first architectural part A1 to derive each weight matrix of numerical weights separated from the private text datasets from the set of annotated training data. Next, the method includes, in step C, concatenating the weight matrices received from the client computing device 10 in a second architectural part A2 of the neural network that is executed only on the server computing device 20 to obtain a single concatenated weight matrix, and in step D, training an NLP model in the second architectural part A2 using the concatenated weight matrix.

[0019] The system shown in FIG. 2 shows an apparatus for training a neural network as an NLP model according to the embodiment of FIG. 1. In this embodiment, the apparatus includes a plurality of n decentralized client computing devices 10 (Client 1, Client 2,..., Client n), sometimes hereinafter referred to as "clients", and a server computing device 20, sometimes hereinafter referred to as a server or a central server, which communicates with each of the plurality of client computing devices 10. The client computing device 10 is configured to execute a plurality of first architecture parts A1 of the neural network, and each first architecture part A1 receives a set of annotated training data derived from a set of text data that is private to the client computing device 10 on which the first architecture part A1 is executed, and the entire set of training data shares a common encoding. The server computing device 20 is configured to execute a second architecture part A2 of the neural network, and the server computing device 20 has no access to any private text data set. The first architecture part A1 derives each weight matrix of numerical weights separated from the private text data set from the set of annotated training data, and the weight matrices received from the client computing device 10 are concatenated in the second architecture part A2 to obtain a single concatenated weight matrix, and the NLP model is trained in the second architecture part A2 using the concatenated weight matrix.

[0020] Next, a detailed description will be given of an embodiment applied to the training of an NLP model such as a text mining model.

[0021] Figure 3 shows the system architecture according to an embodiment and two stages of training an NLP model. This architecture consists of n clients 10, client 1, client 2, ... client n, each client having similar stages and modules for performing a part of the first "codification" stage (stage 1) and the second "learning" stage (stage 2) of this method, and a central server 20 that performs the remaining part of the second stage in a common module, representing the overall system. In stage 1 - codification, a specific encoding of the input data is prepared to establish a common codification for all clients. In stage 2 - learning, a neural network architecture is constructed and mathematical operations are performed to update the weights of the matrices in the learning process. In this way, a global and unique NLP model such as a text mining model is created, and information from different data sources can be utilized in a decentralized manner while ensuring data retention and confidentiality and increasing the performance of the system.

[0022] As shown on the left side of Figure 3, the first input to the process is a dataset with natural language text samples in each of the n clients. The output of the learning process is an NLP model, e.g., a text mining model, which can be used later to perform estimations and classifications on new text samples at each client, as shown on the right side (model estimation) of the figure.

[0023] Stage 1 - Codification Stage 1 prepares a specific encoding of the input data to establish a common codification used by all of the n clients for training. By maintaining a common vocabulary codification within the training data, the consistency of the learning process can be maintained during training.

[0024] This stage is performed on each client computing device and consists of two components or modules: a vocabulary organization module and a word embeddings module.

[0025] Vocabulary systemization The purpose of the vocabulary systematization module is to ensure a common vocabulary systematization across all involved clients. The input is one or more text datasets from the relevant clients. Figure 4 shows the workflow of the vocabulary systematization process for one client, for ease of understanding. This module runs essentially within the client, except for optical communication with a central server that may be required to share general (non-private) information.

[0026] Referring to Figure 4, the vocabulary systematization procedure includes the following process. 1. Remove the label All text datasets are already annotated with labels. For example, in Named Entity Recognition (NER), labels refer to the type of entity (e.g., person, place, disease, organization, etc.), or in text classification, labels refer to the type of content text (e.g., sports, finance, science, etc.). A shared label architecture is maintained among clients, and labels are common enough that they can be openly shared among clients without concerns about data leakage.

[0027] Therefore, in step 1, each client analyzes its dataset to extract all distinguishable labels included in the annotations and sends a list of those labels to the server (i.e., there is client-server communication to share this information). Thus, the server collects the list of labels from each client, which becomes input for the next step.

[0028] 2. Check existing labelsIn step 2, the list of labels from each client is compared against the information stored in the "label dictionary" maintained by the server. If a label in the list extracted by the client already exists in the label dictionary, the process proceeds to the next step, step 3. Otherwise, a further step, step 2A, is performed to include the new label in the label dictionary. A unique central shared list of labels is then provided to maintain consistency in the next step where the labels are tagged.

[0029] 3. Tagging labels In step 3, labels are tagged with numerical representations ranging from 1 to N (N = total number of labels) to ensure consistency across all clients. At this point, the vectorized label representations (vectors of labels), which are the parameters used for training, are obtained.

[0030] 4. Specify the character representation. In step 4, a common (alphanumeric) character-level representation is established for the vocabulary, which will be followed later. The central server determines the characters included in this representation (depending on the language of the dataset), and this character-level representation is sent to each client (i.e., there is server-client communication to share this information). Each client must adhere to the specified character-level representation in order to maintain a common encoding among all clients and to maintain consistency in subsequent steps.

[0031] Word embedding The word embedding module prepares and encapsulates the final word-level representations of the client's dataset for subsequent training. This process is performed on the client side, according to the character-level representation settings shared by the central server in step 4 of the lexical structuring procedure. Numerical representations of text samples are required for the use of the client's dataset in the training process. The method for converting text representations to numerical representations is as follows: · Tagging text-level expressionsWhen the character-level representation (char-level) is sent by the server, each character is mapped to a numerical representation ranging from 1 to m (where m is the total number of characters). · Create a character sequence Initially, the central server specifies two common configuration parameters and sends this information to each client. These parameters are "maximum_length_of_sentence" and "maximum_length_of_word". Both parameters must be fixed and unique for all clients. Next, using the specified parameters, each client divides the sample into sentences, and each sentence into tokens. In each sentence, all of its tokens are converted to a numerical representation shown by the server in character-level representation. Every word in each sentence follows this character-level representation, fixed to the two specified parameters. A word cannot be longer than maximum_length_of_word, and a sentence cannot be longer than maximum_length_of_sentence. Words and / or sentences shorter than the maximum length are padded with zeros until the maximum length is reached. This is to maintain symmetric dimensionality of the final generator matrix as needed for training.

[0032] For example, in the sentence "The global financial crisis has left Iceland's economy in shambles," the embeddings for each word with "maximum_length_of_word" = 20 are as follows:

number

[0033] The final output of the word embedding module is a three-dimensional matrix (word embedding matrix) that represents the input sample sentence word by word according to its character-level representation, for example, as follows:

number

[0034] Therefore, the output of stage 1 is the word embedding matrix and label vector obtained for each client. This output is used as input to stage 2.

[0035] In NER use cases, each label in the vector corresponds to each word in the matrix. For example, in the previous example sentence, the vector of labels for that sentence in NER use cases is as follows:

number

[0036] Stage 2 - Learning Stage 2 involves building the neural network architecture and performing mathematical operations to update the matrix weights in the learning process. These weight values ​​are updated at each learning step by a formula that approximates the estimated output with the actual value (including label annotations). The estimated output is compared with the actual value, the deviation is calculated, and the error is propagated backward to the previous layer to correct and adjust the weights.

[0037] The input to Stage 2 comes from Stage 1. As mentioned above, the main inputs required are the word embedding matrix and the associated label vector. In addition, other configuration parameters are required, such as "maximum_length_of_sentence" and "maximum_length_of_word," which are also used in the previous stage, or other things required for the neural network configuration, such as the number of neurons in each layer, dropout value, activation type, optimizer, or loss equation.

[0038] Figure 5 shows an example of a learning architecture for a neural network design, where there are two clients, for example, client 1 and client 2.

[0039] This architecture consists of two parts: the first part which runs on the client side and the second part which runs on the server side.

[0040] The first part of each client consists of the following components: - An embedding layer encapsulated in a time-distributed layer [in the example above, dimensions (number of characters in vocab × maximum_length_of_word) = (118 × 20)]. The input is a (? × 100 × 20) reference to a word embedding matrix transformed from a text dataset (number of samples × maximum_length_of_sentence × maximum_length_of_word). The purpose of the embedding layer is to capture the relationships between words by mapping words with similar meanings to similar numerical vector representations (word embeddings). The vector values ​​are learned and updated as the neural network is trained. The time-distributed layer updates the values ​​in the matrix it applies during training and applies the same operation to each occurrence of a word in the text (effectively treating each word as a time series). This results in better definitions of different meanings of words and eliminates ambiguity in word relationships. The output of the embedding layer is a matrix that numerically represents the relationships between words in the input samples. Each row of the output matrix contains a vector value of each word compared to the rest of the word. - A long short-term memory (LSTM) layer with several configuration parameters encapsulated in a time-distributed layer. The LSTM layer contains a memory module that remembers the previous step in the training process and determines whether the weight updates improve the results from the past, deciding whether to accept or discard the updates. A matrix representing the input samples is input, followed by word embeddings from the previous layer, and updates are made based on annotations of the input samples (entity annotations in the case of NER, and possibly binary classifications of "0" or "1" in the case of relation extraction). The matrix updates depend on these annotations and the properties of the subsequent LSTM layer. Again, the time-distributed layer treats the input samples as time-series data and applies the same operations to each input.

[0041] Once the process is executed, the weights updated by the LSTM layer are shared with the server, and the NLP model continues to be trained along with all the information from different clients. At this point, the shared matrix contains only numerical weights that are separated from the text in the original dataset, and there is no way to reconstruct and retrieve the original text dataset, thereby guaranteeing the privacy of the information.

[0042] Next, on the server side, the architectural components are as follows: - A concatenate layer (or merged layer) for concatenating weight matrices from different clients. The weight matrices from client 1 to client n are sequentially concatenated into a single matrix to begin learning in a common way. - A dropout layer on the connected matrix to avoid overfitting the model during training. - Bidirectional LSTM (BiLSTM) layer (A BiLSTM layer consists of two LSTM layers, one taking input in the forward direction and the other in the reverse direction. The BiLSTM layer learns the bidirectional long-term dependencies between steps in the sequence data, effectively increasing the amount of information available to the network and thereby improving the context available to the algorithm (for example, knowing which words directly follow and precede a given word in a sentence)). - The final dense layer encapsulated in a time-distributed layer. The dense layer is divided into parts according to the number of clients, i.e., one part per client (in the illustrated example, there are two clients, so there are two parts). When each client constructs an embedding with the same specifications, the input shape is the same for each client, and similarly, the output shape is the same for each client. The server creates a layer with the correct shape, and therefore the BiLSTM layer outputs the corresponding shape. Furthermore, in the concatenated layer, the weight matrices from different clients are concatenated in a continuous manner, so the output of the BiLSTM layer can be appropriately divided for input to each part of the dense layer. Thus, in this example, the final output of the dense layer is the trained model, but it has two matrices (output_client1 and output_client2) corresponding to the final weight matrix that better fits each respective client.

[0043] Furthermore, an intermediate communication channel must be provided to send backpropagation errors from the server to the client to adjust the weights within the first layer, and similarly, to share the updated weights in the forward direction from the client to the server.

[0044] In this example, the final output is a unique text mining model, but depending on the task required, it could be an NER model, a text classification model, a topic categorization model, or something else.

[0045] The following describes the components that support the overall system activity.

[0046] Model estimation Once a "text mining model" is created, it can be used to perform new estimations, classifications, or categorizations. Figure 6 shows this architectural part of the model estimation module in one client (Client 1).

[0047] The Model Estimation module takes a "text mining model" and a new text paragraph as input. The "Run Model" component receives these inputs and proceeds through the following steps. 1. Preprocess the new text paragraphs to clean and curate the text: Among other cutting-edge approaches, this may include actions such as punctuation removal and word normalization (singular form, lemmatization, stemming, etc.).

[0048] 2. The new text is divided into sentences, and each sentence is divided into tokens. Next, the sentences and tokens are mapped and converted to character lexical representations, as described in step 1. Thus, at this point, a word embedding matrix of the input text is obtained.

[0049] 3. The "text mining model" is run using a word embedding matrix of the new text as input to the model. This run outputs a prediction result for the new text. The results vary depending on the text mining task. For example, in an NER task, the result is an entity type label for each word in the text (e.g., person, organization, disease, place, etc.); in text classification, the result is the theme of the entire text (e.g., sports, finance, health, etc.); and in topic categorization, the result is a list of topics about the text (e.g., [AI, NLP, machine learning] or [security, privacy, confidentiality], etc.).

[0050] Examples The following examples are based on experiments conducted to simulate a decentralized NER model (DeNER) using datasets from two independent sources (i.e., two simulated clients). Known public datasets were used for the experiments, where annotated named entities are, among other things, general domains such as place, person, organization, or time.

[0051] The workflow for this example is as follows:

[0052] input: A known public dataset of named entity annotations, split into two documents to simulate two independent clients. The annotated entities are as follows: • Geo = Geographical Entity · org=Organization · per=Person • gpe = Geopolitical Entity • tim = Time indicator • art = artifact eve = Event · nat=Natural Phenomenon

[0053] The dataset format follows the IOB naming convention for tagging tokens, which is as follows: • I: A prefix before a tag indicates that the tag is inside the chunk. • B: A prefix before a tag indicates that the tag is the beginning of a chunk. • O: This tag indicates that the token does not belong to a chunk (outside).

[0054] Figure 7 shows a snapshot of the data.

[0055] There are 17 tags, which are shared between the two datasets (= the two clients).

[0056] procedure: Stage 1 - Systematization: 1. Vocabulary systemization To proceed with this step, communication must be established between the central server and each client to request a list of labels. Initially, the server iteratively requests labels from client 1. In this example, since this is the first request to any of the clients, all of client 1's labels are stored. For example, if client 1 has the labels B-geo, I-geo, B-gpe, I-gpe, B-per, I-per, B-eve, I-eve, and O in its list, all of these labels will be included in the central server's system, within the system's "label dictionary". Next, the server requests its labels from client 2. For example, if client 2 has the labels B-geo, I-geo, B-nat, I-nat, B-per, I-per, B-tim, I-tim, B-art, I-art, B-org, I-org, and O in its list, in the check of labels from client 2, B-geo, I-geo, B-per, and I-per are discarded because they have already been provided by client 1. The remaining labels in client 2's list are added to the central server's "label dictionary," resulting in a unique centrally shared list of labels.

[0057] After collecting a centrally shared list of labels, this list is vectorized, as described above, to convert it into a numerical representation, so that the resulting vectorized label representation (a vector of labels) can later be used by a neural network. In particular, the list of labels is enumerated from 1 to 17 (17 = the total number of labels) to obtain the following representation: [Table 1]

[0058] Finally, in this step, a common character-level representation of the vocabulary that all clients should follow is specified. For example, the alphanumeric character-level representation to follow, covering the entire alphabet of the dataset in this experiment, may be as follows:

number

[0059] This character-level representation is sent to all clients, and each client must adhere to this representation in order to maintain a common encoding among all clients.

[0060] 2. Word Embedding This module prepares and encapsulates the final word-level representation of the client's dataset for subsequent training. This action is performed on the client side according to the character-level representation settings shared by the central server in the previous step.

[0061] In each client, each character of the vocabulary is encoded according to a numerical representation ranging from 1 to 117 (117 = the total number of characters), just as was done previously for the label tags. Additionally, an extra representation "UNK" is added at position 0 to handle potential new unknown characters or errors, hence a total of 118.

[0062] Next, a character sequence is created. The central server specifies the parameters "maximum_length_of_sentence=100" and "maximum_length_of_word=20" and sends them to each client. Each client divides the sample into sentences, and each sentence into tokens. Across each sentence, all of its tokens are converted into numerical representations shown by the server in character-level representations. Every word in each sentence follows this character-level representation, fixed to the two parameters mentioned above.

[0063] For example, the embeddings for each word with "maximum_length_of_word" = 20 in the sentence "The global financial crisis left Iceland's economy in chaos" are as follows:

number

[0064] The final output of word embedding is a three-dimensional matrix (word embedding matrix) that represents the input sample sentence word by word according to its character-level representation.

number

[0065] This word embedding matrix and label vector are the output for this stage for each client and will be used as input in the next step. In this use case, each label in the vector has a correspondence with each word in the matrix. For example, the label vector for the previous example sentence is as follows:

number

[0066] Stage 2 - Learning: Figure 5 shows the specific settings used in this experiment, which are as follows: - In the example, the dimension (number of characters in the vocab × maximum_length_of_word) = (118 × 20). This is encapsulated in a time-distributed layer. - Input of (? × 100 × 20) references to a word embedding matrix converted from a text dataset (number of samples × maximum_length_of_sentence × maximum_length_of_word). - The sample size for this experiment will be 23,979 per client.

[0067] The design of the neural network with specific values ​​for this experiment is shown in Figure 8.

[0068] Training was performed using an early stopping callback that required training to be terminated if the validation loss ("val_loss") was less than 0.0557. In this proposal, 80 epochs were required to reach a "val_loss" of 0.0554. The final snapshot of the process is as follows:

number

[0069] [Target / Impact - Quantitative improvement in performance (increase in F-score)] Evaluation: Comparison of results When this NER use case was trained separately for each client using conventional methods, and a separate model was built for each client, the following F-scores were obtained. [Table 2]

[0070] However, when training using the decentralized supervised learning approach (DeNER) proposed for this NER, the following f-scores were obtained. [Table 3]

[0071] Therefore, the results obtained using the proposed DeNER model can be seen to be more than 10 points better than the results of each dedicated model trained on each client. The proposed DeNER model combines named entity recognition from different clients into a single solution, allowing one client to learn from the knowledge of another client without sharing any text data, thus maintaining the privacy of sensitive information. Although the DeNER model requires more epochs to train to reach a specified validation loss because it combines more information, its performance is far better than that of local models reaching the same validation loss.

[0072] Figure 9 is a block diagram of a computing device, such as a data storage server, which can be used as a server computing device 20 or a client computing device 10 to embody the present invention, perform some of the operations of a method for embodying the present invention, and perform some of the tasks of the apparatus of one embodiment. For example, the computing device of Figure 9 may be used to perform all the tasks of the server computing device 20 shown in Figures 2 and 3, or all the tasks of one of the client computing devices 10 shown in Figures 2, 3, and 6, and to perform one or more of the processes described with reference to Figures 1, 3, 4, and 5.

[0073] The computing device includes a processor 993 and memory 994, which may be configured to perform tasks of, for example, a first architectural portion A1 or a second architectural portion A2 of a neural network. The computing device further includes a network interface 997 for communication with other computing devices, for example, with at least one other computing device of an embodiment of the present invention.

[0074] For example, one embodiment may consist of a network of such computing devices. Optionally, the computing devices further include one or more input mechanisms, such as a keyboard and a mouse 996, and one or more display units, such as a monitor 995. The components are connectable to each other via a bus 992.

[0075] Memory 994 may include computer-readable media, and the term may refer to a single or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) configured to carry computer-executable instructions or to store data structures such as a first architectural part A1 or a second architectural part A2 of a neural network. Computer-executable instructions may be accessible by, for example, a general-purpose computer, a dedicated computer, or a dedicated processing device (e.g., one or more processors), and may include instructions and data that cause these to perform one or more functions or operations. For example, computer-executable instructions may include instructions to perform all or part of a task or function or process executed by a client computing device 10 or a server computing device 20, as described with reference to Figures 1, 2, 3, 4, 5, or 6. Such instructions may also be executed by one or more processors 993. Thus, the term “computer-readable storage medium” may include any medium that can store, encode, or carry instruction sets for machine execution and cause a machine to perform one or more of the methods of this disclosure. Therefore, the term “computer-readable storage medium” may be considered to include, but is not limited to, solid-state memory, optical media, and magnetic media. For example, such computer-readable storage medium may include non-temporary computer-readable storage mediums, including random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk read-only memory (CD-ROM) or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, and flash memory devices (e.g., solid-state memory devices).

[0076] The processor 993 is configured to control the computing device and perform processing operations, executing computer program code stored in memory 994 to perform, for example, the methods described with reference to Figures 1, 3, 4, 5, and / or 6 and 19, and the methods defined in the claims. Memory 994 stores data to be read and written by the processor 993. When referred to herein, the processor may include one or more general-purpose processing devices such as a microprocessor, a central processing unit, etc. The processor may include a composite instruction set compute (CISC) microprocessor, a reduced instruction set compute (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor that implements other instruction sets or combinations of instruction sets. Furthermore, the processor may include one or more dedicated processing devices such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), a network processor, etc. In one or more embodiments, the processor is configured to execute instructions and operations for performing the operations discussed herein.

[0077] The display unit 995 can display a representation of data stored by the computing device, and can also display a cursor, dialog boxes, and screens that enable interaction between the user and programs and data stored in the computing device. The input mechanism 996 may enable the user to input data and commands into the computing device.

[0078] The network interface (network I / F) 997 can connect to a network such as the Internet and can connect to other such computing devices via the network. The network I / F 997 can control the data that is input and output to and from other devices via the network.

[0079] Other peripherals such as microphones, speakers, printers, power supply units, fans, cases, scanners, and tracker balls may be included in the computing device.

[0080] A method for embodying the present invention can be performed on a computing device such as the one shown in Figure 9. Such a computing device does not need to have all the components shown in Figure 9, but may consist of a subset of these components. A method for embodying the present invention can be performed by a single computing device that communicates with one or more data storage servers over a network. The computing device may be a data storage device itself that stores at least a portion of the data.

[0081] The method for realizing the present invention may be performed by a plurality of computing devices working together. One or more of the plurality of computing devices may be data storage servers that store at least a portion of the data.

[0082] The above-described embodiments of the present invention may be used, advantageously, independently of any other embodiment, or in any feasible combination with one or more other embodiments.

[0083] The following list of definitions explains the meaning of several technical terms in the context of the proposed invention. • Machine Learning: A subfield of computer science that "gives computers the ability to learn without being explicitly programmed." It explores the study and construction of algorithms that can learn from data and make predictions about that data. • Deep learning: In contrast to task-specific algorithms, it is part of a broader family of machine learning methods based on training data representations. Artificial neural networks (ANNs / NNs) are an information processing paradigm inspired by how biological nervous systems, such as the brain, process information. They can be used to extract patterns and detect trends that are too complex to be noticed by humans or other computer methods. • Named Entity Recognition (NER): The Named Entity Recognition (NER) approach identifies and extracts references to entities in unstructured text. These entities are units such as nouns (including people, organizations, and places) and numerical representations (including time, dates, money, and percentage representations). • Decentralized systems: Systems in which lower-level components act on local information to achieve global goals. This form of control is known as distributed control, or control in which each component of the system is equally responsible for contributing to global and complex behavior by acting on local information in an appropriate manner. • Distributed architecture: An environment in which its components are presented on different platforms, and several components can cooperate with each other through a communication network to achieve a specific purpose or goal. • f-score: A traditional measure of a model's accuracy on a given dataset.

[0084] The following note is left regarding the above embodiment. (Note 1) A method for training a neural network as a natural language processing (NLP) model, The steps include inputting each set of annotated training data into a plurality of first architectural parts of the neural network, wherein the first architectural parts are executed on each of the client computing devices of a plurality of distributed client computing devices communicating with a server computing device, each set of training data is derived from a set of text data that is private to the client computing device on which the first architectural parts are executed, the server computing device has no access to any of the private sets of text data, and all sets of training data share a common encoding. The steps include: using the first architectural part, deriving the respective weight matrices of the numerical weights separated from the private text data set from the aforementioned annotated training data set; In the second architectural portion of the neural network executed on the server computing device, the steps include concatenating the weight matrices received from the client computing device to obtain a single concatenated weight matrix, In the second architectural part, the steps include training the NLP model using the connected weight matrix, A method that includes this. (Note 2) The method according to Appendix 1, further comprising the step of preprocessing a private set of text data on each client computing device to derive a set of training data for the common encoding. (Note 3) The method according to Appendix 2, wherein in the preprocessing described above, a systematization common to all client computing devices is applied to the text data. (Note 4) The method according to Appendix 2 or 3, wherein, in the preprocessing described above, the text data is mapped to a vector of real numbers using mapping parameters common to all client computing devices. (Note 5) The aforementioned pretreatment is Performing lexical organization processing on the set of text data within each client computing device to ensure that a common lexical organization is provided by the client computing device among all the training data, Using a predefined common character-level representation and predefined common configuration parameters, each client computing device performs a word embedding process in which the text data is mapped to a vector of real numbers. The method described in Appendix 2, including the method described in Appendix 2. (Note 6) A device for training a neural network as a natural language processing (NLP) model, A plurality of distributed client computing devices, each executing a plurality of first architectural parts of the neural network, wherein each first architectural part receives a set of annotated training data derived from a set of text data that is private to the client computing device on which the first architectural part is executed, and the entire set of training data shares a common encoding. A server computing device that communicates with each of the aforementioned plurality of client computing devices, the server computing device executes the second architectural portion of the neural network, and the server computing device does not have access to any of the sets of private text data, including The first architectural part derives the respective weight matrices of the numerical weights separated from the private text data set from the annotated training data set, The weight matrix received from the client computing device is concatenated in the second architectural part to obtain a single concatenated weight matrix, and the NLP model is trained in the second architectural part using the concatenated weight matrix. (Note 7) Each client computing device is the apparatus described in Appendix 6, which preprocesses a private set of text data to derive a set of training data for the common encoding. (Note 8) The apparatus described in Appendix 7, wherein preprocessing the private set of text data includes applying a systematization common to all client computing devices to the text data. (Note 9) The apparatus according to Appendix 7 or 8, wherein preprocessing the private set of text data includes mapping the text data to a vector of real numbers using mapping parameters common to all of the client computing devices. (Note 10) Preprocessing the private set of text data mentioned above is Performing lexical organization processing on the set of text data within each client computing device to ensure that a common lexical organization is provided by the client computing device among all the training data, Using a predefined common character-level representation and predefined common configuration parameters, each client computing device performs a word embedding process in which the text data is mapped to a vector of real numbers. The apparatus described in Appendix 7, including the apparatus described in Appendix 7.

Claims

1. A method for training a neural network as a natural language processing (NLP) model, The steps include inputting each set of annotated training data into a plurality of first architectural parts of the neural network, wherein each first architectural part runs on each of a plurality of distributed client computing devices communicating with a server computing device, each set of training data is derived from a set of text data that is private to the client computing device on which the first architectural part runs, the server computing device has no access to any of the private sets of text data, and all sets of training data share a common encoding. The steps include: using the first architectural part, deriving the respective weight matrices of the numerical weights separated from the private text data set from the aforementioned annotated training data set; In the second architectural portion of the neural network executed on the server computing device, the steps include concatenating the weight matrices received from the client computing device to obtain a single concatenated weight matrix, In the second architectural part, the steps include training the NLP model using the connected weight matrix, Includes, Each set of the training data in the common encoding is: Performing a vocabulary systematization process on the private text data set within each client computing device to ensure that a common vocabulary systematization is provided by the client computing device across the entire set of training data, wherein the vocabulary systematization process establishes a common alphanumeric character-level representation of the vocabulary, using characters determined by the server computing device. Using the aforementioned common alphanumeric character-level representation and predefined common configuration parameters, each client computing device performs word embedding processing in which the text data is mapped to a vector of real numbers. A method derived by preprocessing the set of private text data on each client computing device.

2. A device for training a neural network as a natural language processing (NLP) model, A plurality of distributed client computing devices, each executing a plurality of first architectural parts of the neural network, wherein each first architectural part receives a set of annotated training data derived from a set of text data that is private to the client computing device on which the first architectural part is executed, and the entire set of training data shares a common encoding. A server computing device that communicates with each client computing device, the server computing device executes the second architectural portion of the neural network, and the server computing device does not have access to any of the private text data sets, including The first architectural part derives the respective weight matrices of the numerical weights separated from the set of private text data from the set of annotated training data, The weight matrices received from the client computing device are concatenated in the second architectural part to obtain a single concatenated weight matrix, and the NLP model is trained in the second architectural part using the concatenated weight matrix. Each client computing device Performing a vocabulary systematization process on the private text data set within each client computing device to ensure that a common vocabulary systematization is provided by the client computing device across the entire set of training data, wherein the vocabulary systematization process establishes a common alphanumeric character-level representation of the vocabulary, using characters determined by the server computing device. Using the aforementioned common alphanumeric character-level representation and predefined common configuration parameters, each client computing device performs word embedding processing in which the text data is mapped to a vector of real numbers. An apparatus that preprocesses the set of private text data to derive each set of the training data in the common encoding.

Images