Information processing device

Abstract

[Problem] To train a generation model by properly using tokenization rules. [Solution] This information processing device trains a generation model on the basis of a token string obtained by tokenizing data included in a dataset. The information processing device comprises one or a plurality of memories, and one or a plurality of processors. In a case of using a token string obtained from second data to train the generation model, the one or plurality of processors tokenize the second data in accordance with a tokenization rule that is different from a tokenization rule for obtaining a token string used for training the generation model from first data.

Description

Information processing device

[0001] The present disclosure relates to an information processing device.

[0002] Large-scale language models are used in various fields. For example, large-scale language models are trained using token sequences obtained by tokenizing each sentence included in a training dataset. In conventional training methods, the same tokenization rules are applied to each sentence.

[0003] S. Borgeaud, et.al., “Improving language models by retrieving from trillions of tokens,” arXiv:2112.04426v3, https: / / arxiv.org / pdf / 2112.04426v3.pdf, 7 Feb 2022. Y. Wata et al., “Membership inference attacks using sampling against large-scale language models,” Proceedings of the 30th Annual Conference of the Association for Natural Language Processing (NLP2024), A11-3, pp. 3015-3020, March 2024.

[0004] One of the non-limiting problems that the embodiments of the present disclosure aim to solve is training a generative model by using different tokenization rules.

[0005] According to one embodiment, an information processing device trains a generative model based on a token sequence obtained by tokenizing data included in a dataset. The information processing device includes one or more memories and one or more processors. When a token sequence obtained from second data is used to train the generative model, the one or more processors tokenize the second data according to a tokenization rule that differs from the tokenization rule used to obtain the token sequence used to train the generative model from first data.

[0006] A block diagram schematically showing an example of an information processing apparatus according to an embodiment. A flowchart showing an example of processing by the information processing apparatus according to an embodiment. A block diagram schematically showing an example of an information processing apparatus according to an embodiment. A block diagram schematically showing an example of hardware implementation of the information processing apparatus according to an embodiment.

[0007] The problems to be solved by the embodiments of the present disclosure are not limited to the problems described above, and as further examples of some problems that are not limited to these, problems corresponding to the effects described in the embodiments can also be considered. In other words, a problem corresponding to at least one of the effects described in the description of the embodiments of the present disclosure can be considered to be a problem to be solved by the present disclosure.

[0008] The problem to be solved by the present disclosure can be, for example, to train a language model by recognizing not only the beginning of a sentence but also meta-information of the sentence, such as the content and type of the sentence, for sentences included in a training dataset. The problem to be solved by the present disclosure can also be, for example, to make it difficult to reproduce a training dataset to which predetermined meta-information has been added from a language model trained in this way. The problem to be solved by the present disclosure can also be to make it easier to answer sentences influenced by a training dataset to which predetermined meta-information has been added from a language model trained in this way. Of course, the problem to be solved by the present disclosure is not limited to these, and can also be a problem that can be solved within the scope of the present disclosure. Note that a language model is an example of a generative model that receives a token sequence, for example, and generates data based on the token sequence, and each embodiment of the present disclosure may be applied to such a generative model. The generative model may be a Transformer-based machine learning model. The generative model may be a base model capable of fine-tuning, etc.

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described with reference to the drawings. The drawings and the description of the embodiments are given by way of example only and are not intended to limit the present invention.

[0010] Although examples of tokenization are shown for several sentences, they are shown as non-limiting examples and will not necessarily be tokenized as shown. In addition, although the explanation uses Japanese sentences, the method is of course applicable to other languages, and can also be applied to text that follows a predetermined format or text that has a predetermined regularity, rather than sentences.

[0011] That is, the scope of "text" applicable in the present disclosure can include texts used professionally as well as commonly used texts. Texts used professionally can include, for example, literature, academia, art, and / or music. Some non-limiting examples of texts used professionally include classical literature, academic papers, descriptions showing the arrangement of atoms, molecules, etc. used in the field of chemistry, descriptions of genes, etc. used in the field of biology, descriptions of source code, intermediate code, assembly, etc. used in the field of information technology, musical scores, lyrics, etc.

[0012] In the present disclosure, the phrase "to assign a predetermined token" may mean either assigning (inserting) a predetermined token while tokenizing data, or assigning a predetermined token to a token string obtained by tokenizing data. In other words, "to assign a predetermined token" may mean obtaining a token string obtained by tokenizing with the predetermined token attached.

[0013] A predetermined token added to the beginning of a token sequence corresponding to data may be called a start token, and as will be described below, there may be multiple types of start tokens. For example, when describing "adding a start token," this may mean either following the start token with tokens obtained sequentially while tokenizing data, or adding a start token to the beginning of a token sequence obtained by tokenizing data. Furthermore, adding different start tokens may mean adding different start tokens using different tokenizers.

[0014] (First embodiment)

[0015] 1 is a block diagram schematically illustrating an example of an information processing device according to an embodiment. The information processing device 1 includes at least a processing circuit 10 and a storage unit 12. In addition to these, the information processing device 1 may include other components necessary for driving the information processing device 1, such as a control unit for controlling the entire information processing device 1, a power supply unit for supplying power to each component, and an interface for transmitting and receiving data between the information processing device 1 and the outside.

[0016] In this figure, the information processing device 1 is shown as a single device, but it may be configured as a system in which multiple information processing devices 1 cooperate to perform processing. Also, it may be configured such that at least one of multiple processing circuits 10 or multiple storage units 12 is provided in one information processing device 1. It may also be configured in a combination of these, in which one or multiple processing circuits 10 and / or one or multiple storage units 12 are provided in multiple information processing devices 1.

[0017] The processing circuit 10 is a circuit for performing model training. In one embodiment, the processing circuit 10 is a circuit for training a language model (including a large-scale language model, or other generative model). The processing circuit 10 may, for example, include a preprocessor 100, a tokenizer 102, an encoder 104, a trainer 106, and a postprocessor 108.

[0018] The memory unit 12 is a circuit that stores data related to model training. The memory unit 12 is configured to be able to store a first data set 120, a second data set 122, a vocabulary 124, a vectorization function 126, and parameters 128. Each piece of data may be stored in a different area or in a different housing.

[0019] The preprocessor 100 forms a model by, for example, acquiring parameters related to a model to be trained from the storage unit 12. The preprocessor 100 also performs preprocessing related to tokenization on data such as sentences acquired from a dataset. The preprocessing related to tokenization is, for example, a process of extracting sentences to be used for training from a dataset and defining a start token for the sentences according to the type of dataset.

[0020] For example, the preprocessor 100 can define different start tokens when extracting sentences from the first dataset 120 and when extracting sentences from the second dataset 122. That is, the start token can also be treated as meta-information of the sentence that indicates whether the sentence is from the first dataset 120 or the second dataset 122. Furthermore, when the preprocessor 100 defines a special start token, it performs a process of adding the start token to the vocabulary 124. The preprocessor 100 may define the vocabulary 124 in a blank area in the vocabulary 124, or may define the vocabulary 124 by adding an area to the vocabulary 124.

[0021] In the following description, the first dataset 120 is a publicly available corpus, while the second dataset 122 is a corpus containing a target that the user particularly wants to train on. The second dataset 122 is a corpus containing data that the user wants to keep secret, such as a corpus containing data that is kept secret only between parties who have signed a confidentiality agreement, a corpus that is used in training but that the user does not want third parties to know about as know-how information for training, a corpus containing sensitive information, a domain-specific corpus that the user does not want third parties to know about, or a corpus containing data that the user owns the copyright to but does not want others to use.

[0022] Note that the user in the present disclosure may include a person who trains a model using the information processing device 1 and a person who provides a model trained by the information processing device 1, and is not limited to a specific person.

[0023] For example, when extracting sentences from the first data set 120 as training data, the preprocessor 100 sets a first token, which is a start token assigned to general sentences already included in the vocabulary 124, as the start token. For example, when extracting sentences from the second data set 122 as training data, the preprocessor 100 sets a second token, which is a special start token different from the first token, as the start token.

[0024] The tokenizer 102 executes a process of dividing the extracted sentence into tokens and outputting a token sequence. Here, a token refers to a component of data (a sentence, for example), and the token sequence obtained by tokenizing data using the tokenizer 102 is a sequence of multiple components obtained by dividing the data into data components. In a broad sense, the tokenizer 102 may include a scanner. The tokenizer 102 divides sentences into tokens, for example, by lexical analysis. The tokenizer 102 executes tokenization so as to divide the data into tokens suitable for the subsequent encoding and training processes.

[0025] The tokenizer 102 divides a sentence into tokens by, for example, morphological analysis, syntactic analysis, semantic analysis, and contextual analysis according to the language of the training target, but this disclosure does not exclude other implementations, and various other techniques may be used. For example, tokens may be divided into any unit using any technique, such as at the byte level, character, subword, word, word piece, or sentence piece.

[0026] The tokenizer 102 assigns the starting token defined by the preprocessor 100 as the starting token for each sentence depending on the dataset from which the sentence to be tokenized is extracted. In other words, the tokenization rules may include rules regarding the starting token of a token sequence, and the tokenizer 102 may use different tokenization rules depending on the dataset containing the data to be tokenized (sentences, for example). Using different tokenization rules may conceptually include using tokenizers with different tokenization rules.

[0027] As a non-limiting example, the tokenizer 102 may use different tokenization rules depending on at least one of the attributes, type, classification, or domain of a dataset that includes the data to be tokenized.

[0028] As a non-limiting example, when a token sequence obtained from data (for example, a sentence) included in a first dataset is used to train a language model, tokenizer 102 may tokenize the data included in the first dataset according to rules different from the tokenization rules for obtaining a token sequence to be used to train a language model from data (for example, a sentence) included in a second dataset.

[0029] As a non-limiting example, the tokenizer 102 may tokenize the sentence and then assign a start token defined by the preprocessor 100 to the sentence.

[0030] Furthermore, the tokenizer 102 may not be configured as a single tokenizer but may be configured as a set of multiple tokenizers. In this case, instead of the preprocessor 100 defining the start token, the preprocessor 100 may notify the tokenizer 102 of the identifier of the acquired data set, so that the tokenizer 102 may perform tokenization using an appropriate start token according to the data set.

[0031] The encoder 104 vectorizes each token in the token sequence output from the tokenizer 102 and converts each token into a point in a vector space (e.g., a vector space of five or more dimensions). The vector space is not limited to a vector space with a specific number of dimensions, and may be, for example, a vector space with five or more dimensions. Although the encoder 104 is called an encoder, it does not refer to a specific encoder in so-called machine learning, and may be, for example, an encoder that performs a process of converting tokens into vectors using some kind of function.

[0032] For example, the encoder 104 vectorizes the tokens using a vocabulary 124 and a vectorization function 126 stored in the storage unit 12. The encoder 104 may vectorize the tokens of the acquired identifiers by obtaining an identifier corresponding to each token from the vocabulary 124 and inputting the identifier to an embedding layer that is the vectorization function 126 that vectorizes the identifier, or may vectorize the tokens of the acquired identifiers by referring to a lookup table that corresponds to the vectorization function 126 that vectorizes the identifier. The identifiers of the tokens may be, for example, one-hot vectors.

[0033] The vectorization function 126 for the embedding layer may be, for example, a matrix representation of vectors for token identifiers, i.e., the encoder 104 vectorizes the second tokens defined in the vocabulary 124 defined by the preprocessor 100 using the vectorization function 126.

[0034] Similarly, when using a lookup table, vectors corresponding to the identifiers of each token in the vocabulary 124 are stored as row data of the table, and the encoder 104 can convert the tokens into vectors by extracting the vectors corresponding to the token identifiers. Similarly, the second token can be vectorized using a vectorization function 126 expressed as a lookup table.

[0035] The trainer 106 trains the model by inputting the vector of each token into the model to be trained in order to calculate the weights, and optimizing the parameters of the model to be trained from the resulting data. By providing an appropriate starting token, the trainer 106 can train a model that generates language using a probability distribution for a token sequence starting from this starting token.

[0036] By performing training using data from a first dataset to which the first token has been added, the model is able to generate sentences that have the characteristics of the sentences contained in the first dataset. By performing training using data from a second dataset to which the second token has been added, the model is able to generate sentences that have the characteristics of the sentences contained in the second dataset.

[0037] In this way, by using different starting tokens, it is possible to train a model that generates data with the characteristics of each dataset. Meanwhile, after training is complete, the model selects the next token according to a probability distribution defined by the trained weights, based on the combination of tokens for the data to be generated. Therefore, even if the starting token is deleted, appropriate sentences can be generated by using the trained model.

[0038] For training, typically, the encoder 104 and the trainer 106 perform data conversion and training for a model described as a Transformer with an Attention function, and each optimized parameter is stored (output) in the parameter 128 area of ​​the memory unit 12. Without being limited to this, the encoder 104 and the trainer 106 may perform data conversion and training for using Attention and Long Short Term Memory (LSTM) in a Recurrent Neural Network (RNN) as the model to be trained.

[0039] The information processing device 1 is not limited to the above, and may train any model using data processed by a Natural Language Processing (NLP) technique that implements tokenization and vectorization. The trainer 106 may thus use any model training technique that trains a language model using tokenized and vectorized data. This training technique may be a general technique.

[0040] The post-processor 108 performs post-processing after the training device 106 completes optimization. The information processing device / information processing system that generates data (hereinafter referred to as the data generating device) realizes data generation by using the same tokenizer 102 for tokenizing data and the same encoder 104 for vectorization as those in the information processing device 1.

[0041] As described above, the data generation device can obtain a series of sentence data starting from a start token in the vocabulary 124 by acquiring the start token. In other words, by performing random sampling starting from the start token, a third party can generate a large number of sentences similar to the data used for training by the information processing device 1. Therefore, by using these many generated sentences, a third party may be able to infer the content and format of the dataset used for training by the information processing device 1.

[0042] To avoid this, the post-processor 108 deletes information related to the second token from the tokenizer 102. Specifically, the post-processor 108 deletes information related to the second token defined in the vocabulary 124, i.e., the second token and the identifier of the second token, or deletes the association between them, thereby reducing the possibility of recovering data starting from the second token.

[0043] The data starting from the second token is the second data set 122, and by performing the above-described post-processing, it becomes difficult for a third party to infer the contents of the second data set 122 even if they perform random sampling using the model generated by the information processing device 1. As described above, the second data set 122 is information that the user of the information processing device 1 wants to keep secret. As a result, by performing post-processing by the post-processor 108, it becomes possible to conceal what data was used for training.

[0044] Furthermore, in addition to or instead of deleting the second token from the vocabulary 124, the post-processor 108 may delete information related to the second token from the vectorization function 126. By deleting the information related to the second token from the vocabulary 124, it becomes difficult to detect the starting position in the vector space from which data related to the second data set is generated, thereby achieving the same effect as above. However, since there is a possibility that prediction may be made to some extent, it is more desirable for the post-processor 108 to also delete information related to the second token from the vocabulary 124 when deleting the information related to the second token from the vectorization function 126.

[0045] As a specific process, for example, the post-processor 108 deletes information related to the second token from the vocabulary 124. The post-processor 108 can delete information related to the second token from the vocabulary 124 by deleting the second token and / or an identifier of a token related to the second token from the list of the vocabulary 124.

[0046] In addition, the post-processor 108 can delete information related to the second token from the vectorization function 126, for example, by deleting the row (or column, as the case may be) related to the second token from a matrix indicating the embedding layer of the vectorization function 126, or by deleting the row (or column, as the case may be) related to the second token from a lookup table of the vectorization function 126.

[0047] That is, the post-processor 108 removes information about the second token from the vocabulary 124 used by the tokenizer 102 and / or the vectorization function 126 that indicates the transformation function used by the encoder 104 .

[0048] By using a vocabulary 124, vectorization function 126, and parameters 128 that are common to the information processing device 1, the data generation device is able to generate sentences using a model trained using data included in the first data set 120 and the second data set 122. The data generation device may be able to reproduce the data included in the first data set 120 by random sampling using data starting from the first token, but because the data of the second token has been deleted from the vocabulary 124 and / or the vectorization function 126, it becomes difficult to reproduce the sentences included in the second data set 122 even by random sampling using an appropriate starting token.

[0049] Next, the above process will be explained in more detail using a flowchart.

[0050] For example, the following sentence is used as the first dataset (a dataset that may be made public):

[0051] "When I woke lightly, the buzzing of the bees still lingered, its deep, resonant sound clearly lingering in my ears. I turned around and ran to the entrance door beside my bed. I put my face close to the brass metal fitting, through which only the keyhole was visible. However, the surface of the metal fitting did not reflect my face. It simply reflected a dim yellow light."

[0052] For example, the following sentences are used as the second dataset (the dataset to be kept secret):

[0053] "Since Hosui had not publicly announced his solution to the St. Alexei Cathedral murders, rumors began to circulate that the case was about to go cold. On the tenth day, the investigation team was forced to abandon their pursuit of Lazarev's murderer. This was because a poisoner suddenly began wandering around the Furuyagi mansion, which had been shrouded in mystery for 400 years and was said to be the holy family since the days of the Usuki Yaso Seminary, like a pitch-black wind. It was rumored that a mysterious terror like this would surely occur at some point in the Furuyagi mansion, commonly known as the Black Death Mansion."

[0054] FIG. 2 is a flowchart showing an example of processing by the information processing device 1 according to an embodiment.

[0055] The processing circuit 10 first defines a starting token to be assigned to a dataset to be concealed in a vocabulary (S100). Subsequently or in parallel with this, the processing circuit 10 adds an element (e.g., a row) that vectorizes the starting token to a function that performs vectorization (e.g., a matrix or lookup table representing the embedding layer described above), thereby defining the vectorization of the starting token (S102). The processes of S100 and S102 may be performed by, for example, the preprocessor 100.

[0056] A starting token is defined for each of the first and second data sets, and the first token that is the starting token for the first data set can be a starting token already defined in the vocabulary. In the processes of S100 and S102, the processing circuit 10 particularly defines a second token that is the starting token for the second data set and defines it in the vocabulary and the vectorization function.

[0057] The processing circuit 10 may, for example, generate a token <bos>In response to this, the processing circuit 10 may use, for example, the token <st>This string of tokens is given as a non-limiting example.

[0058] The processing circuit 10 extracts data to be used for training from the first data set or the second data set (S104). Subsequently, the processing circuit 10 sets a corresponding start token depending on whether the process of S104 is extraction from the first data set or extraction from the second data set (S106). The processes of S104 and S106 may be executed by the preprocessor 100.

[0059] For example, the processing circuit 10 sets a first token if the data is extracted from a first data set, and sets a second token if the data is extracted from a second data set.

[0060] In addition, when training data is acquired from the same dataset consecutively for a certain number of epochs in training, an appropriate start token may be set the first time, and the initially set start token may be used during subsequent training using data extracted from the same dataset. The same applies when using data extracted from the same dataset in batch processing or mini-batch processing.

[0061] The processing circuit 10 performs tokenization on the extracted training data (S108). The token sequence obtained as a result of tokenization is a token sequence whose beginning is the start token set in S106 and whose subsequent tokens correspond to the training data. The processing of S108 may be performed by the tokenizer 102. The processing circuit 10 can perform tokenization using a vocabulary whose start token has already been defined in S100. For example, the first sentence of the first dataset above may be tokenized as follows:

[0062] <bos>When I woke up, the buzzing sound of the bees still lingered, its deep, resilient resonance clearly lingering in my ears.

[0063] For example, the first sentence of the second dataset above may be tokenized as follows:

[0064] <st>Since Hosui had not publicly announced his solution to the murder at St. Alexei Cathedral, rumors began to circulate that the case might soon be unsolved. On the tenth day, the investigation's leadership was forced to abandon its pursuit of Lazarev's murderer.

[0065] Note that both of the above are Japanese sentences, and so have been divided into tokens using, for example, morphological analysis, but it is possible to perform tokenization appropriate to the data used as a dataset, such as by language.

[0066] For example, in the case of English text, tokenization can be performed based on the commonly used word separation using spaces and classification such as appropriate conjugation.

[0067] For example, in the case of source code, tokenization can be performed on at least one of the following elements: space, operator, parentheses, comma, period, colon, semicolon, quotation mark, double quotation mark, exclamation mark, question mark, pound sign, dollar sign, ampersand, tilde, pipe, tab, newline, and escape sequence, as a symbol that separates or gives meaning to an element.

[0068] Subsequently, or in parallel, processing circuitry 10 may replace each token in the tokenized data with the token's identifier based on the vocabulary.

[0069] As an example, as shown above, the processing circuit 10 assigns a first token as a starting token for data extracted from the first data set, and assigns a second token as a starting token for data extracted from the second data set.

[0070] As another example, the processing circuit 10 may tokenize the extracted data to obtain a corresponding identifier, and then assign a start token. In this case, the processing circuit 10 can assign the identifier for the start token defined in S100 as the identifier for the start token of the data that has been tokenized and converted into the identifier for each token.

[0071] As yet another example, the processing circuit 10 may use a different tokenizer for each data set. For example, the processing circuit 10 may use a first tokenizer to generate a token sequence with a first token as a starting token for the data of a first data set, and may use a second tokenizer to generate a token sequence with a second token as a starting token for the data of a second data set.

[0072] In this way, by providing different starting tokens, the processing circuit 10 <bos>In training using data given the first token, training is performed to generate data with the features of the first dataset, and the second token is generated. <st>In training using data given the second dataset, training can be performed to generate data having the features of the second dataset.

[0073] The processing circuit 10 vectorizes the data obtained by tokenizing (S110). The process of S110 may be performed by the encoder 104. The processing circuit 10 can perform appropriate vectorization even when a second token is included, using the vectorization function for the start token defined in S102.

[0074] The processing circuit 10 optimizes the generative model by any appropriate method using the vectorized data, i.e., the data converted into embedding vectors (S112). The process of S112 may be performed by the trainer 106. Note that in the optimization, it is possible to optimize not only the generative model but also at least a part of the vectorized function.

[0075] The process of S112 may be repeatedly executed for one or more data items.

[0076] The processing circuit 10 determines whether the training is complete (S114). Any condition can be used as the condition for completing the training. For example, the processing circuit 10 may determine the completion of the training based on an evaluation using some evaluation function, the training time, or a predetermined number of epochs.

[0077] If it is determined that the training is not complete (S114: NO), the processing circuit 10 continues optimizing the generative model by repeatedly executing the process from S104.

[0078] If it is determined that the training is complete (S114: YES), the processing circuit 10 deletes at least one of the second token defined in the vocabulary in S100 and the data related to the conversion of the second token defined in the vectorization function in S102 (S116). That is, the processing circuit 10 deletes the data of the start token related to the dataset to be kept secret from at least one of the vocabulary and the vectorization function. The processing of S116 may be executed by the post-processor 108.

[0079] By the processing circuit 10 deleting the second token from the vocabulary, it becomes difficult to obtain data for the second token, which is the starting token, and it becomes difficult to reproduce the second data set by random sampling using this second token as the starting token.

[0080] By the processing circuit 10 deleting the data relating to the second token from the vectorization function, it becomes difficult to convert the second token into a vector space, which in turn makes it difficult to reproduce the second data set by random sampling.

[0081] However, if only the second token is removed from the vocabulary, it may be possible to recover the second token by reverse engineering the vectorization function, so it is more desirable to also remove the second token from the vectorization function 126.

[0082] Preferably, in addition to the above, the processing circuit 10 deletes information related to the second token from the parameters 128, along with the second token defined in S100 and / or the transformation data related to the second token defined in the vectorization function 126. For example, the processing circuit 10 can delete the input node corresponding to the second token and the edge (including the weight) connecting to the input node from the parameters 128, from the model to be trained.

[0083] This process makes it possible to delete information related to the second token from the model being trained, making reverse engineering more difficult. In other words, this deletion process makes it difficult for a third party to discover the existence of the second token even when referencing the network related to the trained model.

[0084] This process can also be applied to the following modifications or embodiments to the extent that there is no contradiction.

[0085] 1, the vectorization function 126 and the parameters 128 are shown separately for convenience, but this is not limiting. For example, the vectorization function 126 may be incorporated as a training target model. Even in this case, it is more desirable for the processing circuit 10 to delete information about the input node corresponding to the second token in the training target model and the edge connected to the input node.

[0086] In the present embodiment, as an example, the process of S116 is executed after the training is completed, but this is not limited thereto. The processing circuit 10 may execute the process of S116, for example, after completing training using at least the second data set. That is, for example, when training on the first data set is executed after training on the second data set, the processing circuit 10 may execute the process of S116 before training on the first data set.

[0087] As described above, according to this embodiment, for a dataset that is to be kept secret, a starting token that is different from that of other datasets is temporarily defined during training, and data related to the starting token is deleted after training is completed, thereby concealing the starting token of the dataset that is to be kept secret and making it difficult to restore the contents of the dataset by random sampling from the starting token.As a result, a user can prevent third parties from easily reproducing the contents of a dataset that the user does not want known to third parties, even if the language model (generative model) is made public.

[0088] In the above embodiment, the tokenizer 102 uses different tokenization rules depending on the data set including the data to be tokenized. However, the tokenizer 102 may use different tokenization rules depending on the data to be tokenized. For example, the tokenizer 102 may use different tokenization rules depending on at least one of the attribute, type, classification, and domain of the data to be tokenized.

[0089] (Variation)

[0090] In the first embodiment described above, the reproduction of data by random sampling is suppressed by deleting information about the second token assigned to the second data set to be kept secret from the vocabulary and / or vectorization function, but the present invention is not limited to this. Even if the second token is not deleted from the vocabulary and vectorization function, it is difficult for a third party who recognizes the first token as the start token to perform random sampling.

[0091] Therefore, the process of S116 in Fig. 2 may be omitted. In other words, the trained model may be made public without deleting the start token corresponding to the data set that is to be kept secret.

[0092] As described above, it is possible to configure the model so that it is not necessary to delete the second token from the vocabulary and the embedding vector corresponding to the second token. This configuration makes it possible to train the model on data with low confidentiality that has different characteristics for each domain.

[0093] For example, when data is added to a dataset and fine-tuning is performed on a pre-trained model, it becomes possible to perform domain-specific training for each data set. It also makes it easier for third parties to perform domain-specific transfer learning by using open source software for secondary use. Of course, users can select an appropriate license and provide appropriate information to third parties.

[0094] For example, in an information processing device 1 that uses a trained model that has been fine-tuned through domain-specific training, the processing circuit 10 can convert data generated by the trained model into domain-specific data by inputting a token sequence in which the starting token is the second token to the trained model. That is, the processing circuit 10 can tokenize data input by a user, such as a prompt, according to a tokenization rule selected from a plurality of tokenization rules, and cause the trained model to generate data based on the token sequence obtained thereby.

[0095] For example, the tokenization rule may be selected from either a rule that sets the start token as the first token or a rule that sets the start token as the second token. Which rule is selected may be determined by the processing circuit 10 in accordance with a selection by a user of the information processing device 1, or may be determined by the processing device 10 in accordance with the attributes of the user, or may be determined by the processing device 10 analyzing a prompt input by a user to use a trained model and in accordance with the analysis results, and is not limited to a specific method.

[0096] For example, in an open source project or the like, if additional domain-specific learning by third parties is permitted, the vocabulary, embedding layer, and parameters formed without deleting the information related to this second token may be made public.

[0097] (Second embodiment)

[0098] Although the above description has been given of a case where there are two data sets, the present invention is not limited to this. The information processing device 1 can also set multiple concealed data sets for three or more training data sets.

[0099] 3 is a block diagram schematically illustrating an example of an information processing device according to an embodiment. The storage unit 12 may store a first data set 1201, a second data set 1202, ..., an n-th data set 120n (n >= 3). The first data set 1201 is a data set that may be made public, and the second data set 1202, ..., the n-th data set 120n are data sets that must be kept secret.

[0100] If there are multiple datasets that you want to keep secret, and you want to incorporate the characteristics of each dataset, such as from different domains, into the model, the processing circuit 10 can train each dataset with a different starting token.

[0101] The preprocessor 100 defines the data relating to the second token, ..., n-th token in the vocabulary 124 and the vectorization function 126, respectively.

[0102] The tokenizer 102 performs tokenization by assigning, for example, a first token to the first data set 1201, a second token to the second data set 1202, ..., and an n-th token to the n-th data set 120n as a starting token. In this way, the tokenizer 102 may use different tokenization rules depending on the data to be tokenized or the data set including that data.

[0103] The encoder 104 converts the token sequence into a vector sequence using a vectorization function 126 that the preprocessor 100 defined for a particular starting token.

[0104] The trainer 106 can perform training of the model using a sequence of tokens starting from each starting token.

[0105] After training is complete, the post-processor 108 may remove the second token, ..., nth token from the vocabulary 124 and / or the vectorization function 126.

[0106] As described above, even when there are three or more datasets used for training and two or more datasets to be kept secret, it is possible to set a unique start token for each dataset, as in the above-described embodiment, to train with the characteristics of each dataset, and to make it difficult to reproduce the contents of the dataset to be kept secret.

[0107] (Third embodiment)

[0108] In each of the above-described embodiments, attention has been focused on the start token, but it is also possible to make the reproducibility of the data of the dataset even more difficult by using other tokens. For example, even if the start token is concealed, it may be possible to reproduce the data of the second dataset from data using phrases (characters, words, expressions, etc.) that frequently appear in the sentences of the second dataset. As a more specific example, for example, by starting with a period ".", it may be possible to obtain the sentence following the period by random sampling.

[0109] The information processing device 1 can convert characters (words, expressions) that frequently appear in such sentences into special tokens and perform training. The processing circuit 10 may, for example, set special tokens as follows and tokenize the data included in the second dataset. That is, the tokenization rules may include rules regarding tokenizing specific data (e.g., characters), and the tokenizer 102 may use different tokenization rules depending on the data to be tokenized or the dataset containing that data. Note that frequently appearing characters (words, expressions) may be characters (words, expressions) that are more likely to appear in the dataset than other characters, for example.

[0110] <st>Because Hosui did not publicly announce the solution to the murder at St. Alexei Temple,<ST:、> It was about the tenth day that rumors of entering the labyrinth began to spread.<ST:、> From that day on, the main investigation team<ST:、> I had to abandon my pursuit of Lazarev's murderer.<ST:。> / <st:lf> / <st>That is to say<ST:、> / It has been a constant presence for four hundred years.<ST:、> At the Furuyagi mansion, which has been called the sacred family since the Usuki Yaso Seminary<ST:、> Suddenly, a truly black wind-like poisonous murderer began to wander about.<ST:。> / <st> / the / <ST:、> The Furuyagi mansion, commonly known as the Black Death Mansion, is<ST:、> It was rumored that someday, a mysterious horror like this would surely occur.<ST:。> / <st:lf>

[0111] In the above example, the comma<ST:、> , a period<ST:。> , line break code <st:lf>It is replaced with a token. By using such special tokens, it becomes possible to make it more difficult to restore data from data such as characters frequently included in a general dataset. In the above example, an expression "Black Death Mansion", which is a unique expression, may be converted into a special token such as "<ST: Black Death Mansion>", defined in training, and deleted after training.

[0112] The addition of special tokens to the vocabulary, the addition of a vectorization function, and the deletion of these added information after training can be realized in the same manner as in the above-described embodiment.

[0113] As described above, it is also possible to define special tokens other than the start token and execute training. By using special tokens, it becomes possible to suppress the restoration of data in the dataset with higher accuracy.

[0114] In the above, commas and line break codes are used, but special tokens may be set for other symbols such as commas, periods, and colons. As another example, in the case of a dataset specialized for a domain, special tokens can also be set for words frequently used in that domain.

[0115] As other examples of symbols for setting special tokens, various parentheses, ellipsis, black dot, hyphen, quotation mark, double quotation mark, exclamation mark, question mark, dash, wavy dash, colon, semicolon, apostrophe, etc. can be mentioned, but it is not limited to these.

[0116] By deleting the information related to the special token from the input layer side, a third party using the trained model can appropriately use the model to generate a sentence including a normal token corresponding to the special token, for example, in the above example, the general token "," corresponding to "<ST:,">, and obtain a sentence incorporating the characteristics of the anonymized dataset. On the other hand, sampling randomly using this general token "," to sample randomly using "<ST:," becomes difficult as long as there is no means to know the deleted special token.

[0117] (Fourth embodiment)

[0118] In sentence training, end tokens are sometimes implemented in the same way as start tokens, as follows:

[0119] <bos>When I woke up, the buzzing sound of the bees still lingered, its deep, resilient resonance clearly lingering in my ears. <eos> / <lf> / <bos>I turned around and ran to the entrance door at the headboard of my bed. <eos> / <bos>I put my face close to the brass piece of metal that had only the keyhole open. <eos> / <bos>However, the surface of the metal fittings did not reflect my face. <eos> / <bos>It merely reflected a pale yellow light. <eos> / <lf>

[0120] When using an end token, it is also possible to set an end token for the second dataset, i.e., the concealed dataset, and delete information related to this end token after training, as follows: That is, the tokenization rules may include rules related to the end token of a token sequence, and the tokenizer 102 may use different tokenization rules depending on the data to be tokenized or the dataset containing that data.

[0121] <st>Since Hosui had not publicly announced his solution to the murder at St. Alexei Cathedral, rumors began to circulate that the case might soon be unsolved. On the tenth day, the investigation's leadership was forced to abandon its pursuit of Lazarev's murderer. <st:eos> / <lf> / <st>This is because, in the Furiyagi mansion, which has been a sacred family since the days of the Usuki Yaso Seminary and which has been a part of the family for over 400 years, a poisonous murderer suddenly began wandering about like a pitch-black wind. <st:eos> / <st>It was rumored that someday, a mysterious horror like this would surely occur in the Furuyagi mansion, commonly known as the Black Death Mansion. <st:eos> / <lf>

[0122] In this way, an end token can also be set as a special token.

[0123] (Fifth embodiment)

[0124] Depending on the tokenization rules, there may be cases where the start token does not exist. In such cases, as in the above-described embodiment, for example, <st>After training, we define a special token called <st>The information processing device 1 may perform training by deleting information relating to this. This is also an example of a tokenization rule regarding the starting token of a token sequence.

[0125] As another example, training may be performed on the data included in the second dataset with the second token left blank (which is equivalent to not adding the second token). This is also an example of a tokenization rule regarding the starting token of a token sequence.

[0126] The information processing device 1 performs training using different tokenization rules depending on the data to be tokenized or the dataset containing that data, and thus, even in this form, it is possible to improve the readability of the data contained in the second dataset obtained by random sampling, as in the above-mentioned embodiments.

[0127] In the above-described embodiments, tokenizing, which converts data into a token string, has been described as being different from the process of obtaining a vector (vector string) for each token in the token string, but the concept of tokenizing in the present disclosure may also include the process of obtaining a vector from each token. In other words, tokenizing data may include the process of tokenizing data and obtaining data (e.g., a vector) in a format that can be input to a model.

[0128] The token sequence obtained by tokenizing may be a token sequence in which each token is expressed as data in a format to be input to a model (e.g., a vector). The tokenizing rules may include rules used by the encoder 104 when vectorizing each token in the token sequence obtained by tokenizing (e.g., a rule for obtaining an identifier corresponding to a token, a rule for vectorizing the obtained identifier, etc.). In other words, the tokenizing rules may include rules for obtaining data in a format to be input to a model (e.g., a vector of the token) from a token.

[0129] For example, the tokenization rules may include multiple tables (e.g., vocabularies) in which different identifiers are associated with the same token, multiple functions that obtain different vectors for the same identifier, or multiple tables (e.g., lookup tables) in which different vectors are associated with the same token.

[0130] According to the present disclosure, it is possible to realize data obfuscation in a dataset that one wishes to conceal using various methods as described above, which makes it possible to conceal, for example, data that must be protected for security or privacy reasons, know-how on corpus usage, and the like.

[0131] The embodiments of the present disclosure are not limited to model generation, but can also be used in additional learning, such as model fine-tuning, transfer learning, or distillation. In the example embodiment described above, a model already trained on a first dataset can be additionally trained using data from a second dataset.

[0132] Without being limited to this, it is also possible to apply additional learning to a model that has already been trained on a first dataset using data from the first dataset and the second dataset, or additional learning to a model that has already been trained on the first dataset and the second dataset using data from at least one of the first dataset or the second dataset.

[0133] The embodiments of the present disclosure can also be described as follows. Furthermore, these can be combined or partially substituted without departing from the spirit of the present disclosure.

[0134] (1) An information processing device that trains a generative model based on a token sequence obtained by tokenizing data included in a dataset, the information processing device comprising: one or more memories; and one or more processors, wherein when a token sequence obtained from second data is used to train the generative model, the one or more processors tokenize the second data according to a tokenization rule that differs from the tokenization rule for obtaining the token sequence used to train the generative model from first data.

[0135] (2) The information processing device according to (1), wherein the tokenization rule includes any one of a rule regarding a start token of a token string, a rule regarding tokenization of specific data, or a rule regarding an end token of a token string.

[0136] (3) The information processing device described in (1), wherein the one or more processors define a first token in a vocabulary that is set as a starting token, and when data included in a first dataset is used to train the generative model, the information processing device trains the generative model based on a token sequence in which the first token is set as a starting token, unlike when data included in a second dataset is used to train the generative model.

[0137] (4) The information processing device described in (1), wherein the one or more processors define a first token and a second token different from the first token in a vocabulary, train the generative model based on a token sequence in which the first token is set as a starting token when data included in a first dataset is used to train the generative model, and train the generative model based on a token sequence in which the second token is set as a starting token when data included in a second dataset is used to train the generative model.

[0138] (5) The information processing device according to (4), wherein the one or more processors delete the second token from the vocabulary after training using at least the second dataset is completed.

[0139] (6) The information processing device described in (4), wherein the one or more processors, after completing training using at least the second dataset, delete an element that converts the second token into a vector from a function that converts data included in the vocabulary into a vector.

[0140] (7) The information processing device according to (5) or (6), wherein the one or more processors delete information relating to the second token in the network relating to the generative model after training using at least the second dataset is completed.

[0141] (8) The information processing device described in (6) or (7) that cites (6), wherein the function is an embedding layer, and the one or more processors delete a row or column related to the second token from a matrix representing the embedding layer.

[0142] (9) The information processing device according to (6) or (7) citing (6), wherein the function is a lookup table, and the one or more processors delete rows or columns related to the conversion of the second token from the lookup table.

[0143] (10) The information processing device according to any one of (4) to (9), wherein the one or more processors further set a third token to an n-th token (n >= 3, or only the third dataset / third token when n == 3) corresponding to the third dataset to the n-th dataset, respectively, in the vocabulary, and when acquiring training data from the third dataset to the n-th dataset, train the generative model using data to which a token corresponding to the dataset of the training data has been assigned.

[0144] (11) The information processing device described in (10), wherein the one or more processors, after completing training using at least one of the third data set to the nth data set, delete from the vocabulary tokens among the third tokens to the nth tokens that correspond to the data set for which training has been completed.

[0145] (12) The information processing device described in (11), wherein the one or more processors, after completing training using at least one of the third to nth data sets, delete an element from a function that converts data included in the vocabulary into a vector, the element converting a token corresponding to the data set for which training has been completed among the third to nth tokens into a vector.

[0146] (13) The information processing device according to any one of (4) to (12), wherein the generative model is a model trained in advance using at least the first dataset and the first token, and the one or more processors at least: define the second token in the vocabulary; extract training data from the second dataset; and tune the generative model using the second token.

[0147] (14) The information processing device according to any one of (4) to (13), wherein the second token is empty.

[0148] (15) The information processing device according to any one of (4) to (14), wherein the one or more processors convert one or more tokens in the second dataset into special tokens, and train the generative model using the special tokens.

[0149] (16) The information processing device according to (15), wherein the one or more tokens include a token that is more likely to be present in the second data set than other tokens.

[0150] (17) The information processing device according to (15) or (16), wherein the one or more tokens include at least one of a comma, a period, a comma, a period, or a line feed code.

[0151] (18) The information processing device according to any one of (15) to (17), wherein the one or more processors delete the special token from the vocabulary after training using the second dataset is completed.

[0152] (19) An information processing device that causes the generative model obtained by training according to any one of (1) to (18) to generate data, the information processing device comprising: one or more memories; and one or more processors, wherein the one or more processors cause the trained generative model to generate data based on a token sequence obtained by tokenizing data, and the tokenizing rule is selected from a plurality of tokenizing rules.

[0153] (20) The information processing device according to any one of (1) to (19), wherein the token sequence obtained by tokenizing data is a sequence of a plurality of components obtained by dividing the data into the components of the data.

[0154] (21) The information processing device according to any one of (1) to (20), wherein the generative model is a language model that generates data based on a token sequence.

[0155] (22) The information processing device according to any one of (1) to (21), wherein the first data is data included in a first data set, the second data is data included in a second data set, and the second data set is a data set with higher confidentiality than the first data set.

[0156] (23) The information processing device according to (22), wherein a tokenization rule for utilizing the generative model trained using the first data and the second data is the same as a tokenization rule when training the generative model using the first data.

[0157] (24) An information processing method executed in any one of the information processing devices (1) to (23).

[0158] (25) A program that causes a processing circuit to execute the information processing method described in (24).

[0159] (26) A non-transitory computer-readable medium storing a program that causes a processing circuit to execute the information processing method described in (24).

[0160] Some or all of the devices (information processing devices 1) in the above-described embodiments may be configured as hardware, or may be configured as software (programs) executing information processing by a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), etc. When configured as software information processing, software that realizes at least some of the functions of each device in the above-described embodiments may be stored on a non-transitory storage medium (non-transitory computer-readable medium) such as a CD-ROM (Compact Disc-Read Only Memory) or a USB (Universal Serial Bus) memory, and the software information processing may be executed by loading the software into a computer. The software may also be downloaded via a communication network. Furthermore, all or part of the software processing may be implemented in a circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array), thereby allowing the software information processing to be executed by hardware.

[0161] The storage medium that stores the software may be a removable medium such as an optical disk, or a fixed medium such as a hard disk or memory. The storage medium may be provided inside the computer (such as a main storage device or auxiliary storage device) or outside the computer.

[0162] 4 is a block diagram showing an example of the hardware configuration of each device (information processing device 1) in the above-described embodiment. Each device may be realized as a computer 7 including, for example, a processor 71, a main storage device 72 (memory), an auxiliary storage device 73 (memory), a network interface 74, and a device interface 75, all of which are connected via a bus 76.

[0163] Although the computer 7 in FIG. 4 includes one of each component, it may include multiple of the same component. Also, while FIG. 4 shows one computer 7, the software may be installed on multiple computers, and each of the multiple computers may execute the same or different parts of the software. In this case, a distributed computing configuration may be used in which each computer communicates with the other computers via a network interface 74 or the like to execute the processing. In other words, each device (information processing device 1) in the above-described embodiment may be configured as a system in which one or more computers execute instructions stored in one or more storage devices to realize its functions. Furthermore, the system may be configured such that information sent from a terminal is processed by one or more computers located on a cloud, and the processing results are sent to the terminal.

[0164] The various calculations of each device (information processing device 1) in the above-described embodiments may be executed in parallel using one or more processors, or using multiple computers via a network. Furthermore, the various calculations may be distributed to multiple processing cores within a processor and executed in parallel. Furthermore, some or all of the processes, means, etc. disclosed herein may be implemented by at least one processor and storage device provided on a cloud that can communicate with computer 7 via a network. Thus, each device in the above-described embodiments may be implemented in the form of parallel computing using one or more computers.

[0165] The processor 71 may be an electronic circuit (processing circuit, processing circuitry, CPU, GPU, FPGA, ASIC, etc.) that at least controls or performs calculations on a computer. The processor 71 may also be a general-purpose processor, a dedicated processing circuit designed to perform a specific calculation, or a semiconductor device that includes both a general-purpose processor and a dedicated processing circuit. The processor 71 may also include an optical circuit or a calculation function based on quantum computing.

[0166] The processor 71 may perform arithmetic processing based on data or software input from each device, etc., configured inside the computer 7, and may output the calculation results or control signals to each device, etc. The processor 71 may control each component constituting the computer 7 by executing the OS (Operating System) of the computer 7, applications, etc.

[0167] Each device (information processing device 1) in the above-described embodiment may be realized by one or more processors 71. Here, the processor 71 may refer to one or more electronic circuits arranged on one chip, or may refer to one or more electronic circuits arranged on two or more chips or two or more devices. When multiple electronic circuits are used, the electronic circuits may communicate with each other via wire or wirelessly.

[0168] The main memory device 72 may store instructions executed by the processor 71 and various data, etc., and information stored in the main memory device 72 may be read by the processor 71. The auxiliary memory device 73 is a memory device other than the main memory device 72. Note that these memory devices refer to any electronic component capable of storing electronic information and may be semiconductor memory. The semiconductor memory may be either volatile memory or non-volatile memory. The memory device for saving various data, etc. in each device (information processing device 1) in the above-described embodiments may be realized by the main memory device 72 or the auxiliary memory device 73, or may be realized by an internal memory built into the processor 71. For example, the memory unit 12 in the above-described embodiment may be realized by the main memory device 72 or the auxiliary memory device 73.

[0169] When each device (information processing device 1) in the above-described embodiment is configured with at least one storage device (memory) and at least one processor connected (coupled) to this at least one storage device, at least one processor may be connected to one storage device. Also, at least one storage device may be connected to one processor. Also, a configuration in which at least one processor among multiple processors is connected to at least one storage device among multiple storage devices may be included. Also, this configuration may be realized by storage devices and processors included in multiple computers. Furthermore, a configuration in which a storage device is integrated with a processor (for example, a cache memory including an L1 cache and an L2 cache) may be included.

[0170] The network interface 74 is an interface for connecting to the communication network 8 wirelessly or via a wire. The network interface 74 may be an appropriate interface, such as one that conforms to an existing communication standard. The network interface 74 may exchange information with an external device 9A connected via the communication network 8. The communication network 8 may be any one of a WAN (Wide Area Network), a LAN (Local Area Network), a PAN (Personal Area Network), etc., or a combination thereof, as long as information is exchanged between the computer 7 and the external device 9A. An example of a WAN is the Internet, an example of a LAN is IEEE 802.11 or Ethernet (registered trademark), and an example of a PAN is Bluetooth (registered trademark) or NFC (Near Field Communication), etc.

[0171] The device interface 75 is an interface such as USB that directly connects to the external device 9B.

[0172] The external device 9A is a device connected to the computer 7 via a network. The external device 9B is a device directly connected to the computer 7.

[0173] For example, the external device 9A or the external device 9B may be an input device. The input device may be a device such as a camera, a microphone, a motion capture device, various sensors, a keyboard, a mouse, or a touch panel, and provides acquired information to the computer 7. Alternatively, the external device 9A or the external device 9B may be a device equipped with an input unit, a memory, and a processor, such as a personal computer, a tablet terminal, or a smartphone.

[0174] Furthermore, the external device 9A or the external device 9B may be, for example, an output device. The output device may be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) panel, or a speaker that outputs sound or the like. Alternatively, the external device 9A or the external device 9B may be a device including an output unit, a memory, and a processor, such as a personal computer, a tablet terminal, or a smartphone.

[0175] Furthermore, the external device 9A or the external device 9B may be a storage device (memory). For example, the external device 9A may be a network storage or the like, and the external device 9B may be a storage device such as an HDD.

[0176] Furthermore, the external device 9A or the external device 9B may be a device having some of the functions of the components of each device (information processing device 1) in the above-described embodiment. That is, the computer 7 may transmit some or all of the processing results to the external device 9A or the external device 9B, or may receive some or all of the processing results from the external device 9A or the external device 9B.

[0177] In this specification (including the claims), when the expression "at least one of a, b, and c" or "at least one of a, b, or c" (including similar expressions) is used, it includes any of a, b, c, a-b, a-c, b-c, or a-b-c. It may also include multiple instances of any element, such as a-a, a-b-b, a-a-b-b-c-c, etc. Furthermore, it also includes the addition of elements other than the listed elements (a, b, and c), such as having d, as in a-b-c-d.

[0178] In this specification (including claims), when expressions such as "using / using data as input / based on / according to / in response to" (including similar expressions) are used, unless otherwise specified, this includes cases where the data itself is used, or where data that has been processed in some way (e.g., data with noise added, normalized data, features extracted from data, intermediate representations of data, etc.) is used. Furthermore, when a statement is made that a result is obtained "using data as input / based on / according to / in response to" (including similar expressions), this includes cases where the result is obtained based solely on the data, or where the result is influenced by other data, factors, conditions, and / or states other than the data, unless otherwise specified. Furthermore, when a statement is made that "data is output" (including similar expressions), this includes cases where the data itself is used as output, or where data that has been processed in some way (e.g., data with noise added, normalized data, features extracted from data, intermediate representations of data, etc.) is used as output, unless otherwise specified.

[0179] When the terms "connected" and "coupled" are used in this specification (including the claims), they are intended as open-ended terms that include any of direct connection / coupling, indirect connection / coupling, electrically connection / coupling, communicatively connection / coupling, functionally connection / coupling, and physically connection / coupling. These terms should be interpreted appropriately according to the context in which they are used, but any connection / coupling form that is not intentionally or naturally excluded should be interpreted as being included in these terms without limitation.

[0180] In this specification (including the claims), the expression "A configured to B" may include the physical structure of element A having a configuration capable of performing operation B, and the permanent or temporary setting / configuration of element A being configured / set to actually perform operation B. For example, if element A is a general-purpose processor, it is sufficient that the processor has a hardware configuration capable of performing operation B, and is configured to actually perform operation B by setting a permanent or temporary program (instruction). Also, if element A is a dedicated processor or dedicated arithmetic circuit, it is sufficient that the circuit structure of the processor is implemented to actually perform operation B, regardless of whether control instructions and data are actually attached.

[0181] When used in this specification (including the claims), terms implying containing or possessing (e.g., "comprising / including" and "having") are intended to be open-ended terms, including cases where something other than the object indicated by the object of the term is contained or possessed. When the object of such a term implies no quantity or a singular number (e.g., an article such as "a" or "an"), the expression should be construed as not being limited to a specific number.

[0182] In this specification (including the claims), even if expressions such as "one or more" or "at least one" are used in some places and expressions that do not specify a quantity or that imply a singular number (expressions using the articles a or an) are used in other places, the latter expressions are not intended to mean "one." In general, expressions that do not specify a quantity or that imply a singular number (expressions using the articles a or an) should be interpreted as not necessarily being limited to a specific number.

[0183] In this specification, when a particular advantage / result is described as being obtained with respect to a particular configuration of an embodiment, it should be understood that the same advantage / result can also be obtained with one or more other embodiments having the same configuration, unless otherwise stated. However, it should be understood that the presence or absence of the effect generally depends on various factors, conditions, and / or circumstances, and that the effect is not necessarily obtained with the configuration. The effect is obtained only by the configuration described in the embodiment when various factors, conditions, and / or circumstances are satisfied, and the effect does not necessarily occur in a claimed invention that defines the same or a similar configuration.

[0184] When terms such as "maximize" and "maximization" are used in this specification (including the claims), they include finding a global maximum, finding an approximation of a global maximum, finding a local maximum, and finding an approximation of a local maximum, and should be interpreted appropriately according to the context in which the terms are used. They also include finding approximations of these maxima probabilistically or heuristically. Similarly, when terms such as "minimize" and "minimization" are used, they include finding a global minimum, finding an approximation of a global minimum, finding a local minimum, and finding an approximation of a local minimum, and should be interpreted appropriately according to the context in which the terms are used. They also include finding approximations of these minima probabilistically or heuristically. Similarly, when terms such as "optimize" and "optimization" are used, they include finding a global optimum, finding an approximation of a global optimum, finding a local optimum, and finding an approximation of a local optimum, and should be interpreted appropriately according to the context in which the terms are used. It also includes finding approximations of these optimum values ​​probabilistically or heuristically.

[0185] In this specification (including claims), when multiple pieces of hardware perform a predetermined process, the pieces of hardware may cooperate to perform the predetermined process, or some of the hardware may perform all of the predetermined process. Furthermore, some of the hardware may perform part of the predetermined process, and other hardware may perform the rest of the predetermined process. In this specification (including claims), when an expression such as "one or more pieces of hardware perform a first process, and the one or more pieces of hardware perform a second process" (including similar expressions) is used, the hardware performing the first process and the hardware performing the second process may be the same or different. In other words, it is sufficient that the hardware performing the first process and the hardware performing the second process are included in the one or more pieces of hardware. Note that the hardware may include an electronic circuit, a device including an electronic circuit, or the like.

[0186] In this specification (including the claims), when multiple storage devices (memories) store data, each of the multiple storage devices may store only a portion of the data, or may store the entire data. Also, a configuration in which only some of the multiple storage devices store data may be included.

[0187] Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the individual embodiments described above. Various additions, modifications, substitutions, and partial deletions are possible within the scope of the conceptual idea and spirit of the present disclosure, which is derived from the content defined in the claims and their equivalents. For example, when numerical values ​​or formulas are used in the above-described embodiments, they are shown for illustrative purposes and do not limit the scope of the present disclosure. Furthermore, the order of each operation shown in the embodiments is also illustrative and does not limit the scope of the present disclosure.< / st> < / st> < / lf> < / st:eos> < / st> < / st:eos> < / st> < / lf> < / st:eos> < / st> < / lf> < / eos> < / bos> < / eos> < / bos> < / eos> < / bos> < / eos> < / bos> < / lf> < / eos> < / bos> < / st:lf> < / st:lf> < / st> < / st> < / st:lf> < / st> < / st> < / bos> < / st> < / bos> < / st> < / bos>

Claims

1. An information processing device that trains a generative model based on a token sequence obtained by tokenizing data included in a dataset, the information processing device comprising: one or more memories; and one or more processors, wherein when a token sequence obtained from second data is used to train the generative model, the one or more processors tokenize the second data according to a tokenization rule that differs from the tokenization rule for obtaining the token sequence used to train the generative model from first data.

2. The information processing device according to claim 1, wherein the tokenization rules include any one of a rule regarding a start token of a token string, a rule regarding tokenization of specific data, or a rule regarding an end token of a token string.

3. The information processing device according to claim 1, wherein the one or more processors define a first token in a vocabulary that is set as a starting token, and when data included in a first dataset is used to train the generative model, the processor trains the generative model based on a token sequence in which the first token is set as a starting token, unlike when data included in a second dataset is used to train the generative model.

4. The information processing device of claim 1, wherein the one or more processors define a first token and a second token different from the first token in a vocabulary, train the generative model based on a token sequence in which the first token is set as a starting token when data included in a first dataset is used to train the generative model, and train the generative model based on a token sequence in which the second token is set as a starting token when data included in a second dataset is used to train the generative model.

5. The information processing device according to claim 4, wherein the one or more processors are configured to remove the second token from the vocabulary at least after training using the second dataset is completed.

6. The information processing device according to claim 4, wherein the one or more processors, after completing at least training using the second dataset, delete an element that converts the second token into a vector from a function that converts data included in the vocabulary into a vector.

7. The information processing device according to claim 5, wherein the one or more processors delete information relating to the second token in the network relating to the generative model after training using at least the second dataset is completed.

8. The information processing device according to claim 6, wherein the function is an embedding layer, and the one or more processors delete a row or column relating to the second token from a matrix representing the embedding layer.

9. The information processing device according to claim 6, wherein the function is a lookup table, and the one or more processors delete a row or column related to the conversion of the second token from the lookup table.

10. The information processing device of claim 4, wherein the generative model is a model that has been trained in advance using at least the first dataset and the first token, and the one or more processors at least: define the second token in the vocabulary; extract training data from the second dataset; and tune the generative model using the second token.

11. The information processing device according to claim 4, wherein the second token is empty.

12. The information processing device of claim 4, wherein the one or more processors convert each of one or more tokens in the second dataset into a special token, and train the generative model using the special token.

13. The information processing device according to claim 12, wherein the one or more tokens include a token that is more likely to be present in the second data set than other tokens.

14. The information processing device according to claim 12, wherein the one or more tokens include at least one of a comma, a period, a comma, a period, or a line feed code.

15. The information processing device of claim 12, wherein the one or more processors remove the special token from the vocabulary after training using the second dataset is completed.

16. An information processing device that causes the generative model obtained by training according to claim 1 to generate data, comprising: one or more memories; and one or more processors, wherein the one or more processors cause the trained generative model to generate data based on a token sequence obtained by tokenizing data, and the tokenization rule is selected from a plurality of tokenization rules.

17. An information processing device according to any one of claims 1 to 16, wherein the token sequence obtained by tokenizing data is a sequence of a plurality of components obtained by dividing the data into the components of the data.

18. An information processing device according to any one of claims 1 to 16, wherein the generative model is a language model that generates data based on a token sequence.

19. An information processing device according to any one of claims 1 to 16, wherein the first data is data included in a first data set, the second data is data included in a second data set, and the second data set is a data set that is more confidential than the first data set.

20. The information processing device described in claim 19, wherein the tokenization rules for using the generative model trained using the first data and the second data are the same as the tokenization rules when training the generative model using the first data.

Images