Control program, control method, and information processing device

Abstract

Controlling the operation of the intermediate representation in a direction that yields a reasonable output. [Solution] The intermediate representation generation unit 603 generates an intermediate representation corresponding to the input data using a trained model M. The distribution learning unit 604 learns the distribution of latent representations corresponding to the generated intermediate representations according to a predetermined encoder E and a predetermined decoder D. The sample selection unit 605 selects a latent representation sample according to the probability distribution from the learned distribution. The reconstruction unit 606 generates a new intermediate representation corresponding to the selected sample using a predetermined decoder D. The output generation unit 607 generates output data corresponding to the newly generated intermediate representation using the trained model M.

Description

[Technical Field]

[0001] The present invention relates to a control program, a control method, and an information processing device. [Background technology]

[0002] Traditionally, deep learning models that handle sequential information, such as Transformer models, have demonstrated high performance in understanding input information and predicting structured outputs (see, for example, Non-Patent Document 1 below). Furthermore, large-scale pre-trained models with high expressive power, such as AlphaFold2 for protein structure prediction, are known (see, for example, Non-Patent Document 2 below). There are cases where it is desirable to leverage the expressive power of such pre-trained models and generate diverse, high-quality outputs without changing the parameters of the pre-trained models. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Vaswani, Ashish, et al. “Attention is all you need.” Advances in neural information processing systems 30 (2017). [Non-Patent Document 2] Jumper, John, et al. “Highly accurate protein structure prediction with AlphaFold.” Nature 596.7873 (2021): 583-589. [Overview of the project] [Problems that the invention aims to solve]

[0004] However, with conventional techniques, when attempting to modify the output by performing some operation on the intermediate representation of a trained model, the range of operations on the intermediate representation corresponding to a valid output is unclear, making it difficult to determine how to perform the operation to obtain a valid output.

[0005] In one aspect, the present invention aims to control the operation of the intermediate representation in a direction that yields a reasonable output. [Means for solving the problem]

[0006] According to one embodiment, a control program, control method, and information processing device are proposed that generate an intermediate representation corresponding to input data using a trained model, learn the distribution of latent representations corresponding to the generated intermediate representation according to a predetermined encoder that generates the latent representation from the intermediate representation and a predetermined decoder that corresponds to the predetermined encoder and generates an intermediate representation different from the intermediate representation, select a sample of the latent representation according to a probability distribution from the learned distribution, generate a new intermediate representation corresponding to the selected sample using the predetermined decoder, and generate output data corresponding to the generated new intermediate representation using the trained model. [Effects of the Invention]

[0007] According to one embodiment, it becomes possible to control the operation of the intermediate representation in a direction that yields a reasonable output. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is an explanatory diagram showing one embodiment of the control method according to the embodiment. [Figure 2] Figure 2 is an explanatory diagram showing an example of the system configuration of the information processing system 200. [Figure 3] Figure 3 is a block diagram showing an example of the hardware configuration of the control device 201. [Figure 4] Figure 4 is an explanatory diagram showing a specific example of input data 400. [Figure 5]Figure 5 is an explanatory diagram showing a specific example of intermediate representation 500. [Figure 6] Figure 6 is a block diagram showing an example of the functional configuration of the control device 201. [Figure 7] Figure 7 is an explanatory diagram (part 1) showing a first example of operation of the control device 201. [Figure 8] Figure 8 is an explanatory diagram (part 2) showing a first example of operation of the control device 201. [Figure 9] Figure 9 is an explanatory diagram (part 1) showing a second example of operation of the control device 201. [Figure 10] Figure 10 is an explanatory diagram (part 2) showing a second example of the operation of the control device 201. [Figure 11] Figure 11 is an explanatory diagram (part 3) showing a second example of operation of the control device 201. [Figure 12] Figure 12 is an explanatory diagram (part 4) showing a second example of operation of the control device 201. [Figure 13] Figure 13 is an explanatory diagram (part 1) showing a third example of operation of the control device 201. [Figure 14] Figure 14 is an explanatory diagram (part 2) showing a third example of operation of the control device 201. [Figure 15] Figure 15 is an explanatory diagram showing a fourth example of operation of the control device 201. [Figure 16] Figure 16 is a flowchart showing an example of the learning process procedure. [Figure 17] Figure 17 is a flowchart showing an example of the generation process procedure. [Modes for carrying out the invention]

[0009] Embodiments of the control program, control method, and information processing device according to the present invention will be described in detail below with reference to the drawings.

[0010] (An embodiment of the control method according to the embodiment) Figure 1 is an explanatory diagram showing one embodiment of the control method according to the embodiment. The information processing device 100 is a computer for controlling operations on intermediate representations corresponding to input data to a trained model. The information processing device 100 is, for example, a server or a PC (Personal Computer).

[0011] Here, a pre-trained model is a machine learning model that has been trained using machine learning methods such as deep learning. Deep learning is also called deep learning. A pre-trained model is information that combines, for example, pre-trained parameters with an algorithm for deriving output data that corresponds to input data based on those pre-trained parameters.

[0012] A trained model, following an algorithm, accepts input data and derives output data by applying trained parameters to the input data. Specifically, a trained model includes an encoder that converts the input data into an intermediate representation and a decoder that converts the intermediate representation into output data. Specifically, the trained model generates output data by converting the input data into an intermediate representation using the encoder and then converting the converted intermediate representation into output data using the decoder.

[0013] An intermediate representation is information obtained by extracting features from input data. For example, by extracting features from sequential input data, an intermediate representation, which is a vector sequence, can be obtained. Operations on the intermediate representation are performed to obtain new output data for the input data. For example, one can manipulate the intermediate representation by adding a small value to the vector sequence to change the values ​​of the vector sequence.

[0014] Examples of pre-trained models include the Transformer model and AlphaFold2. The Transformer model takes sequence information representing a sentence as input data and outputs sequence information representing another sentence as output data. AlphaFold2 takes amino acid sequence information as input data and outputs data representing the structure (three-dimensional structure) of a protein. The Transformer model and AlphaFold2 are large-scale deep learning models with high expressive power.

[0015] For information on the Transformer model, see, for example, Non-Patent Document 1 mentioned above or Reference 1 below. For information on AlphaFold2, see, for example, Non-Patent Document 2 mentioned above.

[0016] Reference 1: Radford, Alec, et al. “Language models are unsupervised multitask learners.” OpenAI blog 1.8 (2019): 9.

[0017] There is a desire to leverage the expressive capabilities of such pre-trained models and generate high-quality, diverse outputs without modifying the parameters of the pre-trained models. For example, one might want to enumerate the multiple forms that a given input sequence can take using AlphaFold2. Another might want to generate diverse sentences using a Transformer model for text generation.

[0018] Here, the intermediate representation of a large-scale model abstractly captures important features of the data and is expected to be suitable for making meaningful changes to the output while preserving the essence of the input data. For this reason, it is conceivable to generate new output data for the input data by manipulating the intermediate representation of a trained model.

[0019] However, there is a problem in that it is difficult to manipulate the intermediate representation in a way that yields a valid output. For example, the Transformer model cannot explicitly calculate the probability distribution of the intermediate representation. Therefore, when attempting to modify the output by applying some operation to the intermediate representation, the range of operations on the intermediate representation that will yield a valid output based on the data distribution of the training data used during training of the trained model is not clear. Consequently, it is not clear what range of operations should be applied to the intermediate representation to obtain a valid output, making it difficult to manipulate the intermediate representation appropriately. Here, we will explain the range of operations on the intermediate representation using Figure 1.

[0020] In Figure 1, the black circle p1 represents the intermediate representation transformed from the input data. Regions R1 and R2 both represent the regions that the intermediate representation can take. Region R1 represents, for example, a region that has been generalized to some extent through learning. Region R2 represents, for example, a region that contains only intermediate representations corresponding to valid outputs. Region R2 corresponds, for example, to the distribution of intermediate representations corresponding to the input data, which is the various training data used during the training of the trained model.

[0021] Region R1 includes intermediate representations corresponding to invalid outputs, so while it may be possible to obtain a valid output, it may also be possible to obtain an invalid output. Therefore, in order to obtain a valid output, it is preferable to control the operations on the intermediate representation so that it falls within region R2. However, since region R2 cannot be explicitly acquired, it is not clear how to specifically control the operations on the intermediate representation to obtain a valid output.

[0022] For example, manipulating an intermediate representation may cause it to extend beyond the region R2, resulting in an intermediate representation that falls outside the data distribution of the training data used to train the model. In this case, the output estimated based on the intermediate representation may be unreliable because it deviates from the data distribution of the training data.

[0023] Therefore, in this embodiment, we will describe a control method that can control operations on the intermediate representation in a direction that yields a valid output.

[0024] In Figure 1, the information processing device 100 has a trained model 110. The trained model 110 includes an encoder 111 that converts input data into an intermediate representation and a decoder 112 that converts the intermediate representation into output data. In the example in Figure 1, the input data to the trained model 110 is referred to as "input data 101". For example, if the trained model 110 is "AlphaFold2", the encoder 111 corresponds to a Transformer encoder. The decoder 112 also corresponds to a Transformer decoder.

[0025] Furthermore, the information processing device 100 has a predetermined encoder 121. The predetermined encoder 121 has, for example, the function of converting an intermediate representation to a latent representation. The predetermined encoder 121 is, for example, an encoder in the VAE (Variational AutoEncoder) method. The information processing device 100 also has a predetermined decoder 122 corresponding to the predetermined encoder 121. The predetermined decoder 122 has the function of converting an input latent representation to an intermediate representation. The predetermined decoder 122 is, for example, a decoder in the VAE method.

[0026] The information processing device 100 generates new output data 106 corresponding to the input data 101 by manipulating the intermediate representation 102 corresponding to the input data 101, as shown below. Specifically, the operation includes generating another intermediate representation 105 based on the intermediate representation 102 corresponding to the input data 101. The output data 106 is different from the output data obtained by directly converting the intermediate representation 102 with the decoder 112.

[0027] (1-1) The information processing device 100 generates an intermediate representation 102 corresponding to the input data 101 using the trained model 110. The input data 101 is, for example, the training data used when training the trained model 110. Specifically, the information processing device 100 generates an intermediate representation 102 corresponding to the input data 101 by converting the input data 101 with the encoder 111.

[0028] (1-2) The information processing device 100 learns a distribution 103 of latent representations corresponding to the generated intermediate representation 102, according to a predetermined encoder 121 and a predetermined decoder 122. For example, the information processing device 100 learns a distribution 103 that is the probability distribution of latent representations onto which the intermediate representation 102 is projected, according to a predetermined encoder 121 and a predetermined decoder 122. Specifically, the distribution 103 represents the probability that each of the multiple latent representations is possible. As a result, the information processing device 100 can obtain a distribution 103 of latent representations corresponding to the intermediate representation 102 that corresponds to a reasonable range for manipulating the intermediate representation 102.

[0029] (1-3) The information processing device 100 selects a latent representation sample 104 from the learned distribution 103. The information processing device 100 selects a latent representation sample 104 from the learned distribution 103 based on a probability distribution, for example. The sample 104 is obtained, for example, by sampling data according to a probability distribution. In this way, the information processing device 100 can obtain a latent representation sample 104 that will be the basis for a new valid intermediate representation 105.

[0030] (1-4) The information processing device 100 generates a new intermediate representation 105 corresponding to the selected sample 104 using a predetermined decoder 122. The information processing device 100 generates a new intermediate representation 105 corresponding to the selected sample 104 by, for example, converting the selected sample 104 with a predetermined decoder 122. This allows the information processing device 100 to perform operations on the intermediate representation 102 within an operating range based on the distribution 103 of latent representations, thereby obtaining a valid new intermediate representation 105.

[0031] (1-5) The information processing device 100 generates output data 106 corresponding to the newly generated intermediate representation 105 using the trained model 110. The information processing device 100 generates output data 106 corresponding to the newly generated intermediate representation 105 by, for example, converting the newly generated intermediate representation 105 with the decoder 112. In this way, the information processing device 100 can obtain valid output data 106.

[0032] Thus, the information processing device 100 can control operations on the intermediate representation 102 in a direction that yields valid output data 106. For example, the information processing device 100 can apply operations to the intermediate representation 102 within an operation range based on the distribution 103 of latent representations, thereby obtaining a new valid intermediate representation 105 and thus valid output data 106. Therefore, the information processing device 100 can obtain valid output data 106 more efficiently than when random operations are applied to the intermediate representation 102.

[0033] Here, we have described the case where the functions of the information processing device 100 are realized by a single computer, but this is not the only case. For example, the functions of the information processing device 100 may be realized through the collaboration of multiple computers. For example, the functions of the information processing device 100 may be realized on the cloud.

[0034] (Example of system configuration for information processing system 200) Next, we will describe an example of the system configuration of the information processing system 200, which includes the information processing device 100 shown in Figure 1. Here, we will explain using the case where the information processing device 100 shown in Figure 1 is applied to the control device 201 within the information processing system 200 as an example.

[0035] Figure 2 is an explanatory diagram showing an example of the system configuration of the information processing system 200. In Figure 2, the information processing system 200 includes a control device 201 and a client device 202. In the information processing system 200, the control device 201 and the client device 202 are connected via a wired or wireless network 210. The network 210 is, for example, the Internet, a LAN (Local Area Network), or a WAN (Wide Area Network).

[0036] Here, the control device 201 is a computer that controls operations on the intermediate representation corresponding to the input data to the trained model 220. Specific examples of input data will be described later using Figure 4. Specific examples of intermediate representations will be described later using Figure 5.

[0037] The control device 201 has a trained model 220. The trained model 220 is, for example, a trained deep learning model such as a Transformer model or AlphaFold2. The trained model 220 includes an encoder 221 and a decoder 222. The control device 201 may, for example, train a deep learning model that will become the trained model 220 using training data. The control device 201 also has a VAE encoder 231 and a VAE decoder 232.

[0038] The trained model 110 shown in Figure 1 corresponds to, for example, the trained model 220. The encoder 111 shown in Figure 1 corresponds to, for example, the encoder 221. The decoder 112 shown in Figure 1 corresponds to, for example, the decoder 222. The predetermined encoder 121 shown in Figure 1 corresponds to, for example, the VAE encoder 231. The predetermined decoder 122 shown in Figure 1 corresponds to, for example, the VAE decoder 232.

[0039] The control device 201 receives a processing request from the client device 202 that requests the generation of various output data based on input data. The processing request includes, for example, input data. The input data is, for example, the training data used when training the trained model 220. In response to the processing request, the control device 201 uses the trained model 220, the encoder 221, and the decoder 222 to generate multiple output data based on the input data. For example, the control device 201 uses the trained model 220 and the input data to learn the distribution of latent representations in accordance with the VAE encoder 231 and the VAE decoder 232, thereby generating multiple output data. The control device 201 transmits the generated multiple output data to the client device 202. The control device 201 is, for example, a server or a PC.

[0040] The client device 202 is a computer used by a user of the information processing system 200. The user is someone who, for example, wishes to predict the structure of a protein from an amino acid sequence or generate another sentence from another sentence. The other sentence could be, for example, a translated sentence. Based on user input via an input device (not shown), the client device 202 generates a processing request and transmits it to the control device 201. The client device 202 receives multiple output data from the control device 201 and outputs them for the user to refer to. The client device 202 could be, for example, a PC, a tablet, or a smartphone.

[0041] This explanation describes a case where the control device 201 and the client device 202 are different devices, but it is not limited to this case. For example, the control device 201 may have the functionality of a client device 202 and may operate as a client device 202. The information processing system 200 may include multiple client devices 202.

[0042] (Examples of applications of Information Processing System 200) The information processing system 200 can be applied, for example, to cases where it is desired to present the user with multiple output data representing the structure of a protein based on input data representing an amino acid sequence. Furthermore, the information processing system 200 can be applied, for example, to cases where it is desired to present the user with multiple output data representing a translated sentence corresponding to an input data representing a sentence.

[0043] (Example hardware configuration of control device 201) Next, we will describe an example of the hardware configuration of the control device 201.

[0044] Figure 3 is a block diagram showing an example of the hardware configuration of the control device 201. In Figure 3, the control device 201 includes a CPU (Central Processing Unit) 301, memory 302, disk drive 303, and disk 304. The control device 201 also includes a communication interface 305, a graphics processing unit (GPU) 306, a portable recording medium interface 307, and a portable recording medium 308. Each component is connected by a bus 300.

[0045] Here, the CPU 301 is responsible for the overall control of the control unit 201. The GPU 306 performs computational processing such as image processing and natural language processing. The CPU 301 and GPU 306 may have multiple cores. The memory 302 may include, for example, ROM (Read Only Memory) or RAM (Random Access Memory). Programs stored in memory 302 are loaded into the CPU 301, causing the CPU 301 to execute the coded processes.

[0046] The disk drive 303 controls the reading and writing of data to the disk 304 according to the control of the CPU 301. The disk 304 stores the data written under the control of the disk drive 303. The disk 304 is, for example, a magnetic disk, an optical disk, etc.

[0047] The communication interface 305 is connected to the network 210 via a communication line, and through the network 210, it is connected to an external computer. The external computer is, for example, the client device 202 shown in Figure 2. The communication interface 305 manages the interface between the network 210 and the inside of the device, and controls the input and output of data from the external computer. The communication interface 305 is, for example, a modem or a LAN adapter.

[0048] The portable recording medium interface 307 controls the reading and writing of data to the portable recording medium 308 according to the control of the CPU 301. The portable recording medium 308 stores the data written under the control of the portable recording medium interface 307. The portable recording medium 308 is, for example, a CD (Compact Disc)-ROM, a DVD (Digital Versatile Disk), or a USB (Universal Serial Bus) memory.

[0049] In addition to the components described above, the control device 201 may also have, for example, an input device, a display, a printer, a scanner, a microphone, or a speaker. Furthermore, the control device 201 may not have, for example, the GPU 306, the portable recording medium I / F 307, or the portable recording medium 308 among the components described above.

[0050] (Example hardware configuration for client device 202) The hardware configuration example of the client device 202 is specifically the same as the hardware configuration example of the control device 201 shown in Figure 3, so a detailed explanation is omitted. In addition to the components shown in Figure 3, the client device 202 may also have, for example, an input device, a display, etc.

[0051] (Specific example of input data 400) Next, using Figure 4, we will explain a specific example of the input data 400 that is input to the trained model 220 shown in Figure 2. Here, we will take the case where the trained model 220 is a "Transformer model" as an example, and assume that sequence information representing a sentence is input to the trained model 220 as input data 400.

[0052] Figure 4 is an explanatory diagram showing a specific example of input data 400. In Figure 4, input data 400 is sequence information showing a sequence of token IDs representing a sentence. A token corresponds to a division of a sentence (text) into units such as words, subwords, or symbols. A token ID is an identifier that identifies a token.

[0053] The input data 400 corresponds to the input text "it's a charming and often affecting journey." after undergoing preprocessing such as tokenization. Preprocessing includes, for example, replacing units such as words, subwords, or symbols with token IDs. Preprocessing may be performed, for example, on the control device 201, or on another computer different from the control device 201. The other computer may be, for example, the client device 202. In the following description, the number of token IDs will be referred to as the "length T of the sequence information".

[0054] (500 specific examples of intermediate expressions) Next, using Figure 5, we will explain a specific example of an intermediate representation 500 obtained by transforming the input data 400 input to the trained model 220 shown in Figure 2 using the encoder 221.

[0055] Figure 5 is an explanatory diagram showing a specific example of the intermediate representation 500. In Figure 5, the intermediate representation 500 is information converted from the input data 400 shown in Figure 4 by extracting features from the input data 400 using the encoder 221. The intermediate representation 500 consists of vectors v1~v corresponding to each token ID, with a length T of sequence information. T This is a vector sequence formed by arranging the vectors v1~v TH is a d-dimensional vector. i,j is the i-th vector v i It is the j-th component.

[0056] Although not shown in the diagram, for example, if the trained model 220 is "AlphaFold2", the input data will be amino acid sequence information. The intermediate representation will consist of a single representation and a pair representation. The single representation is a vector sequence. The pair representation is T×T×d dimensional sequence information that represents the similarity between the vector sequences.

[0057] (Example of functional configuration of control device 201) Next, an example of the functional configuration of the control device 201 will be described using Figure 6.

[0058] Figure 6 is a block diagram showing an example of the functional configuration of the control device 201. The control device 201 includes a storage unit 600, an acquisition unit 601, a model learning unit 602, an intermediate representation generation unit 603, a distribution learning unit 604, a sample selection unit 605, a reconstruction unit 606, an output generation unit 607, and an output unit 608.

[0059] The storage unit 600 is implemented by a storage device such as the memory 302 and disk 304 shown in Figure 3. The following description will focus on the case where the storage unit 600 is included in the control device 201, but is not limited to this. For example, the storage unit 600 may be included in an external device different from the control device 201. In this case, for example, the contents of the storage unit 600 may be accessible from the control device 201 via the network 210.

[0060] The acquisition unit 601 to the output unit 608 function as an example of a control unit. Specifically, the acquisition unit 601 to the output unit 608 realize their functions by, for example, causing the CPU 301 to execute a program stored in a storage device such as memory 302, disk 304, or portable recording medium 308, or by using a communication interface 305 or GPU 306. The processing results of each functional unit are stored in a storage device such as memory 302 or disk 304 as shown in Figure 3.

[0061] The memory unit 600 stores various information that is referenced or updated in the processing of each functional unit. The memory unit 600 stores, for example, a trained model. The trained model M includes, for example, an encoder M_E that converts input data into an intermediate representation and a decoder M_D that converts the intermediate representation into output data. Specifically, the trained model M may be a deep learning model that takes an amino acid sequence as input data and outputs output data representing the structure of a protein. Specifically, the trained model M may be a model that takes sequence information representing a sentence as input data and outputs sequence information representing another sentence as output data. The trained model M is acquired, for example, by the acquisition unit 601. The trained model M is generated, for example, by the model learning unit 602.

[0062] The memory unit 600 stores, for example, a predetermined encoder E that generates a latent representation and a predetermined decoder D corresponding to the predetermined encoder E. The predetermined encoder E converts an intermediate representation to a latent representation. The predetermined encoder E is an encoder in the VAE method. The predetermined decoder D converts a latent representation to an intermediate representation. The predetermined decoder D is a decoder in the VAE method. A template of the predetermined encoder E and a template of the predetermined decoder D are acquired, for example, by the acquisition unit 601. The predetermined encoder E and the predetermined decoder D are learned, for example, by the distribution learning unit 604.

[0063] Specific patterns of combinations between a given encoder E and a given decoder D include, for example, the first, second, or third patterns shown below. In the first pattern, the given encoder E includes a first generator that generates a first vector by fully connecting multiple vectors representing the intermediate representation and then multiplying them by a first weight matrix, and a first multilayer perceptron that generates a latent representation corresponding to the generated first vector. In the first pattern, the given decoder D includes a second multilayer perceptron that generates a second vector corresponding to the latent representation, and a second generator that reconstructs multiple vectors representing the intermediate representation from a third vector obtained by multiplying the generated second vector by a second weight matrix.

[0064] In the second pattern, a given encoder E is a model that generates a latent representation by repeatedly performing a convolution operation on an intermediate representation and identifies parameters that represent the distribution of the latent representation. In the second pattern, a given decoder D is a model that generates an intermediate representation by repeatedly performing a deconvolution operation on the latent representation.

[0065] In the third pattern, a predetermined encoder E includes performing multiple convolution operations on an intermediate representation. The predetermined encoder E is a model that identifies a first parameter representing the distribution of each of the multiple hierarchical latent representations according to the result of each convolution operation. In the third pattern, a predetermined decoder D includes performing a deconvolution operation on each of the multiple latent representations. The predetermined decoder D is a model that fixes a second parameter representing the distribution of the lowest-level latent representation and identifies a second parameter representing the distribution of each of the latent representations other than the lowest-level one.

[0066] The acquisition unit 601 acquires various types of information used in the processing of each functional unit. The acquisition unit 601 stores the acquired information in the storage unit 600 or outputs it to each functional unit. The acquisition unit 601 may also output the information stored in the storage unit 600 to each functional unit. The acquisition unit 601 acquires various types of information, for example, based on user input. The acquisition unit 601 may also receive various types of information from a device other than the control device 201, for example.

[0067] The acquisition unit 601 acquires, for example, a processing request that requests the generation of output data. The processing request includes, for example, input data. The input data is, for example, training data used during the training of a trained model. The processing request may also include a trained model M. The processing request may also include a template for a predetermined encoder E and a template for a predetermined decoder D. Specifically, the acquisition unit 601 acquires a processing request by receiving it from another computer. The other computer is, for example, a client device 202. Specifically, the acquisition unit 601 may acquire a processing request by accepting input of a processing request based on user operation input via an input device (not shown).

[0068] The acquisition unit 601 acquires input data, for example. Specifically, the acquisition unit 601 acquires input data by extracting it from a processing request. Specifically, the acquisition unit 601 may acquire input data by receiving it from another computer. The other computer may be, for example, a client device 202. Specifically, the acquisition unit 601 may acquire input data by accepting input data based on user operation input via an input device (not shown).

[0069] The acquisition unit 601 acquires, for example, a trained model M. Specifically, the acquisition unit 601 acquires the trained model M by extracting it from a processing request. Specifically, the acquisition unit 601 may acquire the trained model M by receiving it from another computer. The other computer is, for example, a client device 202. Specifically, the acquisition unit 601 may acquire the trained model M by receiving input of the trained model M based on user operation input via an input device (not shown).

[0070] The acquisition unit 601 acquires, for example, a template for a predetermined encoder E and a template for a predetermined decoder D. Specifically, the acquisition unit 601 acquires a template for a predetermined encoder E and a template for a predetermined decoder D by extracting them from a processing request. Specifically, the acquisition unit 601 may acquire a template for a predetermined encoder E and a template for a predetermined decoder D by receiving them from another computer. The other computer is, for example, a client device 202. Specifically, the acquisition unit 601 may acquire a template for a predetermined encoder E and a template for a predetermined decoder D by receiving input of a template for a predetermined encoder E and a template for a predetermined decoder D based on user operation input via an input device (not shown).

[0071] The acquisition unit 601 may receive a start trigger to initiate processing in any of the functional units. A start trigger may be, for example, a predetermined operation input by a user. A start trigger may also be, for example, the receipt of predetermined information from another computer. A start trigger may also be, for example, the output of predetermined information by any of the functional units. The acquisition unit 601 may, for example, accept the acquisition of a processing request as a start trigger to initiate processing in the model learning unit 602, the intermediate representation generation unit 603, the distribution learning unit 604, the sample selection unit 605, the reconstruction unit 606, and the output generation unit 607.

[0072] The model learning unit 602 generates a trained model M. The model learning unit 602 generates a trained model M based on training data, for example. This allows the model learning unit 602 to generate output data even when an externally trained model M is not available.

[0073] The intermediate representation generation unit 603 generates an intermediate representation corresponding to the input data using the trained model M. The intermediate representation generation unit 603 generates an intermediate representation corresponding to the input data, for example, by converting the input data into an intermediate representation using the encoder M_E. In this way, the intermediate representation generation unit 603 can obtain an intermediate representation in which the features of the input data have been extracted.

[0074] The distribution learning unit 604 learns the distribution of latent representations corresponding to the generated intermediate representations, according to a predetermined encoder E and a predetermined decoder D. The distribution learning unit 604 uses, for example, the training data used when training a trained model M as input data. The distribution learning unit 604 sets, for example, an objective function. The objective function includes, for example, the likelihood of the output data corresponding to the input data, the reconstruction error of the intermediate representation corresponding to the input data, and the KL divergence regarding the distribution of latent representations. The distribution learning unit 604 uses, for example, the objective function to learn the distribution of latent representations according to a predetermined encoder E and a predetermined decoder D.

[0075] For example, consider the case where a predetermined encoder E and a predetermined decoder D are in the first pattern. In this case, the distribution learning unit 604 specifically learns the distribution of latent representations corresponding to the generated intermediate representation by learning the predetermined encoder E and the predetermined decoder D using the objective function. As a result, the distribution learning unit 604 can obtain a distribution of latent representations corresponding to the intermediate representation that corresponds to a reasonable range for manipulating the intermediate representation.

[0076] Furthermore, for example, there may be a case where the predetermined encoder E and predetermined decoder D are of the second pattern. In this case, the distribution learning unit 604 specifically learns the predetermined encoder E and predetermined decoder D using the objective function and learns the parameters, thereby learning the distribution of latent representations corresponding to the generated intermediate representations. As a result, the distribution learning unit 604 can obtain a distribution of latent representations corresponding to intermediate representations that corresponds to a reasonable range for manipulating the intermediate representations.

[0077] Furthermore, for example, there is a case where a predetermined encoder E and a predetermined decoder D are in a third pattern. In this case, the distribution learning unit 604 specifically uses an objective function to learn a first parameter according to the predetermined encoder E based on the input data, and learns a second parameter according to the predetermined decoder D based on the first parameter. By learning the second parameter, the distribution learning unit 604 learns the distribution of each of the multiple latent representations represented by the second parameter. As a result, the distribution learning unit 604 can obtain a distribution of latent representations corresponding to intermediate representations that corresponds to a reasonable range for manipulating the intermediate representations.

[0078] The sample selection unit 605 selects a sample of latent representation from the learned distribution according to the probability distribution. For example, consider the case where a predetermined encoder E and a predetermined decoder D are in the first pattern. In this case, the sample selection unit 605 obtains a sample by sampling data according to the probability distribution shown in the learned distribution, for example. As a result, the sample selection unit 605 can obtain a sample of latent representation that will serve as the basis for a new, valid intermediate representation.

[0079] Furthermore, for example, there may be a case where a predetermined encoder E and a predetermined decoder D are in the second pattern. In this case, the sample selection unit 605 selects, for example, latent representations that exist within the range of ±1σ from the distribution of latent representations represented by the learned parameters as latent representation samples. As a result, the sample selection unit 605 can obtain latent representation samples that will serve as the basis for a new, valid intermediate representation.

[0080] Furthermore, for example, there may be a third pattern in which a predetermined encoder E and a predetermined decoder D are present. In this case, the sample selection unit 605 selects, for example, a latent representation of any hierarchy that lies within a range of ±1σ from the distribution of latent representations of any hierarchy represented by any of the learned second parameters, as a sample of the latent representation of that hierarchy. As a result, the sample selection unit 605 can obtain a sample of a latent representation that will serve as the basis for a new, valid intermediate representation.

[0081] The reconstruction unit 606 generates a new intermediate representation corresponding to the selected sample using a predetermined decoder D. For example, consider the case where a predetermined encoder E and a predetermined decoder D are in the first pattern. In this case, the reconstruction unit 606 generates a vector corresponding to the selected sample using, for example, a second multilayer perceptron in the predetermined decoder D. The reconstruction unit 606 then reconstructs multiple vectors representing the intermediate representation from the vector obtained by multiplying the generated vector by a second weight matrix using, for example, a second generator in the predetermined decoder D. This allows the reconstruction unit 606 to perform operations on the original intermediate representation within an operating range based on the distribution of latent representations, thereby obtaining a valid new intermediate representation.

[0082] Furthermore, for example, there may be a case where the predetermined encoder E and predetermined decoder D are in the second pattern. In this case, the reconstruction unit 606 generates a new intermediate representation by, for example, repeatedly performing a deconvolution operation on the selected sample with the predetermined decoder D. This allows the reconstruction unit 606 to perform operations on the original intermediate representation within an operating range based on the distribution of latent representations, thereby obtaining a valid new intermediate representation.

[0083] Furthermore, for example, there may be a third pattern in which a predetermined encoder E and a predetermined decoder D are present. In this case, the reconstruction unit 606 repeatedly obtains the latent representation of the next higher layer by performing a deconvolution operation on the latent representations in order from one of the selected layers, based on the latent representation of one of the layers. The reconstruction unit 606 generates a new intermediate representation by performing a deconvolution operation on the latent representation of the top layer, for example. This allows the reconstruction unit 606 to perform operations on the original intermediate representation within an operation range based on the distribution of latent representations, and to obtain a valid new intermediate representation.

[0084] The output generation unit 607 generates output data corresponding to the newly generated intermediate representation using the trained model M. The output generation unit 607 generates output data, for example, by converting the newly generated intermediate representation into output data using the trained model M's decoder M_D. This allows the output generation unit 607 to obtain a variety of valid output data.

[0085] The output unit 608 outputs the processing result of at least one of the functional units. The output format is, for example, display on a display, print output to a printer, transmission to an external device via the communication I / F 305, or storage in a storage device such as the memory 302 or disk 304 shown in Figure 3. This allows the output unit 608 to notify the user of the processing result of at least one of the functional units, thereby improving the usability of the control device 201.

[0086] The output unit 608 outputs, for example, the output data generated by the output generation unit 607. Specifically, the output unit 608 transmits the generated output data to another computer. The other computer is, for example, the client device 202. Specifically, the output unit 608 outputs the generated output data so that it can be viewed by the user. This allows the output unit 608 to make valid output data available externally.

[0087] Here, we have described the case in which the output generation unit 607 generates output data using the trained model M, but this is not the only case. For example, the output generation unit 607 may not use the trained model M when generating output data.

[0088] In this case, the storage unit 600 stores a first encoder E_1 that converts a vector into a latent representation, and a first decoder D_1 corresponding to the first encoder E_1. The first encoder E_1 may, for example, be included in the processing request. The first decoder D_1 may, for example, be included in the processing request. The first decoder D_1 converts the latent representation into a vector. The first encoder E_1 is, for example, an encoder in the VAE method. The first decoder D_1 is, for example, a decoder in the VAE method. The first encoder E_1 is acquired, for example, by the acquisition unit 601. The first decoder D_1 is acquired, for example, by the acquisition unit 601.

[0089] The memory unit 600 stores, for example, a first model M_1 that converts an intermediate representation into a vector. The first model M_1 may be included in a processing request, for example. The first model M_1 is, for example, an encoder in the AutoBot method. The first model M_1 is acquired, for example, by the acquisition unit 601. The memory unit 600 stores, for example, a second model M_2 that converts a vector into output data. The second model M_2 is, for example, a decoder in the AutoBot method. The second model M_2 is generated, for example, by the distribution learning unit 604.

[0090] The acquisition unit 601 acquires, for example, the first encoder E_1 and the first decoder D_1. Specifically, the acquisition unit 601 acquires the first encoder E_1 and the first decoder D_1 by extracting them from a processing request. Specifically, the acquisition unit 601 may acquire the first encoder E_1 and the first decoder D_1 by receiving them from another computer. The other computer is, for example, a client device 202. Specifically, the acquisition unit 601 may acquire the first encoder E_1 and the first decoder D_1 by receiving input for the first encoder E_1 and the first decoder D_1 based on user operation input via an input device (not shown).

[0091] The acquisition unit 601 acquires, for example, the first model M_1. Specifically, the acquisition unit 601 acquires the first model M_1 by extracting it from a processing request. Specifically, the acquisition unit 601 may acquire the first model M_1 by receiving it from another computer. The other computer is, for example, the client device 202. Specifically, the acquisition unit 601 may acquire the first model M_1 by accepting input of the first model M_1 based on user operation input via an input device (not shown).

[0092] The intermediate representation generation unit 603 generates an intermediate representation corresponding to the input data using the trained model M. The intermediate representation generation unit 603 generates a first vector corresponding to the generated intermediate representation using the first model M_1. The intermediate representation generation unit 603 generates the first vector, for example, by converting the generated intermediate representation into a first vector using the first model M_1. In this way, the intermediate representation generation unit 603 can obtain a first vector from which the features of the input data have been extracted.

[0093] The distribution learning unit 604 uses the training data used during the training of the trained model M as input data. The distribution learning unit 604 sets an objective function that includes the likelihood of the output data corresponding to the input data and the KL divergence regarding the distribution of latent representations. Using the set objective function, the distribution learning unit 604 learns the distribution of latent representations corresponding to the generated first vector according to the first encoder E_1 and the first decoder D_1, and also learns the second model M_2. As a result, the distribution learning unit 604 can obtain the distribution of latent representations corresponding to the first vector. The distribution learning unit 604 can prepare the second model M_2 so that it can generate output data.

[0094] The sample selection unit 605 selects latent representation samples from the learned distribution according to the probability distribution. This allows the sample selection unit 605 to obtain latent representation samples that will form the basis of new, valid vectors.

[0095] The reconstruction unit 606 generates a new second vector, different from the first vector, corresponding to the selected sample, using the first decoder D_1. The reconstruction unit 606 generates the new second vector, for example, by converting the selected sample into the new second vector using the first decoder D_1. This allows the reconstruction unit 606 to perform operations on the original first vector within an operating range based on the distribution of latent representations, thereby obtaining a valid new second vector.

[0096] The output generation unit 607 generates output data corresponding to the newly generated second vector using the trained second model M_2. For example, the output generation unit 607 generates output data by converting the newly generated second vector into output data using the trained second model M_2. This allows the output generation unit 607 to obtain a variety of valid output data.

[0097] Here, we have described a case where the control device 201 includes an acquisition unit 601, a model learning unit 602, an intermediate representation generation unit 603, a distribution learning unit 604, a sample selection unit 605, a reconstruction unit 606, an output generation unit 607, and an output unit 608, but it is not limited to this. For example, the control device 201 may not include any of the functional units. Specifically, the control device 201 may not include the model learning unit 602.

[0098] (First example of operation of control device 201) Next, a first example of the operation of the control device 201 will be described using Figures 7 and 8.

[0099] Figures 7 and 8 are explanatory diagrams showing a first example of operation of the control device 201. In Figure 7, the control device 201 has a trained model that includes an encoder 701, which is a Transformer encoder, and a decoder 702, which is a Transformer decoder. The control device 201 has a VAE model that includes a VAE encoder 711 and a VAE decoder 712. The VAE model has the function of reconstructing an intermediate representation. The intermediate representation is a set of T d-dimensional vectors. The control device 201 has the training data used when training the trained model. In the following description, characters preceded by ~ may be written as "character ~".

[0100] The control device 201 sets the training data to input data x. In the following explanation, the intermediate representation obtained by transforming input data x with encoder 701 may be referred to as "intermediate representation H". The output obtained by directly transforming intermediate representation H with decoder 702 may be referred to as "output y". The latent representation obtained by transforming intermediate representation H with VAE encoder 711 may be referred to as "latent representation z". The intermediate representation obtained by transforming latent representation z with VAE decoder 712 may be referred to as "intermediate representation H'". The intermediate representation obtained by transforming a sample z~ selected from the distribution Pψ(z) of latent representation z with VAE decoder 712 may be referred to as "intermediate representation H~". The output obtained by transforming intermediate representation H~ with decoder 702 may be referred to as "output y~".

[0101] The control device 201 fixes the parameters of the trained model. The control device 201 sets the objective function. The objective function represents, for example, a weighted sum of the likelihood of the output y, the reconstruction error of the intermediate representation H, and the KL divergence with respect to the distribution Pψ(z) of the latent representation. Here, the VAE encoder 711 fully connects T d-dimensional vectors that serve as intermediate representations of the input, and then generates a (d×T)×d-dimensional weight matrix W. e This includes preprocessing to generate a d-dimensional vector by multiplying by a function. The VAE encoder 711 includes a multilayer perceptron that converts a d-dimensional vector into another d-dimensional vector.

[0102] The VAE decoder 712 is a model that includes a multilayer perceptron that converts a d-dimensional vector into a d-dimensional vector. The VAE decoder 712 converts a d-dimensional vector into a d×(d×T) weight matrix W d The process includes generating a d×T-dimensional vector by multiplying it by a factor, and then post-processing to reconstruct T intermediate d-dimensional vectors from the generated d×T-dimensional vector. The VAE decoder 712 may include a gating mechanism 800, which will be described later in Figure 8, and may reconstruct T intermediate d-dimensional vectors from the d-dimensional vector output of a multilayer perceptron.

[0103] The control device 201 learns the distribution Pψ(z) of the latent representation z by training a VAE model using the input data x to minimize the objective function. Training the VAE model corresponds to updating the VAE model. In the following explanation, the probability value of the latent representation z is denoted as p(z). The control device 201 selects a sample z~ with a relatively high probability from the learned distribution Pψ(z). The control device 201 converts the selected sample z~ into an intermediate representation H~ using the VAE decoder 712. The control device 201 converts the converted intermediate representation H~ into an output y~ using the decoder 702.

[0104] As a result, the control device 201 can obtain a variety of outputs y~ that are different from the output y obtained by directly converting the intermediate representation H with the decoder 702. The control device 201 can perform operations on the intermediate representation H within an operating range based on the distribution Pψ(z), obtain a new and valid intermediate representation H~, and obtain a valid output y~. Now, let's move on to the explanation of Figure 8, which shows an example of the gating mechanism 800.

[0105] In Figure 8, the gating mechanism 800 determines the output o of [b,k,d] corresponding to the latent representation z of [b,d], based on the past outputs of [b,k,d] and the latent representation z of [b,d]. b is the batch size. The gating mechanism 800 includes processing units 801 to 807. The gating mechanism 800 combines the past outputs of [b,k,d] and the latent representation z of [b,d] via processing units 801 to 807 to determine the output o of [b,k,d].

[0106] Processing unit 801 is a masked self-attention. Processing unit 802 represents multiplication by matrix G. Processing unit 803 represents multiplication by matrix G'. Processing unit 804 represents addition. Processing unit 805 represents the application of σ. Processing unit 806 is matrix W V This represents multiplication. Processing unit 807 represents elemental product. For details on the gating mechanism 800, please refer to reference 2 below.

[0107] Reference 2: Montero, Ivan, Nikolaos Pappas, and Noah A. Smith. “Sentence bottleneck autoencoders from transformer language models.” arXiv preprint arXiv:2109.00055 (2021).

[0108] (Second example of operation of control device 201) Next, a second example of the operation of the control device 201 will be explained using Figures 9 to 12.

[0109] Figures 9 to 12 are explanatory diagrams showing a second operation example of the control device 201. In Fig. 9, the control device 201 has a learned model including an encoder 901 which is a Transformer encoder and a decoder 902 which is a Transformer decoder. The control device 201 has a VAE model including a VAE encoder 911 and a VAE decoder 912. The control device 201 has the learning data used during the learning of the learned model.

[0110] Here, since the intermediate representation which is the output of the Transformer tends to be of extremely high dimension, it is considered preferable to recognize the intermediate representation as a series of vectors in the same space, capture the information of the latent representation, reduce it to a low-dimensional representation, and facilitate the learning of the distribution of the latent representation. Therefore, the control device 201 applies a model including a convolution operation to the VAE encoder 911. Specifically, the control device 201 applies the model 1000 described later in Fig. 10 to the VAE encoder 911. Also, the control device 201 applies a model including a deconvolution operation to the VAE decoder 912. Specifically, the control device 201 applies the model 1010 described later in Fig. 10 to the VAE decoder 912. Now, let's move on to the description of Fig. 10.

[0111] Fig. 10 shows an example of the model 1000 which becomes the VAE encoder 911 and an example of the model 1010 which becomes the VAE decoder 912. The model 1000 includes processing units 1001 to 1003. The model 1000 converts the intermediate representation H of B×T×d H into the latent representation z0 of B×d z and has a function of obtaining the parameters (μ,σ) of B×d z representing the distribution of the latent representation z0.

[0112] The processing unit 1001 converts the intermediate representation H of B×T×d H into the intermediate representation of B×T´×d zIt corresponds to a convolution operation that reduces the number of channels to 1 / r. Processing unit 1002 corresponds to a one-dimensional convolution operation in the sequence direction that reduces the number of channels to 1 / r. Model 1000 is obtained from processing unit 1002 with B×T'×d z The output representation of / r is B×T' / r×d z This includes a reshape operation that converts to m. Model 1000 has a processing unit 1004 which consists of a processing unit 1002 and a reshape operation. E This process is repeated several times. The processing unit 1003 estimates the distribution of latent representation z0.

[0113] Model 1010 includes processing units 1011 and 1012. Model 1010 selects B×1×d from the distribution represented by the parameters (μ, σ) via processing units 1011 and 1012. z The latent representation z is B × T' × d H It has the function of converting to the intermediate representation H'.

[0114] The processing unit 1011 corresponds to a one-dimensional deconvolution operation in the sequence direction that expands the number of channels by a factor of r. Model 1010 is, for example, B × r obtained in the processing unit 1011. i × (r × d z The output representation of ) is B × r i+1 ×d z This includes a reshape operation that converts to m. Model 1010 has a processing unit 1013 which consists of a processing unit 1011 and a reshape operation. D The process is repeated a certain number of times. Processing unit 1012 calculates B × T × d H The output representation is B×T'×d H This corresponds to a convolution operation that expands to . Next, with respect to the case where r=2, specific examples of processing units 1004 and 1013 will be explained using Figures 11 and 12.

[0115] Figure 11 shows a specific example of processing unit 1004. In Figure 11, conv1d is performed by a one-dimensional convolution operation, resulting in a size k E Using the kernel, size (B × T × d in Input ) for size (B×T×d out This is the process of converting to the output of ). B is the batch size. T is the sequence length. din d is the input channel size. out This is the output channel size.

[0116] In Figure 11, the intermediate representation that serves as the input, as shown in Table 1101, is (B,T',d z The values ​​displayed in each rectangle are the component indices. Rectangles with the same background color use the same index. The intermediate representation shown in Table 1101 is expressed by conv1d as shown in Table 1111 (B,T',d z It is converted to the output representation of / r).

[0117] The output representation shown in Table 1111 is shown in Table 1112 (B,T' / r,d z The output representation shown in Table 1112 is reshaped to the output representation shown in Table 1113 (B,d z The output representation is permitted as / r,r,T' / r). The output representation shown in Table 1113 is shown as (B,T' / r,d z The output representation of ) is reshaped.

[0118] To improve learning efficiency, it is sometimes preferable to calculate the average of the output representations shown in Table 1121, which are obtained by rearranging the input components to have the same shape as the output, thereby generating the output representation shown in Table 1122, and adding it to the output representation shown in Table 1114. Next, we will move on to the explanation of Figure 12.

[0119] Figure 12 shows a specific example of a processing unit 1013. In Figure 12, conv1d is performed by a one-dimensional deconvolution operation, resulting in a size k D Using the kernel, size (B × T × d in Input ) for size (B×T×d out This is the process of converting to the output of ). B is the batch size. T is the sequence length. d in d is the input channel size. out This is the output channel size.

[0120] In Figure 12, the output representation of the input, as shown in Table 1201, is (B,T',dz The values ​​displayed in each rectangle are the component indices. Rectangles with the same background color use the same index. The intermediate representation shown in Table 1201 is expressed by conv1d as shown in Table 1211 (B,T',rd z It is converted to the output representation of ).

[0121] The output representation shown in Table 1211 is shown in Table 1212 (B,T' / r,d z The output representation of ,r) is reshaped and permuted. The output representation shown in Table 1212 is shown in Table 1213 as (B,rT',d z The output representation is reshaped. To improve learning efficiency, it is sometimes preferable to add the output representation shown in Table 1220, which is obtained by repeating and reshaping the input, to the output representation shown in Table 1213.

[0122] Returning to the explanation of Figure 9, the control device 201 has the training data used during the training of the trained model. The control device 201 sets the training data as input data x. The control device 201 fixes the parameters of the trained model. The control device 201 sets the objective function. The objective function represents, for example, a weighted sum of the likelihood of the output y, the reconstruction error of the intermediate representation H, and the KL divergence with respect to the distribution Pψ(z) of the latent representation.

[0123] The control device 201 learns the distribution Pψ(z) of the latent representation z by training a VAE model using the input data x to minimize the objective function. Training the VAE model corresponds to updating the VAE model. In the following explanation, the probability value of the latent representation z is denoted as p(z). The control device 201 selects a sample z~ with a relatively high probability from the learned distribution Pψ(z). The control device 201 converts the selected sample z~ into an intermediate representation H~ by repeatedly performing an inverse convolution operation with the VAE decoder 912. The control device 201 converts the converted intermediate representation H~ into an output y~ with the decoder 902.

[0124] As a result, the control device 201 can obtain a variety of outputs y~ that are different from the output y obtained by directly converting the intermediate representation H with the decoder 902. The control device 201 can perform operations on the intermediate representation H within an operating range based on the distribution Pψ(z), obtain a new and valid intermediate representation H~, and obtain a valid output y~. The control device 201 can use a VAE model that includes convolution operations to facilitate learning the distribution of latent representations.

[0125] (Third example of operation of control device 201) Next, a third example of the control device 201's operation will be described using Figures 13 and 14.

[0126] Figures 13 and 14 are explanatory diagrams showing a third operation example of the control device 201. As described above, the intermediate representation, which is the output of the Transformer, tends to be very high-dimensional. It is considered preferable to recognize the intermediate representation as a sequence of vectors in the same space, capture information from the latent representation, and reduce it to a low-dimensional representation, thereby making it easier to learn the distribution of the latent representation.

[0127] Therefore, in Operation Example 3, the control device 201 applies a model to the VAE encoder 911 that includes a convolution operation and layers the latent representation. For more information on layering latent representations, see, for example, Reference 3 below. The latent representation is information that extracts features from the intermediate representation at different levels of abstraction for each layer.

[0128] Reference 3: Child, Rewon. “Very deep vaes generalize autoregressive models and can outperform them on images.” arXiv preprint arXiv:2011.10650 (2020).

[0129] Specifically, the control device 201 applies Model 1300, which will be described later in Figure 13, to the VAE encoder 911. Furthermore, the control device 201 applies a model to the VAE decoder 912 that includes a deconvolution operation and layers the latent representation. Specifically, the control device 201 applies Model 1400, which will be described later in Figure 14, to the VAE decoder 912. Now, let's move on to the explanation of Figure 13.

[0130] Figure 13 shows an example of Model 1300, which is the VAE encoder 911. Model 1300 includes processing units 1301 to 1307, etc. Model 1000 performs B × T × d via processing units 1301 to 1307, etc. H From the intermediate representation H, the latent representation z of hierarchy l l B×Td represents the distribution z / r l The posterior distribution parameter (μ) ql ,σ ql It has the function of finding ). l = 0, 1, ..., s.

[0131] Processing unit 1301 is conv1d. Processing unit 1302 estimates the distribution. Processing unit 1303 is ConvScaling. Processing unit 1304 estimates the distribution. Model 1000 includes multiple processing units similar to processing units 1303 and 1304, and repeats the one-dimensional convolution operation of ConvScaling. Processing unit 1305 is the lowest layer ConvScaling. Processing unit 1306 estimates the distribution. ConvScaling includes, for example, processing unit 1307 and a reshape operation. Specifically, Model 1300 uses the posterior distribution parameter (μ) for l=0,1,···,s. ql ,σ ql We then calculate the posterior distribution parameters, for example, the variance and mean of a normal distribution. Next, we move on to the explanation of Figure 14.

[0132] Figure 14 shows an example of Model 1400, which is the VAE decoder 912. Model 1400 includes processing units 1401 to 1406, etc. Model 1400, via processing unit 1401, has a posterior distribution parameter (μ) at l=0. ql ,σ qlB×Td selected from the distribution represented by ). z The latent representation z0 is B × T × d H It has the function of converting to the intermediate representation H'. Model 1400, via processing units 1402~1406 etc., has the posterior distribution parameter (μ) at l=1,···,s. ql ,σ ql B×Td selected from the distribution represented by ). z / r l latent expression z l It has the function of converting.

[0133] Processing unit 1401 is conv1d. Processing unit 1402 is DeconvScaling. Processing unit 1403 estimates the distribution. Model 1000 includes multiple processing units similar to processing units 1402 and 1403, and performs a one-dimensional convolution operation of DeconvScaling for each layer. Processing unit 1404 is the DeconvScaling of the lowest layer. Processing unit 1405 estimates the distribution. DeconvScaling includes, for example, processing unit 1406 and a reshape operation. Specifically, Model 1400 uses prior distribution parameters (μ) for l=0,1,···,s-1. pl ,σ pl )

[0134] The following explanation assumes that the output is language. The control unit 201 has the training data used during the training of the trained model. The control unit 201 sets the training data as input data x. The control unit 201 fixes the parameters of the trained model. The control unit 201 sets the objective function λ NLL NLL+λ MSE MSE+Σβ l KLD l Set KLD. l =KLD(q l ||p l ) = -0.5(1 + log(σ ql 2 )-logσ pl 2 -(σ ql 2 +(μ ql -μ pl ) 2 ) / (σpl 2 ) is p l , q l is Td z / r l is a normal distribution in the dimension of NLL = Σ i T (-log p(y i )). T is the sequence length. p(y i ) is the predicted probability of the model for the correct word y i . MSE = Σ i T ((||H i '- H i ||) 2 / (||H i ||) 2 ). H i , H i ' are the intermediate representations before and after restoration, corresponding to the position i in the sequence length T

[0135] The control device 201 learns the prior distribution parameters (μ l , σ pl ) representing the distribution of the latent representation z pl of each layer l by learning the VAE model using the input data x so as to minimize the objective function. The learning of the VAE model corresponds to the update of the VAE model. The control device 201 selects a sample z pl , σ pl that belongs to a specific layer l and has a relatively high probability from the distribution represented by the prior distribution parameters (μ l ~ of a specific layer l. The specific layer l is, for example, the lowest layer. The control device 20 refers to the model 1400 and performs a deconvolution operation on the sample z l ~, and repeats converting the sample z l ~ into the output representation of the layer above to convert the selected sample z l ~ into the intermediate representation H~. The control device 20 converts the converted intermediate representation H~ into the output y~ by the decoder 902

[0136] As a result, the control device 201 can obtain a variety of outputs y~ that are different from the output y obtained by directly converting the intermediate representation H with the decoder 902. The control device 201 can perform operations on the intermediate representation H within an operating range based on the distribution Pψ(z), obtain a new and valid intermediate representation H~, and obtain a valid output y~. The control device 201 can use a VAE model that includes convolution operations to make it easier to learn the distribution of latent representations. Furthermore, the control device 201 can improve the expressive power of latent representations by hierarchizing them.

[0137] Here, we have described the case where a specific hierarchy l is the lowest level, but this is not the only case. For example, a specific hierarchy l may be a level other than the lowest level. Alternatively, the control device 201 may select multiple levels as the specific hierarchy l. As a result, the control device 201 can generate latent representations z of various levels of abstraction. l Since a valid output y~ can be obtained via this method, it becomes easier to obtain a variety of outputs y~.

[0138] (Fourth example of operation of control device 201) Next, Figure 15 will be used to explain the fourth operation example of the control device 201. The fourth operation example corresponds to a case where the control device 201 does not use a trained model when generating output data.

[0139] Figure 15 is an explanatory diagram showing a fourth example of operation of the control device 201. In Figure 15, the control device 201 has an encoder 1500 of a trained model. The control device 201 has an encoder 1510 of the AutoBot. The control device 201 has a VAE model that includes a VAE encoder 1520 and a VAE decoder 1530. The control device 201 has an AutoBot decoder 1540. For more information on the AutoBot, see, for example, reference 2 above.

[0140] In the example shown in Figure 15, the control device 201 directly predicts the structure from the latent representation z obtained from the intermediate representation H. The control device 201 uses the latent representation z generated by the encoder 1510. A This is projected onto another latent representation z by the VAE encoder 1520, and the latent representation z is processed by the VAE decoder 1530. A Restore ' and latent expression z A The ' is output by decoder 1540 as h t Specify that it should be output to [a specific format].

[0141] The control device 201 holds the training data used during the training of the trained model. The control device 201 sets the training data as input data x. The control device 201 fixes the parameters of the trained model. The control device 201 sets the objective function NLL+βKLD. KLD = -0.5(1+logσ 2 -μ 2 -σ 2 ) μ and σ are the posterior distribution parameters. NLL = Σ i T (-logp(y i )) is the sequence length. p(y i ) is the correct word y i This is the model's predicted probability for [the given value].

[0142] The control device 201 learns the posterior distribution parameters (μ,σ) and the decoder 1540 by training a VAE model using the input data x to minimize the objective function. Training the VAE model corresponds to updating the VAE model. Training the decoder 1540 corresponds to updating the decoder 1540. The control device 201 selects a sample z~ with a relatively high probability from the distribution represented by the posterior distribution parameters (μ,σ). The control device 201 then selects the selected sample z l ~ is represented as latent z by VAE decoder 1530 A Convert to ~. The control device 201 converts the latent representation z A ~ is output h by decoder 1540 t Convert to ~.

[0143] As a result, the control device 201 can output various outputs h tThe control device 201 can perform operations on the intermediate representation H within a distribution-based operating range, and obtain a valid output h t You can obtain ~.

[0144] As shown in each example of operation, the control device 201 can improve the quality of the output generated by operations on the intermediate representation. Conventionally, it is not clear whether the manipulated intermediate representation is within the range of the data distribution, and it is possible that an invalid output may be obtained. In contrast, the control device 201 can identify operations on the intermediate representation necessary to generate a valid output by sampling through the latent space, and can efficiently obtain a valid output.

[0145] (Learning process steps) Next, an example of a learning process procedure executed by the control device 201 will be described using Figure 16. The learning process is implemented, for example, by the CPU 301 shown in Figure 3, a storage device such as the memory 302 and disk 304, and a communication I / F 305.

[0146] Figure 16 is a flowchart illustrating an example of the learning process procedure. In Figure 16, the control device 201 acquires the training data to be used for distributional learning (step S1601). The training data does not necessarily have to be the same as the data used when training the trained model. The control device 201 initializes the VAE model (step S1602). Similar to any other example of operation, the control device 201 learns the distribution of latent representations in the VAE model by training the VAE model based on the acquired training data and the initialized VAE model (step S1603). The control device 201 then terminates the learning process.

[0147] (Generation process procedure) Next, an example of a generation process procedure executed by the control device 201 will be described using Figure 17. The generation process is realized, for example, by the CPU 301 shown in Figure 3, a storage device such as the memory 302 and disk 304, and a communication I / F 305.

[0148] Figure 17 is a flowchart of an example of the generation process procedure. In Figure 17, the control device 201 selects a sample of latent representations with a relatively high probability from the learned distribution of latent representations (step S1701). The control device 201 generates output data based on the selected sample, as in any of the other operation examples (step S1702). The control device 201 outputs the output data (step S1703). The control device 201 terminates the generation process.

[0149] As explained above, the control device 201 can generate an intermediate representation corresponding to the input data using a trained model. The control device 201 can learn the distribution of latent representations corresponding to the generated intermediate representation, depending on a predetermined encoder that generates latent representations and a predetermined decoder that corresponds to the predetermined encoder. The control device 201 can select a sample of latent representation corresponding to a probability distribution from the learned distribution. The control device 201 can generate a new intermediate representation corresponding to the selected sample using a predetermined decoder. The control device 201 can generate output data corresponding to the newly generated intermediate representation using a trained model. As a result, the control device 201 can obtain valid output data.

[0150] The control device 201 allows the use of training data from a pre-trained model as input data. The control device 201 allows setting an objective function that includes the likelihood of the output data corresponding to the input data, the reconstruction error of the intermediate representation corresponding to the input data, and the KL divergence regarding the distribution of latent representations. The control device 201 allows learning the distribution of latent representations according to a predetermined encoder and a predetermined decoder using the objective function. As a result, the control device 201 can learn the distribution of latent representations with high accuracy, making it easier to obtain valid output data.

[0151] The control device 201 can utilize a predetermined encoder that includes a first generator and a first multilayer perceptron that generates a latent representation corresponding to the generated first vector. The first generator generates the first vector by fully connecting multiple vectors representing the intermediate representation and then multiplying them by a first weight matrix. The control device 201 can have a predetermined decoder that includes a second multilayer perceptron that generates a second vector corresponding to the latent representation and a second generator. The second generator reconstructs multiple vectors representing the intermediate representation from a third vector obtained by multiplying the generated second vector by a second weight matrix. In this way, the control device 201 can utilize a predetermined combination of an encoder and a predetermined decoder in a way that allows the distribution of latent representations to be learned.

[0152] According to the control device 201, a predetermined encoder can be used to generate a latent representation by repeatedly performing a convolution operation on an intermediate representation and to identify parameters that represent the distribution of the latent representation. According to the control device 201, a predetermined decoder can be used to generate an intermediate representation by repeatedly performing an inverse convolution operation on the latent representation. According to the control device 201, the distribution of the latent representation can be identified by identifying the parameters. As a result, since the control device 201 uses a corresponding combination of a predetermined encoder that includes a convolution operation and a predetermined decoder, it is possible to capture information from the latent representation, reduce it to a low-dimensional representation, and make it easier to learn the distribution of the latent representation.

[0153] The control device 201 can utilize a predetermined encoder that includes performing multiple convolution operations on an intermediate representation. The predetermined encoder makes it possible to identify a first parameter that represents the distribution of each of the multiple hierarchical latent representations according to the result of each convolution operation. The control device 201 can utilize a predetermined decoder that includes performing a deconvolution operation on each of the multiple latent representations. The predetermined decoder fixes a second parameter that represents the distribution of the lowest layer latent representation and makes it possible to identify a second parameter that represents the distribution of each of the latent representations other than the lowest layer. The control device 201 can learn the first parameter according to the predetermined encoder based on the input data, and learn the second parameter according to the predetermined decoder based on the first parameter. The control device 201 can learn the distribution of each of the multiple latent representations represented by the second parameter. As a result, the control device 201 can utilize corresponding combinations of a predetermined encoder that includes convolution operations and a predetermined decoder, making it easier to capture information from latent representations, reduce it to a low-dimensional representation, and learn the distribution of latent representations.

[0154] According to the control device 201, a first vector corresponding to the generated intermediate representation can be generated by the first model. According to the control device 201, the training data used during the training of the trained model can be used as input data. According to the control device 201, an objective function can be set that includes the likelihood of the output data corresponding to the input data and the KL divergence regarding the distribution of latent representations. According to the control device 201, using the objective function, the distribution of latent representations corresponding to the generated first vector can be learned according to the first encoder and first decoder that generate latent representations, and a second model that converts the vector into output data can be learned. According to the control device 201, a sample of latent representation corresponding to the probability distribution can be selected from the learned distribution. According to the control device 201, a second vector corresponding to the selected sample can be generated by the first decoder. According to the control device 201, output data corresponding to the generated second vector can be generated by the trained second model. As a result, the control device 201 can generate output data without using the trained model.

[0155] According to the control device 201, a deep learning model can be adopted as the trained model, which takes amino acid sequences as input data and outputs output data representing the structure of a protein. As a result, the control device 201 can obtain output data representing a variety of valid protein structures.

[0156] According to the control device 201, the trained model can employ a model that takes sequence information representing a sentence as input data and outputs sequence information representing another sentence as output data. This allows the control device 201 to obtain output data representing a variety of valid sentences.

[0157] The control method described in this embodiment can be implemented by executing a pre-prepared program on a computer such as a PC or workstation. The control program described in this embodiment is recorded on a computer-readable recording medium and executed by being read from the recording medium by the computer. The recording medium can be a hard disk, flexible disk, CD (Compact Disc)-ROM, MO (Magneto Optical Disc), DVD (Digital Versatile Disc), etc. Furthermore, the control program described in this embodiment may be distributed via a network such as the Internet.

[0158] With regard to the embodiments described above, the following additional information is disclosed.

[0159] (Note 1) An intermediate representation corresponding to the input data is generated using the trained model. The distribution of latent representations corresponding to the generated intermediate representations is learned according to a predetermined encoder that generates the latent representations from the intermediate representations and a predetermined decoder that corresponds to the predetermined encoder and generates intermediate representations different from the intermediate representations. From the learned distribution, select a sample of the latent representation corresponding to the probability distribution. A new intermediate representation corresponding to the selected sample is generated by the predetermined decoder. The trained model generates output data corresponding to the newly generated intermediate representation. A control program characterized by causing a computer to perform a process.

[0160] (Note 2) The learning process described above is: The control program according to Appendix 1, characterized in that it uses the training data used during the training of the trained model as the input data, and learns the distribution of the latent representation according to the predetermined encoder and the predetermined decoder, using an objective function that includes the likelihood of the output data corresponding to the input data, the reconstruction error of the intermediate representation corresponding to the input data, and the KL divergence relating to the distribution of the latent representation.

[0161] (Note 3) The predetermined encoder includes a first generator that generates a first vector by fully connecting a plurality of vectors representing intermediate representations and then multiplying them by a first weight matrix, and a first multilayer perceptron that generates the latent representation corresponding to the generated first vector, The control program according to Appendix 2, characterized in that the predetermined decoder includes a second multilayer perceptron that generates a second vector corresponding to the latent representation, and a second generator that reconstructs a plurality of vectors representing the intermediate representation from a third vector obtained by multiplying the generated second vector by a second weight matrix.

[0162] (Note 4) The predetermined encoder is a model that generates the latent representation by repeatedly performing a convolution operation on the intermediate representation and identifies parameters that represent the distribution of the latent representation. The aforementioned decoder is a model that generates an intermediate representation by repeatedly performing a deconvolution operation on the latent representation, The learning process described above is: The control program according to Appendix 2, characterized in that the distribution of the latent representation is determined by specifying the aforementioned parameters.

[0163] (Note 5) The predetermined encoder is a model that includes repeating a convolution operation multiple times on an intermediate representation and identifies a first parameter that represents the distribution of each of the hierarchical multiple latent representations according to the result of each convolution operation, The aforementioned predetermined decoder is a model that includes performing a deconvolution operation on each of the multiple latent representations, fixing a second parameter that represents the distribution of the lowest layer latent representation, and identifying a second parameter that represents the distribution of each of the latent representations other than the lowest layer. The learning process described above is: The control program according to Appendix 2, characterized in that it learns a first parameter according to a predetermined encoder based on the input data, and learns a second parameter according to a predetermined decoder based on the first parameter, thereby learning the distribution of each of the multiple latent representations represented by the second parameter.

[0164] (Note 6) The first vector corresponding to the generated intermediate representation is generated in the first model, The training data used during the training of the aforementioned trained model is adopted as the input data, and the distribution of the latent representation corresponding to the generated first vector is learned using an objective function that includes the likelihood of the output data corresponding to the input data and the KL divergence relating to the distribution of the latent representation, in accordance with a first encoder that generates the latent representation from the first vector and a first decoder that corresponds to the first encoder and generates a vector different from the first vector, and a second model is learned that converts the different vector into output data. From the learned distribution, select a sample of the latent representation corresponding to the probability distribution. The first decoder generates a second vector corresponding to the selected sample. The output data corresponding to the generated second vector is generated using the trained second model. The control program described in Appendix 1, characterized in that it causes a computer to perform the processing.

[0165] (Note 7) The control program according to any one of Notes 1 to 6, characterized in that the trained model is a deep learning model that takes an amino acid sequence as input data and outputs output data representing the structure of a protein.

[0166] (Note 8) The control program described in any one of Notes 1 to 6, characterized in that the trained model is a model that takes sequence information representing a sentence as input data and outputs sequence information representing another sentence as output data.

[0167] (Note 9) An intermediate representation corresponding to the input data is generated using the trained model. The distribution of latent representations corresponding to the generated intermediate representations is learned according to a predetermined encoder that generates the latent representations from the intermediate representations and a predetermined decoder that corresponds to the predetermined encoder and generates intermediate representations different from the intermediate representations. From the learned distribution, select a sample of the latent representation corresponding to the probability distribution. A new intermediate representation corresponding to the selected sample is generated by the predetermined decoder. The trained model generates output data corresponding to the newly generated intermediate representation. A control method characterized by having a computer perform the processing.

[0168] (Note 10) An intermediate representation corresponding to the input data is generated using the trained model. The distribution of latent representations corresponding to the generated intermediate representations is learned according to a predetermined encoder that generates the latent representations from the intermediate representations and a predetermined decoder that corresponds to the predetermined encoder and generates intermediate representations different from the intermediate representations. From the learned distribution, select a sample of the latent representation corresponding to the probability distribution. A new intermediate representation corresponding to the selected sample is generated by the predetermined decoder. The trained model generates output data corresponding to the newly generated intermediate representation. An information processing device characterized by having a control unit. [Explanation of Symbols]

[0169] 100 Information Processing Devices 101,400 Input Data 102,105,500 Intermediate expression 103 Distribution 104 specimen 106 Output data 110,220 pre-trained models 1000, 1010, 1300, 1400 models 111,221,701,901,1500,1510 encoder 112,222,702,902,1540 Decoder 121 A predetermined encoder 122 A predetermined decoder 200 Information Processing Systems 201 Control device 202 Client Devices 210 Network 231,711,911,1520 VAE encoder 232,712,912,1530 VAE decoder 300 bus 301 CPU 302 memory 303 Disk Drive 304 disks 305 Communication I / F 306 GPU 307 Portable recording medium interface 308 Portable recording media 600 Storage section 601 Acquisition Department 602 Model Learning Department 603 Intermediate representation generator 604 Distribution Learning Department 605 Specimen Selection Section 606 Restoration Section 607 Output generation unit 608 Output section 800 gating mechanism Processing Units 801-807, 1001-1003, 1011, 1012, 1301-1307, 1401-1406 1004,1013 processing units 1101,1111~1114,1121,1122,1201,1211~1213,1220 Table

Claims

1. An intermediate representation corresponding to the input data is generated using the trained model. The distribution of latent representations corresponding to the generated intermediate representations is learned according to a predetermined encoder that generates the latent representations from the intermediate representations and a predetermined decoder that corresponds to the predetermined encoder and generates intermediate representations different from the intermediate representations. From the learned distribution, select a sample of the latent representation corresponding to the probability distribution. A new intermediate representation corresponding to the selected sample is generated by the predetermined decoder. The trained model generates output data corresponding to the newly generated intermediate representation. A control program characterized by causing a computer to perform a process.

2. The learning process described above is: The control program according to claim 1, characterized in that it uses the training data used during the training of the trained model as the input data, and learns the distribution of the latent representation according to the predetermined encoder and the predetermined decoder, using an objective function that includes the likelihood of the output data corresponding to the input data, the reconstruction error of the intermediate representation corresponding to the input data, and the KL divergence relating to the distribution of the latent representation.

3. The predetermined encoder includes a first generator that generates a first vector by fully connecting a plurality of vectors representing intermediate representations and then multiplying them by a first weight matrix, and a first multilayer perceptron that generates the latent representation corresponding to the generated first vector, The control program according to claim 2, characterized in that the predetermined decoder includes a second multilayer perceptron that generates a second vector corresponding to the latent representation, and a second generator that reconstructs a plurality of vectors representing an intermediate representation from a third vector obtained by multiplying the generated second vector by a second weight matrix.

4. The predetermined encoder is a model that generates the latent representation by repeatedly performing a convolution operation on the intermediate representation and identifies parameters that represent the distribution of the latent representation. The aforementioned decoder is a model that generates an intermediate representation by repeatedly performing a deconvolution operation on the latent representation, The learning process described above is: The control program according to claim 2, characterized in that the distribution of the latent representation is identified by identifying the aforementioned parameters.

5. The predetermined encoder is a model that includes repeating a convolution operation multiple times on an intermediate representation and identifies a first parameter that represents the distribution of each of the hierarchical multiple latent representations according to the result of each of the convolution operations, The aforementioned predetermined decoder is a model that includes performing a deconvolution operation on each of the multiple latent representations, fixing a second parameter that represents the distribution of the lowest layer latent representation, and identifying a second parameter that represents the distribution of each of the latent representations other than the lowest layer. The learning process described above is: The control program according to claim 2, characterized in that it learns a first parameter according to a predetermined encoder based on the input data, and learns a second parameter according to a predetermined decoder based on the first parameter, thereby learning the distribution of each of the multiple latent representations represented by the second parameter.

6. The first model generates a first vector corresponding to the generated intermediate representation. The training data used during the training of the aforementioned trained model is adopted as the input data, and the distribution of the latent representation corresponding to the generated first vector is learned in accordance with a first encoder that generates the latent representation from the first vector and a first decoder that corresponds to the first encoder and generates a vector different from the first vector, and a second model is learned that converts the different vector into output data. From the learned distribution, select a sample of the latent representation corresponding to the probability distribution. The first decoder generates a second vector corresponding to the selected sample. The output data corresponding to the generated second vector is generated by the trained second model. The control program according to claim 1, characterized in that it causes a computer to perform the processing.

7. An intermediate representation corresponding to the input data is generated using the trained model. The distribution of latent representations corresponding to the generated intermediate representations is learned according to a predetermined encoder that generates the latent representations from the intermediate representations and a predetermined decoder that corresponds to the predetermined encoder and generates intermediate representations different from the intermediate representations. From the learned distribution, select a sample of the latent representation corresponding to the probability distribution. A new intermediate representation corresponding to the selected sample is generated by the predetermined decoder. The trained model generates output data corresponding to the newly generated intermediate representation. A control method characterized by having a computer perform the processing.

8. An intermediate representation corresponding to the input data is generated using the trained model. The distribution of latent representations corresponding to the generated intermediate representations is learned according to a predetermined encoder that generates the latent representations from the intermediate representations and a predetermined decoder that corresponds to the predetermined encoder and generates intermediate representations different from the intermediate representations. From the learned distribution, select a sample of the latent representation corresponding to the probability distribution. A new intermediate representation corresponding to the selected sample is generated by the predetermined decoder. The trained model generates output data corresponding to the newly generated intermediate representation. An information processing device characterized by having a control unit.

Images