Information processing device, information processing method, and program

Abstract

To generate a neural network having higher performance by learning.SOLUTION: An information processing device outputs, based on input data and a first neural network, an embedded representation of the input data, outputs, based on the embedded representation and a second neural network connected to a subsequent stage of the first neural network, first inference results that are inference results related to a predetermined task, outputs, based on the embedded representation and a third neural network connected to a subsequent stage of the first neural network, second inference results that are inference results related to a type of the input data, outputs, based on the first inference results and the second inference results, evaluation results, and updates, based on the evaluation results, some or all of weight parameters of the first neural network and some or all of weight parameters of the third neural network.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Neural networks acquire high performance for various problems through learning. Various techniques are known for learning neural networks (see, for example, Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Haotian Liu, 3 others, "Improved Baselines with Visual Instruction Tuning", [online], [searched on February 8, 2020], Internet<https: / / arxiv.org / abs / 2310.03744> Summary of the Invention [Problem to be solved by the invention]

[0004] However, it is desirable to generate neural networks with higher performance through training.

[0005] Therefore, the present invention has been proposed to solve such problems, and it is desirable to provide a technique that makes it possible to generate a neural network with higher performance through learning. [Means for solving the problem]

[0006] In order to solve the above problem, according to one aspect of the present invention, there is provided an information processing device comprising: a first processing unit that outputs an embedded representation of input data based on input data and a first neural network; a second processing unit that outputs a first inference result that is an inference result related to a predetermined task based on the embedded representation and a second neural network connected downstream of the first neural network; a data type determination unit that outputs a second inference result that is an inference result related to the type of the input data based on the embedded representation and a third neural network connected downstream of the first neural network; an evaluation unit that outputs an evaluation result based on the first inference result and the second inference result; and an update unit that updates some or all of the weight parameters of the first neural network and some or all of the weight parameters of the third neural network based on the evaluation result.

[0007] The update unit may update some or all of the weight parameters of the second neural network based on the evaluation result.

[0008] The update unit may update some or all of the weight parameters of the first neural network and some or all of the weight parameters of the third neural network by backpropagation based on the evaluation result.

[0009] A gradient inversion layer may be connected to a stage preceding the third neural network and a stage following the first neural network, and the gradient inversion layer may invert the sign of the gradient calculated by the backpropagation algorithm based on the evaluation result by multiplying the gradient by a negative coefficient.

[0010] The evaluation unit may calculate a task evaluation result based on the first inference result, calculate a data type evaluation result based on the second inference result, and calculate the evaluation result based on the task evaluation result and the data type evaluation result.

[0011] The evaluation unit may control a weight by which the data type evaluation result is multiplied based on an embedded representation of the input data.

[0012] The evaluation unit may increase the weight as the relative value of the similarity between embedded expressions of heterogeneous data, which is the similarity between embedded expressions of heterogeneous data and embedded expressions of the input data, increases relative to the similarity between embedded expressions of homogeneous data, which is the similarity between embedded expressions of homogeneous data and embedded expressions of the input data.

[0013] The relative value may be a ratio of the sum of similarities of embedding expressions between the same type of data to the sum of similarities of embedding expressions between the different types of data.

[0014] The relative value may be a ratio of a maximum similarity of the embedding expression between the homogeneous data to a maximum similarity of the embedding expression between the heterogeneous data.

[0015] The relative value may be a difference obtained by subtracting the sum of the similarities of the embedding expressions between the same type of data from the sum of the similarities of the embedding expressions between the different types of data.

[0016] The relative value may be a difference obtained by subtracting the maximum similarity of the embedding expressions between the same type of data from the maximum similarity of the embedding expressions between the different types of data.

[0017] The first processing unit may be an encoding unit, and the second processing unit may be a decoding unit.

[0018] In order to solve the above problem, according to another aspect of the present invention, there is provided an information processing method executed by a computer, comprising: outputting an embedded representation of input data based on input data and a first neural network; outputting a first inference result that is an inference result related to a predetermined task based on the embedded representation and a second neural network connected downstream of the first neural network; outputting a second inference result that is an inference result related to a type of the input data based on the embedded representation and a third neural network connected downstream of the first neural network; outputting an evaluation result based on the first inference result and the second inference result; and updating some or all of the weight parameters of the first neural network and some or all of the weight parameters of the third neural network based on the evaluation result.

[0019] In order to solve the above problem, according to another aspect of the present invention, there is provided a program that causes a computer to function as: a first processing unit that outputs an embedded representation of input data based on input data and a first neural network; a second processing unit that outputs a first inference result that is an inference result related to a predetermined task based on the embedded representation and a second neural network connected downstream of the first neural network; a data type determination unit that outputs a second inference result that is an inference result related to the type of the input data based on the embedded representation and a third neural network connected downstream of the first neural network; an evaluation unit that outputs an evaluation result based on the first inference result and the second inference result; and an update unit that updates some or all of the weight parameters of the first neural network and some or all of the weight parameters of the third neural network based on the evaluation result. [Effects of the Invention]

[0020] As described above, the present invention provides a technique that enables a neural network with higher performance to be generated through learning. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a diagram illustrating an example of the functional configuration of a learning device 10 according to a first embodiment of the present invention. [Figure 2] 4 is a flowchart showing an example of an operation in a learning stage executed by the learning device 10 according to the first embodiment of the present invention. [Figure 3] FIG. 10 is a diagram illustrating an example of the functional configuration of a learning device 20 according to a second embodiment of the present invention. [Figure 4] FIG. 10 is a diagram showing an example of weights ri calculated by the evaluation unit 221. [Figure 5] 1 is a diagram showing a hardware configuration of an information processing device 900 as an example of a learning device 10 according to a first embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0022] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant explanations will be omitted.

[0023] (0. Overview of the embodiment) An outline of an embodiment of the present invention will be described.

[0024] Neural networks (hereafter referred to as "NN") achieve high performance on various problems through learning. However, it is desirable to generate NNs with even higher performance through learning. In addition, various types of AI (Artificial Intelligence) models constructed using NNs are being researched.

[0025] In recent years, large-scale multimodal models (hereinafter simply referred to as "LMM") have become known as models that allow multiple types of data to be specified as inputs and outputs to a model and that can perform processing in accordance with the specified inputs and outputs. Examples of data types include language, images, and video. In this specification, video is defined as data that includes image frames and audio.

[0026] In general, to improve the performance of such LMMs, it is essential to align the embedding representations of heterogeneous data extracted by LMMs in the same space and to utilize the co-occurrence relationships between the heterogeneous data.

[0027] The model described in Non-Patent Document 1 is one of the models that has achieved high performance, especially for two types of data: images and language. This model accepts images and language (e.g., instruction language such as "Please describe the image in detail") as input, and outputs language (e.g., response language such as an explanation of the image). The main components of this model are a large-scale language model (hereinafter also referred to as "LLM"), an encoder that extracts the image embedding, and a converter that converts the image embedding into a space understandable by the LLM. Then, using the above input and output data, the converter trains so that the image embedding is aligned in the same space as the language embedding.

[0028] However, the method described in Non-Patent Document 1 accepts images and language as inputs and performs learning using language as output and teacher labels. In this type of learning, there is a possibility that the alignment of embedding representations between heterogeneous data is insufficient.

[0029] Therefore, in this paper, we mainly propose a method for strongly aligning embeddings between heterogeneous data in the same space. Strongly aligning embeddings between heterogeneous data in the same space can mean that the embeddings between heterogeneous data are closer in the same space. More specifically, we propose a method for strongly aligning embeddings between heterogeneous data in the same space by incorporating a mechanism for discriminating the type of input data into the NN and using adversarial learning.

[0030] In the embodiment of the present invention, an information processing device (hereinafter also referred to as a "learning device") that performs learning of a neural network will be described. In the learning device, learning of the neural network is performed based on learning data (learning stage). After that, in the classification device, test results are output based on the trained neural network and test data (test stage).

[0031] In the embodiments of the present invention, it is mainly assumed that the learning device and the classification device are realized by the same computer. However, the learning device and the classification device may also be realized by different computers. In such a case, a trained NN generated by the learning device is provided to the classification device. For example, the trained NN may be provided from the learning device to the classification device via a recording medium or via communication.

[0032] (1. First embodiment) First, a first embodiment of the present invention will be described. Note that in the embodiment of the present invention, it is mainly assumed that the task performed by the NN is a task of receiving language and images as input and outputting language. However, the type of task performed by the NN does not have to be limited to such a task.

[0033] For example, the task performed by the NN may be to accept multiple types of data as input, such as video (image frames and audio), or language and audio. Furthermore, the task performed by the NN may be to output an image or audio.

[0034] (Learning device configuration) An example of the configuration of a learning device according to a first embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the functional configuration of a learning device 10 according to the first embodiment of the present invention. As shown in Fig. 1, the learning device 10 according to the first embodiment of the present invention includes a heterogeneous mixture dataset 101, an input unit 102, an encoding unit 111, a decoding unit 112, a data type discrimination unit 113, an evaluation unit 121, and an update unit 122.

[0035] The learning device 10 also includes an LMM 110 as an example of an NN. The LMM 110 includes weight parameters 114. The LMM 110 further includes a first NN included in an encoding unit 111 (hereinafter also referred to as an "encoding NN"), a second NN included in a decoding unit 112 (hereinafter also referred to as a "decoding NN"), and a third NN included in a data type discrimination unit 113 (hereinafter also referred to as a "data type discrimination NN").

[0036] The encoding NN is connected to the input unit 102. More specifically, the encoding NN is connected to the subsequent stage of the input unit 102.

[0037] The decoding NN is connected to the subsequent stage of the encoding NN. Note that the subsequent stage of the encoding NN may refer to the front of the LMM 110 in the forward direction with respect to the encoding NN. Here, it is mainly assumed that the decoding NN is directly connected to the encoding NN. However, the decoding NN may also be indirectly connected to the encoding NN. Furthermore, an evaluation unit 121 is connected to the subsequent stage of the decoding NN.

[0038] The data type discrimination NN is connected in parallel to the decoding NN after the encoding NN. Specifically, the data type discrimination NN is connected to the encoding NN via a gradient inversion layer (not shown).

[0039] The gradient inversion layer is connected after the encoding NN and before the data type discrimination NN. Here, we mainly assume that the gradient inversion layer is directly connected to the encoding NN. However, the gradient inversion layer may also be indirectly connected to the encoding NN. Similarly, here, we mainly assume that the gradient inversion layer is directly connected to the data type discrimination NN. However, the gradient inversion layer may also be indirectly connected to the data type discrimination NN.

[0040] Furthermore, the fact that the data type discrimination NN is connected in parallel to the decoding NN can mean that when the stage subsequent to the encoding NN in the LMM 110 branches into multiple routes, the decoding NN is present on one of the multiple routes and the data type discrimination NN is present on the other. Furthermore, an evaluation unit 121 is connected subsequent to the data type discrimination NN.

[0041] In the following description, propagating data in the forward direction of the LMM 110 in the LMM 110 is also simply referred to as "forward propagation." On the other hand, propagating data in the LMM 110 in the direction opposite to the forward direction of the LMM 110 (hereinafter also simply referred to as "backward direction") is also simply referred to as "backward propagation."

[0042] The input unit 102, encoding unit 111, decoding unit 112, data type determination unit 113, evaluation unit 121, and update unit 122, etc., include a calculation device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and their functions can be realized by the calculation device expanding a program stored in a ROM (Read Only Memory) into a RAM and executing it.

[0043] In this case, a computer-readable recording medium on which the program is recorded may also be provided. Alternatively, these blocks may be configured by dedicated hardware or by a combination of multiple pieces of hardware. Data required for the calculation by the calculation device is stored as appropriate in a storage unit (not shown).

[0044] The heterogeneous mixture dataset 101 and the weight parameters 114 are stored in a storage unit (not shown), which may be configured with a memory such as a RAM (Random Access Memory), a hard disk drive, or a flash memory.

[0045] (weight parameter 114) The weight parameters 114 include weight parameters corresponding to the encoding NN, the decoding NN, and the data type discrimination NN. In the initial state, an initial value is set in the weight parameters 114. For example, the initial value set in the weight parameters 114 may be any value. For example, the initial value set in the weight parameters 114 may be a learned value obtained in advance by learning, or may be a random value.

[0046] (Heterogeneous Mixture Dataset 101) The heterogeneous mixture dataset 101 includes multiple types of training data (hereinafter, the training data will also be referred to as "input data") and teacher labels associated with the multiple types of input data. In the embodiment of the present invention, it is mainly assumed that the multiple types of input data are language and images. However, the type of input data may be changed appropriately depending on the task executed by the LMM 110. For example, the multiple types of input data may be video (image frames and audio), or language and audio.

[0047] (input unit 102) The input unit 102 acquires a plurality of types of input data from the heterogeneous mixture dataset 101. For example, the input unit 102 further acquires, from the heterogeneous mixture dataset 101, teacher labels corresponding to the plurality of types of input data.

[0048] The input unit 102 creates mini-batches based on multiple types of input data, and outputs the created mini-batches to the encoding unit 111. For example, the size of the mini-batches created by the input unit 102 is not particularly limited.

[0049] (encoding unit 111) The encoding unit 111 functions as a first processing unit, and obtains an embedding representation of the input data for each data type based on the multiple types of input data included in the mini-batch output from the input unit 102 and the encoding NN. Here, the encoding NN includes an encoder for each data type, but the specific configuration of the encoder for each data type is not limited. For example, an encoder corresponding to a language may include a tokenizer and an embedding layer. Also, an encoder corresponding to an image may include a combination of a Vision Transformer and a transformer with a linear layer.

[0050] More specifically, the encoding unit 111 inputs a language from among multiple types of input data to an encoder corresponding to the language, to which a weight parameter corresponding to the language from among the weight parameters 114 is set, and performs forward propagation in the forward direction of the LMM 110, thereby obtaining data output from the encoder corresponding to the language as an embedded representation of the language.

[0051] Furthermore, the encoding unit 111 inputs the image from among the multiple types of input data to an encoder corresponding to the image, to which a weight parameter corresponding to the image from among weight parameters 114 has been set, and performs forward propagation in the forward direction of the LMM 110, thereby obtaining the data output from the encoder corresponding to the image as an embedded representation of the image. The encoding unit 111 outputs the embedded representation for each data type to the decoding unit 112 and the data type determination unit 113, respectively.

[0052] (Decoding unit 112) The decoding unit 112 functions as a second processing unit, and obtains a first inference result (hereinafter also referred to as a "task inference result"), which is an inference result related to a predetermined task, based on the embedded representation for each data type output from the encoding unit 111 and the decoding NN. Here, the decoding NN is configured to include a decoder, but the specific configuration of the decoder does not need to be limited. For example, if the task performed by the decoding unit 112 is a task to output a response language, the task inference result may be the word next to the input word, and the decoder may be configured to include an arbitrary LLM.

[0053] More specifically, the decoding unit 112 inputs the embedded representation for each data type into a decoder to which a parameter corresponding to the decoder is set among the weight parameters 114, and performs forward propagation in the forward direction of the LMM 110, thereby obtaining the data output from the decoder as the task inference result. The decoding unit 112 outputs the task inference result to the evaluation unit 121.

[0054] (Data type determination unit 113) The data type discrimination unit 113 obtains a second inference result (hereinafter also referred to as a "data type inference result"), which is an inference result related to the type of input data, based on the embedded representation for each data type output from the encoding unit 111 and the data type discrimination NN. For example, the data type inference result may be a value (scalar) indicating the data type. Here, the data type discrimination NN may be configured to include a linear layer, but the specific configuration of the data type discrimination NN need not be limited. Also, as described above, a gradient inversion layer (hereinafter also referred to as a "gradient inversion layer R1") is connected to the preceding stage of the data type discrimination NN.

[0055] (Gradient inversion layer R1) The gradient inversion layer R1 inverts the sign of the gradient output from the data type discrimination NN to the encoding NN when the update unit 122 updates the weight parameters 114. This allows the encoding NN to learn so that it becomes difficult to distinguish the data type based on the embedded representation output from the encoding NN. Therefore, it can be expected that the embedded representations between multiple types of input data will become closer to each other in the same space as learning progresses.

[0056] More specifically, during forward propagation, the gradient inversion layer R1 outputs the embedded representation output in the forward direction from the encoding NN to the data type determination NN as is. Meanwhile, during backward propagation, the gradient output in the backward direction from the data type determination NN is calculated by the update unit 122. The gradient inversion layer R1 inverts the sign of the gradient by multiplying the gradient by a negative coefficient.

[0057] During forward propagation, the gradient inversion layer R1 is applied as an identity function to the embedded representation output from the encoding NN. Meanwhile, during backward propagation, the gradient inversion layer R1 is applied as a function that inverts the sign of the gradient output from the data type discrimination NN. For example, the function that inverts the sign of the gradient can be expressed as the following equation (1).

[0058]

number

[0059] Here, x on the left side of equation (1) is the input to gradient inversion layer R1, and R1(x) is the output from gradient inversion layer R1. I on the right side of equation (1) is a unit matrix. "-q(t)" on the right side of equation (1) is an example of a negative coefficient multiplied by the gradient output in the reverse direction from the data type discrimination NN. q(t) is a coefficient according to the number of learning steps t, and the specific form of q(t) is not particularly limited.

[0060] For example, q(t) may be 1. In this case, the application of the gradient inversion layer R1 always inverts the sign of the gradient. Alternatively, q(t) may be a coefficient that increases in stages from 0 according to the number of learning steps t. This allows the amount of inversion of the sign of the gradient to increase in stages.

[0061] (Evaluation Section 121) The evaluation unit 121 obtains an evaluation result based on the task inference result and the data type inference result. More specifically, the evaluation unit 121 calculates a first loss (hereinafter also referred to as a "task evaluation result") for each mini-batch based on the task inference result and the first goal. In addition, the evaluation unit 121 calculates a second loss (hereinafter also referred to as a "data type evaluation result") for each mini-batch based on the data type inference result and the second goal.

[0062] Here, the first goal to be compared with the task inference result does not need to be limited to a specific goal, and may be the same as a goal used in a general neural network. For example, if the task performed by the decoding unit 112 is to output a response language, the first goal to be compared with the task inference result may be the word next to the input word, or may be included in the language learning data.

[0063] The loss function used to calculate the task evaluation result is not limited to a specific function and may be the same as a loss function used in a general neural network. For example, the loss function used to calculate the task evaluation result may be the cross-entropy error based on the word next to the input word and the task inference result.

[0064] Furthermore, the second goal to be compared with the data type inference result does not need to be limited to a specific goal, and may be the same goal as the goal used in a general neural network. For example, if the types of input data are image and language, the goal to be compared with the data type inference result may be a teacher label assigned to each data type, such as "0 (value indicating image)" or "1 (value indicating language)."

[0065] The loss function used to calculate the data type evaluation result is not limited to a specific function and may be the same as a loss function used in a general neural network. For example, the loss function used to calculate the data type evaluation result may be a cross-entropy error based on the teacher label and the data type inference result.

[0066] The evaluation unit 121 calculates the evaluation result by integrating the task evaluation result and the data type evaluation result. For example, if the task evaluation result is Eu, the data type evaluation result is Ev, the weight multiplied by the data type evaluation result is r, and the evaluation result is E, the evaluation unit 121 can calculate the evaluation result E based on the task evaluation result Eu, the data type evaluation result Ev, and the weight r using the following formula (2):

[0067] E = Eu + rEv (2)

[0068] In the first embodiment of the present invention, the weight r may be 1. The evaluation unit 121 outputs the calculated evaluation result to the update unit 122.

[0069] (Updated part 122) The update unit 122 updates some or all of the weight parameters of the encoding NN and some or all of the weight parameters of the data type discrimination NN based on the evaluation result. As a result, some or all of the weight parameters of the data type discrimination NN are updated so that the data type inference result approaches the teacher label. In other words, learning is performed by the data type discrimination NN so that the data type discrimination unit 113 can discriminate the type of input data.

[0070] Furthermore, as described above, the gradient inversion layer R1 inverts the sign of the gradient output from the data type discrimination NN to the encoding NN, and some or all of the weight parameters of the encoding NN are updated so that the data type inference result becomes farther from the teacher label. That is, adversarial learning is performed by the encoding NN so that the data type discrimination unit 113 cannot discriminate the type of input data. Therefore, it can be expected that the embedding representations between multiple types of input data will become closer in the same space as the learning progresses.

[0071] More specifically, the updating unit 122 may update some or all of the weight parameters of the encoding NN and some or all of the weight parameters of the data type discrimination NN by error backpropagation based on the evaluation results. Note that the updating unit 122 may update some or all of the weight parameters of the decoding NN in addition to some or all of the weight parameters of the encoding NN and some or all of the weight parameters of the data type discrimination NN. This may also allow learning by the decoding NN to progress.

[0072] Each time the weight parameter update is completed, the update unit 122 determines whether the learning termination condition is satisfied. If it is determined that the learning termination condition is not satisfied, the input unit 102 acquires the next input data, and the encoding unit 111, the decoding unit 112, the data type determination unit 113, the evaluation unit 121, and the update unit 122 execute processing based on the next input data again. On the other hand, if it is determined that the learning termination condition is satisfied, the learning is terminated.

[0073] The learning termination condition is not particularly limited and may be any condition that indicates that learning has been performed to a certain extent. Specifically, the learning termination condition may include a condition that the value of the evaluation result is smaller than a threshold. Alternatively, the learning termination condition may include a condition that the change in the value of the evaluation result is smaller than a threshold (a condition that the evaluation result has been reached).

[0074] Alternatively, the learning termination condition may include a condition that the weight parameters have been updated a predetermined number of times. Alternatively, when the accuracy (e.g., accuracy rate) of the data type discrimination NN or the decoding NN is calculated, the learning termination condition may include a condition that the accuracy is greater than a predetermined rate (e.g., 90%).

[0075] An example of the configuration of the learning device 10 according to the first embodiment of the present invention has been described above.

[0076] (Learning stage operation) Next, the flow of operations in the learning stage executed by the learning device 10 according to the first embodiment of the present invention will be described with reference to Fig. 2. Fig. 2 is a flowchart showing an example of operations in the learning stage executed by the learning device 10 according to the first embodiment of the present invention.

[0077] First, the input unit 102 acquires multiple types of input data from the heterogeneous dataset 101 and creates mini-batches (S101). Then, the input unit 102 outputs the created mini-batches to the encoding unit 111. Furthermore, the input unit 102 acquires, from the heterogeneous dataset 101, teacher labels associated with the multiple types of input data.

[0078] Next, the encoding unit 111 extracts an embedded representation for each type of input data using an encoder and weight parameters for each type of input data based on the mini-batch created by the input unit 102. Then, the encoding unit 111 outputs the embedded representation for each type of input data to the decoding unit 112 and the data type determination unit 113 (S102).

[0079] The decoding unit 112 obtains an inference result related to a predetermined task by decoding the NN based on the embedded representation output from the encoding unit 111. Then, the decoding unit 112 outputs the inference result related to the predetermined task to the evaluation unit 121 (S103). The data type determination unit 113 obtains an inference result related to the type of input data based on the embedded representation output from the encoding unit 111. The data type determination unit 113 outputs the inference result related to the type of input data to the evaluation unit 121 (S104).

[0080] The evaluation unit 121 obtains an evaluation result based on the inference result related to the task and the inference result related to the type of input data, and then outputs the evaluation result to the update unit 122 (S105).

[0081] The updating unit 122 updates the weight parameters 114 of the LMM 110 based on the evaluation result output from the evaluation unit 121 (S106). More specifically, the updating unit 122 updates some or all of the weight parameters of the encoding NN and some or all of the weight parameters of the data type discrimination NN by backpropagation based on the evaluation result. At this time, the gradient inversion layer R1 inverts the sign of the gradient output from the data type discrimination NN to the encoding NN.

[0082] If the learning end condition is not met ("NO" in S107), the operation proceeds to S101. On the other hand, if the learning end condition is met ("YES" in S107), the learning by the LMM 110 is ended.

[0083] The flow of operations in the "learning stage" executed by the learning device 10 according to the first embodiment of the present invention has been described above.

[0084] (Summary of the first embodiment) As described above, the learning device 10 according to the first embodiment of the present invention includes a data type discrimination NN that discriminates the type of input data, an encoding NN, and a gradient inversion layer provided between the data type discrimination NN and the encoding NN. As a result, learning is performed by the data type discrimination NN so that the type of input data can be discriminated, and learning is performed by the encoding NN in an adversarial manner so that the type of input data cannot be discriminated.

[0085] Therefore, with the learning device 10 according to the first embodiment of the present invention, it is expected that the embedding representations of multiple types of input data will be more strongly aligned in the same space as the learning progresses, which will have the effect of improving the task execution performance of the decoding NN.

[0086] The first embodiment of the present invention has been described above.

[0087] (2. Second Embodiment) Next, a second embodiment of the present invention will be described.

[0088] In the first embodiment of the present invention, an example was described in which learning is performed by an encoding NN such that the type of input data cannot be determined based on all embedded representations output from the encoding NN, and the embedded representations between different types of data are made similar. However, it is not always good for the embedded representations between different types of data to be made similar. In other words, better task performance may be achieved if the embedded representation is specific to the type of input data.

[0089] For example, consider a case where the task performed by a neural network is to take an image and a language (e.g., a command language) as input and output a language (e.g., a response language). In such a case, there may be embedded representations that are unrelated to the input language (e.g., an embedded representation of a blurred background part in an image). In such a case, there is no need to force the image and language embeddings to be strongly aligned in the same space.

[0090] Therefore, in the second embodiment of the present invention, an example will be described in which the evaluation unit 121 assigns a weight to the data type inference result output from the data type discrimination unit 113 according to the similarity between different types of data, and performs evaluation based on the weighted data type inference result.

[0091] (Learning device configuration) An example of the configuration of a learning device according to a second embodiment of the present invention will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the functional configuration of a learning device 20 according to the second embodiment of the present invention. As shown in Fig. 3, the learning device 20 according to the second embodiment of the present invention and the learning device 10 (Fig. 1) according to the first embodiment of the present invention have the same components as each other, except that the evaluation unit 121 is replaced with an evaluation unit 221. Therefore, in the following description, the functions of the evaluation unit 221 will be mainly described, and detailed descriptions of the functions of the other components will be omitted.

[0092] In the first embodiment of the present invention, the evaluation unit 121 integrates the task evaluation result Eu calculated based on the task inference result output from the decoding unit 112 and the data type evaluation result Ev calculated based on the data type inference result output from the data type determination unit 113, and outputs the evaluation result E. In this case, the first embodiment of the present invention assumes that the weight r multiplied by the data type evaluation result Ev is fixed, as shown in the above equation (2). In the second embodiment of the present invention, the evaluation unit 221 adjusts the weight r.

[0093] More specifically, the evaluation unit 221 calculates the similarity of the embedded representations between data in a mini-batch, and controls the weight r to be multiplied by the data type evaluation result Ev based on the calculated similarity of the embedded representations. For example, the evaluation unit 221 determines combinations of heterogeneous data from the data in the mini-batch based on the teacher labels, and also determines combinations of homogeneous data.

[0094] The evaluation unit 221 sequentially selects data from within the mini-batch, sets it as reference data, and calculates the similarity between the embedded representation of data of a type different from the reference data and the embedded representation of the reference data as the similarity between the embedded representations of the heterogeneous data.The evaluation unit 221 also calculates the similarity between the embedded representation of data of the same type as the reference data and the embedded representation of the reference data as the similarity between the embedded representations of the homogeneous data.

[0095] The evaluation unit 221 calculates the relative value of the sum of the similarities of the embedded expressions between different types of data to the sum of the similarities of the embedded expressions between the same types of data, and the larger the relative value, the larger the weight r corresponding to that reference data is set, thereby strengthening the degree of similarity between the embedded expressions.

[0096] On the other hand, the smaller the relative value, the smaller the weight r corresponding to that reference data is set by the evaluation unit 221, thereby weakening the degree of similarity between the embedded expressions. The processing procedure will be described in more detail below.

[0097] Here, all embeddings are e∈R N×d Let t∈N be the teacher labels assigned to each type of input data. N where N is the batch size and d is the dimension of the embedding representation. First, the evaluation unit 221 calculates a cosine similarity C∈R as an example of similarity for all combinations of embedding representations e in the mini-batch using the following formula (3): N×N Calculate.

[0098] ∀i, j∈[1,...,N], C ij =e i ·e j T / (||e i ||2·||e j ||2) ··· (3)

[0099] Next, the evaluation unit 221 applies a softmax function to each row of the cosine similarity C to create a matrix A in which the cosine similarity C is transformed so that the sum of each row is 1. However, if the cosine similarity (Cii , i∈[1,...,N]) is not subject to the softmax function, and A ii =0.

[0100] Then, the evaluation unit 221 applies the following formula (4) to each row of the matrix A to calculate the ratio a of the sum of the cosine similarities between the embedded representation of data of a type different from that of the reference data i and the embedded representation of the reference data i to the sum of the cosine similarities between the embedded representation of data of the same type as the data type of the reference data i and the embedded representation of the reference data i. i is calculated as an example of a relative value.

[0101] ∀i∈[1,...,N], a i =Σ j∈[1、...、N] A ij I[t i ≠t j ] / Σ j∈[1、...、N] A ij I[t i =t j ] ···(4)

[0102] Here, I is an indicator function, and if the condition in [ ] is satisfied, the value of the indicator function I is 1, and if the condition in [ ] is not satisfied, the value of the indicator function I is 0.

[0103] The denominator of the above formula (4) corresponds to the sum of the cosine similarities between the embedded representations of the reference data and the other data when the other data is of the same type as the reference data. On the other hand, the numerator of the above formula (4) corresponds to the sum of the cosine similarities between the embedded representations of the reference data and the other data when the other data is of a different type from the reference data. In other words, the more similar the reference data is to the same type of data, the higher the proportion a i The more similar the reference data is to the heterogeneous data, the smaller the ratio a i becomes larger.

[0104] Finally, the evaluation unit 221 calculates the proportion a of data having the same teacher label t (i.e., homogeneous data).i The ratio a is calculated by equation (5) so that the average value of i Adjust the adjusted rate a i , the weight r i It is calculated as follows.

[0105] ∀i∈[1,...,N], r i =a i Σ j∈[1、...、N] I[t i =t j ] / Σ j∈[1、...、N] a j I[t i =t j ] ···(5)

[0106] The evaluation unit 221 uses the weight r calculated in this way. i The evaluation result E can be calculated by substituting the weight r calculated by the evaluation unit 221 with reference to FIG. i An example of this will be described.

[0107] FIG. 4 shows the weight r calculated by the evaluation unit 221. i 4 is a diagram showing an example of a feature space F1. Referring to FIG. 4, a feature space F1 is shown. Here, it is assumed that the feature space F1 is a two-dimensional space. The closer the embedded representations are to each other in the feature space F1, the stronger the similarity between the embedded representations.

[0108] △ indicates the position in feature space F1 of the embedding representation of data type 1. Also, ○ indicates the position in feature space F1 of the embedding representation of data type 2, which is different from data type 1. The numbers next to each ○ and △ indicate the weight r of that data. i 4, the evaluation unit 221 calculates the weight r of the data of the embedding representation that is highly similar to the embedding representation of the heterogeneous data. i It can be seen that the calculation is large.

[0109] An example of the configuration of the learning device 20 according to the second embodiment of the present invention has been described above.

[0110] (Learning stage operation) Next, the flow of operations in the "learning stage" performed by the learning device 20 according to the second embodiment of the present invention will be described. Here, the example of operations in the learning stage performed by the learning device 20 according to the second embodiment of the present invention and the example of operations in the learning stage performed by the learning device 10 according to the first embodiment of the present invention have S101 to S104, S106, and S107 in common, but differ in S105. Therefore, the following description will mainly focus on S105, and detailed descriptions of S101 to S104, S106, and S107 will be omitted.

[0111] The evaluation unit 221 obtains an evaluation result based on the inference result related to the task and the inference result related to the type of input data. Then, the evaluation unit 221 outputs the evaluation result to the update unit 122 (S105). More specifically, the evaluation unit 221 integrates the task evaluation result calculated based on the task inference result output from the decoding unit 112 and the data type evaluation result calculated based on the data type inference result output from the data type determination unit 113, and outputs the evaluation result.

[0112] In this case, in the first embodiment of the present invention, it is assumed that the weight r multiplied by the data type evaluation result Ev is fixed, as shown in the above formula (2). In the second embodiment of the present invention, the evaluation unit 221 adjusts the weight r. More specifically, the evaluation unit 221 controls the weight r multiplied by the data type evaluation result Ev based on the similarity between the embedded expressions.

[0113] The flow of operations in the "learning stage" executed by the learning device 20 according to the second embodiment of the present invention has been described above.

[0114] (Summary of the second embodiment) As described above, according to the second embodiment of the present invention, it is possible to achieve the same effects as those achieved by the first embodiment of the present invention.

[0115] Furthermore, according to the second embodiment of the present invention, the greater the similarity between embedded expressions of different types of data, the greater the weight multiplied by the data type evaluation result, thereby strengthening the degree of similarity between the embedded expressions.The greater the similarity between embedded expressions of the same type of data, the smaller the weight multiplied by the data type evaluation result, thereby weakening the degree of similarity between the embedded expressions.This has the effect of further improving the task execution performance of the decoding NN.

[0116] The second embodiment of the present invention has been described above.

[0117] (3. Hardware configuration example) Next, an example of the hardware configuration of the learning device 10 according to the first embodiment of the present invention will be described. Note that the hardware configuration of the learning device 20 according to the second embodiment of the present invention can also be realized in the same way as the hardware configuration of the learning device 10 according to the first embodiment of the present invention.

[0118] Below, an example of the hardware configuration of an information processing device 900 will be described as an example of the hardware configuration of a learning device 10 according to the first embodiment of the present invention. Note that the example of the hardware configuration of the information processing device 900 described below is merely one example of the hardware configuration of the learning device 10. Therefore, the hardware configuration of the learning device 10 may be such that unnecessary components are deleted from the hardware configuration of the information processing device 900 described below, or new components are added.

[0119] 5 is a diagram showing the hardware configuration of an information processing device 900 as an example of the learning device 10 according to the first embodiment of the present invention. The information processing device 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, a host bus 904, a bridge 905, an external bus 906, an interface 907, an input device 908, an output device 909, a storage device 910, and a communication device 911.

[0120] The CPU 901 functions as an arithmetic processing unit and control unit, and controls the overall operation of the information processing device 900 in accordance with various programs. The CPU 901 may also be a microprocessor. The ROM 902 stores programs used by the CPU 901, calculation parameters, etc. The RAM 903 temporarily stores programs used in the execution of the CPU 901, parameters that change as appropriate during the execution, etc. These are interconnected by a host bus 904 that is composed of a CPU bus, etc.

[0121] The host bus 904 is connected to an external bus 906, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 905. It is not necessary to configure the host bus 904, bridge 905, and external bus 906 separately, and these functions may be implemented on a single bus.

[0122] The input device 908 is composed of input means such as a mouse, keyboard, touch panel, buttons, microphone, switches, and levers that allow the user to input information, and an input control circuit that generates an input signal based on the user's input and outputs it to the CPU 901. By operating this input device 908, the user operating the information processing device 900 can input various data to the information processing device 900 and instruct the information processing device 900 to perform processing operations.

[0123] The output device 909 includes, for example, a display device such as a CRT (Cathode Ray Tube) display device, a liquid crystal display (LCD) device, an OLED (Organic Light Emitting Diode) device, or a lamp, and an audio output device such as a speaker.

[0124] The storage device 910 is a device for storing data. The storage device 910 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, and a deletion device for deleting data recorded on the storage medium. The storage device 910 is configured, for example, with an HDD (Hard Disk Drive). This storage device 910 drives a hard disk and stores programs executed by the CPU 901 and various data.

[0125] The communication device 911 is, for example, a communication interface configured with a communication device for connecting to a network, etc. The communication device 911 may be compatible with either wireless communication or wired communication.

[0126] An example of the hardware configuration of the learning device 10 according to the first embodiment of the present invention has been described above.

[0127] (4. Summary) Although the preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, the present invention is not limited to these examples. It is clear that a person skilled in the art to which the present invention pertains can conceive of various modifications and alterations within the scope of the technical ideas set forth in the claims, and it is understood that these also naturally fall within the technical scope of the present invention.

[0128] In the first and second embodiments of the present invention, it is primarily assumed that there are multiple types of training data, and that the training data is images and language. However, as already explained, the type of training data need not be particularly limited. For example, as long as embedded representations appropriate for the type of training data are extracted, data other than images and language may also be used as training data. For example, the training data may include video (image frames and audio), audio, or other time-series data. Furthermore, the first and second embodiments of the present invention may also be applied to an LMM that uses three or more types of training data as input.

[0129] In the second embodiment of the present invention, the case where the cosine similarity between embedded representations is used as the similarity between embedded representations has been mainly described (Equation (3)). However, the similarity between embedded representations does not have to be the cosine similarity between embedded representations. For example, the similarity between embedded representations may be the Euclidean distance or inner product between embedded representations. Alternatively, other indices may be used as an index of the similarity between embedded representations.

[0130] In the second embodiment of the present invention, a case is mainly assumed in which the ratio of the sum of the similarities of the embedded expressions between heterogeneous data to the sum of the similarities of the embedded expressions between homogeneous data is used as an example of a relative value of the similarity of the embedded expressions between heterogeneous data to the sum of the similarities of the embedded expressions between homogeneous data. However, instead of the ratio, the difference obtained by subtracting the sum of the similarities of the embedded expressions between homogeneous data from the sum of the similarities of the embedded expressions between heterogeneous data may be used. Furthermore, instead of the sum, the maximum value may be used. [Explanation of symbols]

[0131] 10, 20 Learning Device 101 Heterogeneous Datasets 102 Input section 110 LMM 111 Encoding section 112 Decoding unit 113 Data type discrimination unit 114 Weight Parameter 121, 221 Evaluation Department 122 Update section

Claims

1. a first processing unit configured to output an embedded representation of the input data based on the input data and a first neural network; a second processing unit that outputs a first inference result, which is an inference result related to a predetermined task, based on the embedded representation and a second neural network connected downstream of the first neural network; a data type determination unit that outputs a second inference result, which is an inference result related to the type of the input data, based on the embedded representation and a third neural network connected downstream of the first neural network; an evaluation unit that outputs an evaluation result based on the first inference result and the second inference result; an updating unit that updates some or all of the weight parameters of the first neural network and some or all of the weight parameters of the third neural network based on the evaluation result; An information processing device comprising:

2. the updating unit updates some or all of the weight parameters of the second neural network based on the evaluation result. The information processing device according to claim 1 .

3. the updating unit updates some or all of the weight parameters of the first neural network and some or all of the weight parameters of the third neural network by a backpropagation algorithm based on the evaluation result. The information processing device according to claim 1 .

4. a gradient inversion layer is connected to the front stage of the third neural network and the rear stage of the first neural network; the gradient inversion layer inverts the sign of the gradient calculated by the backpropagation algorithm based on the evaluation result by multiplying the gradient by a negative coefficient; The information processing device according to claim 3 .

5. the evaluation unit calculates a task evaluation result based on the first inference result, calculates a data type evaluation result based on the second inference result, and calculates the evaluation result based on the task evaluation result and the data type evaluation result; The information processing device according to claim 1 .

6. the evaluation unit controls a weight by which the data type evaluation result is multiplied based on an embedded representation of the input data. The information processing device according to claim 5 .

7. The evaluation unit increases the weight as the relative value of the similarity between the embedded representations of heterogeneous data, which is the similarity between the embedded representations of data heterogeneous to the input data, increases with respect to the similarity between the embedded representations of data homogeneous to the input data and the embedded representation of the input data. The information processing device according to claim 6 .

8. The relative value is a ratio of the sum of similarities of embedding expressions between the same type of data to the sum of similarities of embedding expressions between the different types of data. The information processing device according to claim 7 .

9. The relative value is a ratio of the maximum similarity of the embedding representation between the same type of data to the maximum similarity of the embedding representation between the different types of data. The information processing device according to claim 7 .

10. The relative value is a difference obtained by subtracting the sum of the similarities of the embedding expressions between the heterogeneous data from the sum of the similarities of the embedding expressions between the homogeneous data. The information processing device according to claim 7 .

11. The relative value is a difference obtained by subtracting the maximum similarity of the embedding representation between the heterogeneous data from the maximum similarity of the embedding representation between the homogeneous data. The information processing device according to claim 7 .

12. the first processing unit is an encoding unit, the second processing unit is a decoding unit. The information processing device according to claim 1 .

13. outputting an embedded representation of the input data based on the input data and a first neural network; outputting a first inference result, which is an inference result related to a predetermined task, based on the embedded representation and a second neural network connected downstream of the first neural network; outputting a second inference result, which is an inference result related to the type of the input data, based on the embedded representation and a third neural network connected downstream of the first neural network; outputting an evaluation result based on the first inference result and the second inference result; updating some or all of the weight parameters of the first neural network and some or all of the weight parameters of the third neural network based on the evaluation results; 2. A computer-implemented information processing method, comprising:

14. Computer, a first processing unit configured to output an embedded representation of the input data based on the input data and a first neural network; a second processing unit that outputs a first inference result, which is an inference result related to a predetermined task, based on the embedded representation and a second neural network connected downstream of the first neural network; a data type determination unit that outputs a second inference result, which is an inference result related to the type of the input data, based on the embedded representation and a third neural network connected downstream of the first neural network; an evaluation unit that outputs an evaluation result based on the first inference result and the second inference result; an updating unit that updates some or all of the weight parameters of the first neural network and some or all of the weight parameters of the third neural network based on the evaluation result; A program that functions as a

Images