Learning device, training method, and training program

Abstract

A learning device (10) according to an embodiment has a loss function calculation unit (152) and an update unit (153). The loss function calculation unit (152) calculates, for a model that outputs an estimated value on the basis of attention between a plurality of pieces of time-series data having different modalities, a loss function relating to the magnitude of attention at different times between the pieces of time-series data. The update unit (153) updates parameters of the model using the loss function so that attention at different times between the pieces of time-series data decreases.

Description

Learning device, learning method, and learning program

[0001] The present invention relates to a learning device, a learning method, and a learning program.

[0002] Conventionally, a technique using a Transformer Encoder has been known as an estimator for estimating corresponding output information from a multimodal time series (see, for example, Non-Patent Document 1). Non-Patent Document 1 describes a method of combining multiple input feature time series in the time axis direction and inputting the combined feature time series into a model including a Transformer Encoder.

[0003] During model training in the technology described in Non-Patent Document 1, training data consisting of a multimodal time series and corresponding correct output information is used. The model is trained based on the criterion of minimizing the error function between the estimation result by the model and the correct answer. Meanwhile, during inference, the multimodal time series is input to the trained model to obtain an estimation result.

[0004] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid, “VideoBERT: A Joint Model for Video and Language Representation Learning,” in Proceedings of the IEEE / CVF international conference on computer vision, 2019, pp. 7464-7473.

[0005] However, the conventional techniques have a problem in that it may be difficult to accurately train a model when the amount of training data is small.

[0006] In models that handle multimodal time series, the relationship between feature time series corresponding to each modality at the same time is important. For example, to estimate communication skills from video, relationships such as gestures during speech and synchronization of facial expressions between the speaker and listener are important. In this case, "gestures during speech," "speaker's facial expression," and "listener's facial expression" correspond to each modality.

[0007] Conventional techniques use the attention matrix of a Transformer Encoder to model the relationships between sequential data. However, when the amount of training data is limited, it is difficult to train a model using conventional techniques. This is because conventional techniques require learning a high-dimensional attention matrix from the training data alone.

[0008] In order to solve the above-mentioned problems and achieve the objectives, the learning device is characterized by having a loss function calculation unit that calculates a loss function related to the magnitude of attention at different time steps between multiple time series data of different modalities for a model that outputs an estimated value based on attention between the time series data, and an update unit that uses the loss function to update the parameters of the model so that the attention at different time steps between the time series data is reduced.

[0009] According to the present invention, even when the amount of training data is small, model training can be performed with high accuracy.

[0010] Fig. 1 is a diagram showing an example of the configuration of a learning device according to a first embodiment. Fig. 2 is a diagram explaining the structure of a model. Fig. 3 is a diagram explaining time steps corresponding to simultaneous times between modalities. Fig. 4 is a diagram explaining an attention penalty matrix. Fig. 5 is a flowchart showing the processing flow of the learning device. Fig. 6 is a diagram showing an example of a computer that executes a program.

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below with reference to the accompanying drawings, in which: FIG.

[0012] [Configuration of First Embodiment] The configuration of a learning device according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of a learning device according to the first embodiment.

[0013] The learning device 10 performs learning of a machine learning model (hereinafter simply referred to as a model). The model outputs an estimation result based on the features of input data. The model is, for example, a Transformer (Reference 1: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.).

[0014] In this embodiment, multiple time-series data of different modalities are input to a model. The model outputs an estimated value based on attention between the multiple time-series data. For example, the estimated value is a binary label. Note that the estimated value may be a discrete value other than binary, or may be a continuous quantity. The model may also be referred to as an estimator.

[0015] In conventional model training, model parameters are updated to reduce the error between the estimated value output by the model and the correct estimate. In this embodiment, based on the prior knowledge that "the relationship between corresponding data at the same time in multiple feature time series is important," a loss function (attention penalty loss) is introduced to induce an attention matrix so as to associate corresponding data at the same time.

[0016] As a result, according to this embodiment, by utilizing prior knowledge, it is possible to accurately train a model with a limited amount of training data, i.e., to improve the accuracy of a trained model.

[0017] The model of this embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram illustrating the structure of the model. Note that the model shown in Fig. 2 is an example, and this embodiment can also be applied to other models.

[0018] The model is used to estimate the merits of a salesperson's speech based on video and audio recordings of a conversation between two speakers, a seller and a buyer. The model receives input of data X shown in equation (1).

[0019]

[0020] X ｓｓ is the salesperson's speech data. ｖｓ is the seller's video data. ｖｂ is the buyer's video data. Each data is represented as a scalar or a vector.

[0021] It is assumed that each piece of video data and audio data is data extracted at a time corresponding to a certain section in which the salesperson is speaking. However, since the customer is often silent while the salesperson is speaking, the customer's audio data is not used here.

[0022] The model outputs a binary label l. A label l of "0" means that the salesperson's speech does not show sufficient empathy. A label l of "1" means that the salesperson's speech shows sufficient empathy.

[0023] That is, if the label l is "1", the utterance of the corresponding salesperson is worthy of praise. In this case, it is estimated that there was something good about the utterance.

[0024] The audio encoder uses data X ｓｓ From the time series feature value Z ｓｓ The audio encoder also extracts the data X ｖｓ From the time series feature value Z ｖｓ The video encoder extracts the data X ｖｂ From the time series feature value Z ｖｂ The audio and video encoders can extract features using, for example, pre-trained neural networks.

[0025] The model further transforms the features using a Transformer Encoder layer, and outputs a label l via a Pooling layer and a Softmax layer. The Transformer Encoder layer includes N (e.g., 6) hidden layers. The Transformer Encoder layer may also have H (e.g., 16) heads. Each hidden layer and each head is provided with a corresponding attention mechanism, where N and H are positive integers.

[0026] Salesperson's voice data X ｓｓ , Salesperson video data X ｖｓ , and customer video data X ｖｂ are data of different modalities. Similarly, the feature quantity Z ｓｓ , feature quantity Z ｖｓ , and the feature quantity Z ｖｂ are features with different modalities. Using the subscript of X of each data, the modality is expressed as m∈{ss, vs, vb}. In this case, Z ｍ ∈R Ｄ×Ｔｍ That is, the feature quantity Z ｍ is D x T ｍ where D is the dimension of the feature. ｍ is the input data X ｍ is the length in the time direction.

[0027] Returning to FIG. 1 , the learning device 10 includes a communication unit 11 , an input unit 12 , an output unit 13 , a storage unit 14 , and a control unit 15 .

[0028] The communication unit 11 performs data communication with other devices via a network. For example, the communication unit 11 is a network interface card (NIC).

[0029] The input unit 12 receives input of data. For example, the input unit 12 is an interface connected to input devices such as a mouse and a keyboard.

[0030] The output unit 13 outputs data. The output unit 13 is an interface connected to an output device such as a display and a speaker.

[0031] The storage unit 14 is a storage device such as a hard disk drive (HDD), a solid state drive (SSD), or an optical disk. Note that the storage unit 14 may also be a data-rewritable semiconductor memory such as a random access memory (RAM), flash memory, or non-volatile static random access memory (NVSRAM). The storage unit 14 stores an operating system (OS) and various programs executed by the learning device 10.

[0032] The storage unit 14 stores model information 141. The model information 141 is parameters for constructing a model. For example, the model information 141 is parameters such as a weight matrix for each layer included in a Transformer.

[0033] The control unit 15 controls the entire learning device 10. The control unit 15 is, for example, an electronic circuit such as a central processing unit (CPU), a micro processing unit (MPU), or a graphics processing unit (GPU), or an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0034] The control unit 15 also has an internal memory for storing programs that define various processing procedures and control data, and executes each process using the internal memory. The control unit 15 also functions as various processing units when the various programs run. For example, the control unit 15 functions as an estimation unit 151, a loss function calculation unit 152, and an update unit 153.

[0035] The estimation unit 151 calculates an estimated value for the input data using the model. For example, the estimation unit 151 calculates an estimated value for the input data X ｓｓ , Data X ｖｓ , and data X ｖｂ The label l is estimated from

[0036] The loss function calculation unit 152 calculates a loss function that can optimize both the estimation error and the attention penalty error of the model.

[0037] The estimation error is expressed by the error between the estimated value of the label and the correct answer. The attention penalty error is an error related to the magnitude of attention at different time steps in the time series data. In particular, the attention penalty error will be explained in detail later. Below, we will explain how the loss function calculation unit 152 calculates the loss function.

[0038] The loss function calculation unit 152 calculates the loss function using the attention matrix defined by the model in the process of calculating the estimated value. First, the attention matrix will be described.

[0039] The model is the feature Z ｓｓ , feature quantity Z ｖｓ , and the feature quantity Z ｖｂ are concatenated in the time axis direction to form an input vector Z. However, each element of the vector Z may be a matrix.

[0040]

[0041] Next, the Transformer Encoder models the association between the time steps of the input vector using an attention matrix A. The attention matrix A is a scaled dot product of the query matrix Q and the key matrix K. The query matrix Q and the key matrix K are calculated by multiplying the vector of each time step of the input vector by a weight matrix. Therefore, the query matrix Q and the key matrix K can be considered to be composed of submatrices corresponding to the data of each modality, as shown in equations (3) and (4).

[0042]

[0043]

[0044] The weight matrices used to calculate the query matrix Q and the key matrix K are parameters to be learned in each hidden layer of the Transformer Encoder. That is, the model information 141 includes the weight matrices. The update unit 153 updates the weight matrices.

[0045] Each hidden layer of the Transformer Encoder receives the vector output from the previous hidden layer. Then, the Transformer Encoder multiplies the input vector by a weight matrix in each hidden layer to calculate the query matrix Q and the key matrix K.

[0046] Here, for data m of each modality, Q ｍ , K ｍ ∈R ｄ×Ｔｍ That is, Q ｍ , K ｍ is d×T ｍ where d is the dimension of the query matrix and the key matrix. ｍ is the input data X ｍ is the length in the time direction.

[0047] The attention matrix A, defined by the scaled dot product of the query matrix Q and the key matrix K, is composed of block matrices as shown in equation (5).

[0048]

[0049] where softmax(·) is the softmax function. Block matrix A ｍ１，ｍ２ ∈R Ｔｍ１×Ｔｍ２ is modality m １ From modality m ２ This can be seen as attention to.

[0050] Inter-modality attention is calculated by the off-diagonal block matrix A ｍ１，ｍ２ (m １ ≠m ２) As shown in FIG. 3, the diagonal elements of the non-diagonal block matrix (diagonal lines parallel to the diagonal of the rectangle) correspond to time steps at the same time between different modalities. FIG. 3 is a diagram for explaining time steps at the same time between modalities.

[0051] Off-diagonal block matrix A ｖｓ，ｓｓ corresponds to the area where the vertical direction (Q) is VS and the horizontal direction (K) is SS among the nine rectangular areas that make up the rectangle in Figure 3. A straight line is drawn on the diagonal of this area. This is because the block matrix A ｖｓ，ｓｓ This is because the diagonal elements of correspond to the same time steps between different modalities.

[0052] On the other hand, the non-diagonal block matrix A ｓｓ，ｓｓ corresponds to the area where the vertical direction (Q) is SS and the horizontal direction (K) is SS among the nine rectangular areas that make up the rectangle in Figure 3. There is no straight line drawn on the diagonal line of this area. This is because the block matrix A ｓｓ，ｓｓ This is because the diagonal elements of correspond to the same time step between the same modalities.

[0053] From this, the non-diagonal block matrix A ｍ１，ｍ２ (m １ ≠m ２ ) becomes dominant, attention at different times between time series data of different modalities becomes smaller. Since the magnitude of attention is relative, in other words, the off-diagonal block matrix A ｍ１，ｍ２ (m １ ≠m ２ ) becomes dominant, attention at the same time between time series data of different modalities increases.

[0054] In other words, the model tends to follow the prior knowledge that "the relationship between corresponding data at the same time in multiple feature time series is important." Note that attention can also be rephrased as the strength of the relationship between data.

[0055] Therefore, the loss function calculation unit 152 calculates the non-diagonal block matrix A ｍ１，ｍ２(m １ ≠m ２ ) is calculated so that the loss function becomes smaller as the diagonal elements of

[0056] First, the loss function calculation unit 152 defines the attention penalty matrix shown in equations (6) and (7).

[0057]

[0058]

[0059] Here, W ｍ１，ｍ２ ｉ，ｊ is the block matrix W ｍ１，ｍ２ ∈R Ｔｍ１×Ｔｍ２ is the (i, j)-th component of the equation (7). σ is a hyperparameter. Also, from equation (7), the off-diagonal components take positive values.

[0060] In addition, T ｍ１ and T ｍ２ are modalities m １ and modality m ２ In this way, the number of time slots may differ for each modality. However, at least within the range where the attention penalty matrix is ​​defined, the total length of time slots is the same across modalities.

[0061] For example, salesperson's voice data X ｓｓ (Modality ss) and salesperson video data X ｖｓ Let us consider a case where the modality ss is acquired for 12 seconds (from time t=0 to time t=12). ｓｓ is 12. On the other hand, the number of time slots for modalities T ｖｓ is 4.

[0062] That is, the time slot of modality ss includes 12 intervals from time t=0 to 1 second, 1 to 2 seconds, ..., 11 to 12 seconds. On the other hand, the time slot of modality vs includes four intervals from time t=0 to 3 seconds, 3 to 6 seconds, 6 to 9 seconds, and 9 to 12 seconds.

[0063] According to equation (7), when (i, j) = (12, 4), the numerator in the parentheses of exp (i / T ｍ１ -j / T ｍ２ ) ２ (12 / 12-4 / 4) ２ = 0, and W ｓｓ，ｖｓ １２，４ Is W ｍ１，ｍ２ ｉ，ｊ The value that can be taken by (i, j) is the smallest possible value, 0. Note that the component corresponding to (i, j) = (12, 4) is a diagonal component.

[0064] Furthermore, according to equation (7), when (i, j) = (11, 4), the numerator in the parentheses of exp (i / T ｍ１ -j / T ｍ２ ) ２ (11 / 12-4 / 4) ２ = 1 / 144, which is a small value, ｓｓ，ｖｓ １１，４ Is W ｍ１，ｍ２ ｉ，ｊ The value that can be taken by (i, j) is the smallest value. Note that the component corresponding to (i, j) = (11, 4) is a non-diagonal component close to the diagonal component.

[0065] Furthermore, according to equation (7), when (i, j) = (3, 4), the numerator in the parentheses of exp (i / T ｍ１ -j / T ｍ２ ) ２ (3 / 12-4 / 4) ２ = 81 / 144, which is a large value, ｓｓ，ｖｓ ３，４ Is W ｍ１，ｍ２ ｉ，ｊ It is to be noted that the element corresponding to (i, j)=(3, 4) is a non-diagonal element far from the diagonal element.

[0066] In this way, the values ​​of the diagonal elements of the off-diagonal block matrix of the attention penalty matrix are 0, and the values ​​increase as they move away from the diagonal elements. Therefore, when the attention penalty matrix is ​​represented as a heat map, a gradation like that shown in Figure 4 appears. Figure 4 is a diagram explaining the attention penalty matrix.

[0067] The loss function calculation unit 152 sums up the losses of each head h∈{1, ..., H} of each layer n∈{1, ..., N} of the Transformer, and calculates the loss function L ｓｇ is calculated as in equation (8).

[0068]

[0069] Furthermore, the loss function calculation unit 152 calculates a loss function L that represents the estimation error, i.e., the error between the estimation result and the correct label. ｌａｂｅｌ At the same time, the loss function L is calculated as shown in equation (9).

[0070]

[0071] λ ｓｇ Is, L ｌａｂｅｌ L against ｓｇ is a regularization parameter, which is set in advance as a hyperparameter. ｌａｂｅｌ is, for example, the cross-entropy function.

[0072] In this way, the loss function calculation unit 152 calculates a loss function relating to the magnitude of attention at different times in the time series data for a model that outputs an estimated value based on attention between multiple time series data of different modalities.

[0073] The update unit 153 uses a loss function to update the parameters of the model so that attention at different times in the time-series data is reduced.

[0074] 5 is a flowchart showing the processing flow of the learning device 10. As shown in FIG. 5, first, the learning device 10 initializes parameters (step S101). The parameters here are, for example, a weight matrix of a Transformer, and are included in the model information 141.

[0075] Next, the learning device 10 repeats the process of step S2 and updates the parameters until a condition is met, such as the number of iterations exceeding a certain number, or the amount of parameter update falling below a threshold.

[0076] The learning device 10 performs forward calculation (step S102). That is, the learning device 10 calculates estimated values ​​using a model for a plurality of input time-series data of different modalities.

[0077] Next, the learning device 10 calculates the estimation error based on the estimation result (step S103). The estimation error is calculated by, for example, L ｌａｂｅｌ is.

[0078] Furthermore, the learning device 10 calculates an attention penalty loss (attention penalty error) based on the attention matrix (step S104). The attention penalty error is calculated by, for example, L ｓｇ is.

[0079] Here, the learning device 10 performs back propagation (step S105) to optimize a loss function (e.g., L in equation (9)) that combines the estimation error and the attention penalty error, and updates the model parameters (step S106).

[0080] [Effects of the First Embodiment] As described above, the loss function calculation unit 152 calculates a loss function relating to the magnitude of attention at different times in the time-series data for a model that outputs an estimate based on attention between multiple time-series data of different modalities. The update unit 153 uses the loss function to update the parameters of the model so that the attention at different times in the time-series data decreases.

[0081] In this way, by performing training using the attention penalty error, the learning device 10 can generate in the model a tendency for the relationship between corresponding data at the same time to become stronger in the feature time series. As a result, according to this embodiment, the model can be trained accurately even when the amount of training data is small.

[0082] The loss function calculation unit 152 calculates an attention matrix, each component of which corresponds to a combination of two modalities and a combination of two times, based on multiple time-series data, assigns a weight to each component of the attention matrix so that the weight increases the further apart the corresponding two times are, and calculates a loss function including the weighted attention matrix. The update unit 153 updates the model parameters so that each component of the weighted attention matrix decreases.

[0083] In particular, the loss function calculation unit 152 calculates the scaled dot product of the query matrix and key matrix input to one or more layers included in the Transformer model as an attention matrix, multiplies each element of the non-diagonal block matrix that makes up the attention matrix by an attention penalty matrix that assigns a weight that increases the further apart the two corresponding times are, and calculates a loss function that includes the norm of the product of the attention matrix and the attention penalty matrix.

[0084] As explained in equation (5), each block matrix of the attention matrix A corresponds to a combination of two modalities. Also, each element of the block matrix corresponds to a combination of time represented by a time slot. For example, the block matrix A ｖｓ，ｓｓ corresponds to the combination of modality vs and modality ss. ｖｓ，ｓｓ Each element of corresponds to a combination of two time slots assigned to a row and a column, respectively.

[0085] Furthermore, as explained in equations (6), (7), and (8), the attention matrix A is weighted by the attention penalty matrix W so that the greater the distance between two times (e.g., the i-th time slot and the j-th time slot) the greater the weight.

[0086] As a result, according to this embodiment, training of the Transformer in particular can be performed with high accuracy.

[0087] [System Configuration, etc.] The components of each device shown in the figures are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown, and all or part of the devices can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic.

[0088] Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method.In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[0089] [Program] In one embodiment, the study device 10 can be implemented by installing a program that executes the above-described study process as package software or online software on a desired computer. For example, by executing the above-described program on an information processing device, the information processing device can function as the study device 10. The information processing device referred to here includes desktop and notebook personal computers. Other information processing devices also include mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as slate terminals such as PDAs (Personal Digital Assistants).

[0090] The learning device 10 can also be implemented as a server device that provides services related to the above processing to a terminal device used by a user as a client. For example, the server device is implemented as a server device that receives parameters of a model before updating as input and outputs parameters of a model after updating as output. In this case, the server device may be implemented as a web server or as a cloud that provides services related to the above processing through outsourcing.

[0091] 6 is a diagram showing an example of a computer that executes a program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0092] The memory 1010 includes a read-only memory (ROM) 1011 and a random access memory (RAM) 1012. The ROM 1011 stores a boot program such as a basic input / output system (BIOS). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0093] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the program that defines each process of the learning device 10 is implemented as a program module 1093 in which computer-executable code is written. The program module 1093 is stored, for example, on the hard disk drive 1090. For example, a program module 1093 for executing processes similar to those of the functional configuration of the learning device 10 is stored on the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0094] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes the processing of the above-described embodiment.

[0095] The program module 1093 and program data 1094 may not necessarily be stored in the hard disk drive 1090, but may also be stored in a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0096] REFERENCE SIGNS LIST 10 Learning device 11 Communication unit 12 Input unit 13 Output unit 14 Storage unit 15 Control unit 141 Model information 151 Estimation unit 152 Loss function calculation unit 153 Update unit

Claims

1. A learning device comprising: a loss function calculation unit that calculates a loss function relating to the magnitude of attention at different times in a plurality of time series data of different modalities for a model that outputs an estimated value based on attention between the time series data; and an update unit that uses the loss function to update parameters of the model so that attention at different times in the time series data is reduced.

2. The learning device described in claim 1, characterized in that the loss function calculation unit calculates an attention matrix, each component of which corresponds to a combination of two modalities and a combination of two times, based on the multiple time series data, weights each component of the attention matrix so that the weight increases the further apart the corresponding two times are, and calculates the loss function including the weighted attention matrix, and the update unit updates the parameters of the model so that each component of the weighted attention matrix becomes smaller.

3. The learning device described in claim 2, characterized in that the loss function calculation unit calculates the attention matrix as a scaled dot product of a query matrix and a key matrix input to one or more layers included in the model which is a Transformer, multiplies each element of the non-diagonal block matrix that constitutes the attention matrix by an attention penalty matrix that assigns a weighting that is larger the greater the distance between two corresponding times, and calculates the loss function including the norm of the product of the attention matrix and the attention penalty matrix.

4. A learning method executed by a learning device, comprising: a loss function calculation step of calculating a loss function relating to the magnitude of attention at different times in a plurality of time series data of different modalities for a model that outputs an estimated value based on attention between the time series data; and an update step of updating parameters of the model using the loss function so that the attention at different times in the time series data is reduced.

5. A learning program that causes a computer to execute the following steps: a loss function calculation step that calculates a loss function relating to the magnitude of attention at different times in a plurality of time series data of different modalities for a model that outputs an estimated value based on attention between the time series data; and an update step that uses the loss function to update parameters of the model so that attention at different times in the time series data is reduced.