Training device, method, and program

Abstract

This training device comprises: an inter-sensor relationship modeling unit 30 that generates a third feature amount, which is one feature amount, from a sensor mask feature amount using a third parameter set, the sensor mask feature amount being obtained by masking a predetermined-sensor feature amount included in a restored feature amount obtained by restoring a time mask feature amount using a first parameter set trained by a first training unit 50, and generates a fourth feature amount, which is one feature amount, from the third feature amount using a fourth parameter set updated using the third parameter set; and a second training unit 60 that trains the third parameter set using the third feature amount and the fourth feature amount.

Description

Learning apparatus, method and program

[0001] The disclosed technology relates to the technology for building learning models.

[0002] The technology that uses machines to acquire and understand human behavior and events occurring in the surrounding environment is expected to have applications in systems that support people's lives. For example, if customer behavior can be automatically understood in real-world environments such as convenience stores, safety and high-quality service can be maintained even in stores with few staff or unmanned stores. Since a single sensor is insufficient to detect events over a wide area, it is necessary to utilize an integrated system of multiple sensors.

[0003] We consider training such an event detection system using weak labels. A weak label is data that only indicates whether or not a certain event exists in a set of sensor data of a certain length (for example, about one minute). When using data from multiple sensors, the cost of checking and annotating all of it is very high, so using such weak labels is practical.

[0004] A conventional method for detecting weakly labeled events using multiple sensors is the technique described in Non-Patent Document 1. In Non-Patent Document 1, by intentionally deleting a portion of the sensor information during learning and then reconstructing it, the system distills the information necessary to identify an event from the observed signal, thereby enabling cooperation between multiple sensors.

[0005] Masahiro Yasuda, 4 others, “Guided Masked Self-Distillation Modeling for Distributed Multimedia Sensor Event Analysis”, [online], April 2024, [Retrieved August 30, 2020], Internet <URL: https: / / arxiv.org / pdf / 2404.08264>

[0006] Events such as human actions can be interpreted differently depending on past circumstances, not just the moment itself. For example, consider a situation where a customer is standing in front of a cash register and their hands are not visible. Based solely on the information at this moment, it is impossible to determine what the customer is doing. However, if we know that they picked up an item at a previous time, we can use that information to determine what happened at the moment they were standing in front of the register.

[0007] Conventional weak-label event detection devices based on distributed multimodal sensors model the relationships between sensors present at the same time and utilize this to detect events; therefore, they cannot detect events that take into account the time-series connections described above.

[0008] A simple way to solve this problem is to consider a time series model in the Deep Neural Networks (DNNs) used for estimation that takes past time information as input.

[0009] However, in Non-Patent Document 1, such a simple extension is insufficient to correctly detect the time of an event in order to learn event detection based on weak labels. This is because the Multiple Instance Learning (MIL) algorithm used for weak label event detection assumes the independence of each instance (in this case, the observed signal at each time point). If information is simply connected temporally within the DNNs, the assumption of information independence between each time point is not satisfied, and learning based on MIL fails.

[0010] The disclosed technology aims to provide a learning device, method, and program that generates a model that enables the detection of events while considering the time-series connections.

[0011] A learning device according to one aspect of the disclosed technology includes: an information acquisition unit that acquires information from a plurality of sensors including at least two types of sensors; a time series modeling unit that, using a first parameter set, (1) extracts feature quantities for each sensor based on the information of the plurality of sensors; (2) generates a first feature quantity, which is a single feature quantity, from a time mask feature quantity, which is a feature quantity obtained by masking the feature quantity of a predetermined time among the extracted feature quantities; and (3) generates a second feature quantity, which is a single feature quantity, from the first feature quantity, using a second parameter set updated using the first parameter set; a first learning unit that learns the first parameter set using the first feature quantity and the second feature quantity; a third sensor relationship modeling unit that, using a third parameter set, generates a third feature quantity, which is a single feature quantity, from a sensor mask feature quantity, which is a feature quantity obtained by masking the feature quantity of a predetermined sensor within a restored feature quantity, which is a feature quantity obtained by restoring the time mask feature quantity using the first parameter set learned by the first learning unit; and generates a fourth feature quantity, which is a single feature quantity, from the third feature quantity, using a fourth parameter set updated using the third parameter set; and a second learning unit that learns the third parameter set using the third feature quantity and the fourth feature quantity.

[0012] According to the disclosed technology, it is possible to generate a model that enables the detection of events while considering the time-series connections.

[0013] Figure 1 shows an example of the functional configuration of the learning device 1. Figure 2 shows an example of the functional configuration of the time-series modeling unit 20 of the learning device 1. Figure 3 shows an example of the functional configuration of the inter-sensor relationship modeling unit 30 of the learning device 1. Figure 4 shows an example of the functional configuration of the event classification unit 40. Figure 5 shows an example of the processing procedure for the learning method. Figure 6 shows an example of the functional configuration of the estimation device 2. Figure 7 shows an example of the functional configuration of the time-series modeling unit 20 of the estimation device 2. Figure 8 shows an example of the functional configuration of the inter-sensor relationship modeling unit 30 of the estimation device 2. Figure 9 shows an example of the processing procedure for the estimation method. Figure 10 shows an example of the functional configuration of a computer.

[0014] Embodiments of the disclosed technology will be described below with reference to the drawings. Note that components having the same function are numbered identically in the drawings, and redundant explanations are omitted.

[0015] The embodiments of this disclosure will be described in detail below. In the drawings, components having the same function will be numbered the same way, and redundant explanations will be omitted.

[0016] <Character Notation> The symbol "~" (tilde) used in text should ideally be placed directly above the following character, but due to limitations in text notation, it is placed immediately before the character. In mathematical formulas, these symbols are placed in their original position, that is, directly above the character. For example, "~φ" is represented in mathematical formulas as follows: The symbol "^" (hat) used in text should ideally be placed directly above the following character, but due to limitations in text notation, it is placed immediately before the character. In mathematical formulas, these symbols are placed in their original position, that is, directly above the character. For example, "^z" is represented in mathematical formulas as follows: When superscripts and subscripts are placed vertically next to a character, the subscript should ideally be placed directly below the superscript. However, due to limitations in text notation, the subscript is placed to the right of the superscript. In mathematical formulas, these symbols are written in their original positions. For example, "W Q i " is represented in mathematical formulas by the following equation. [Technical Background] This disclosure involves deleting a portion of sensor information during training and then reconstructing it using the data after the deletion. Through this reconstruction, training is performed to extract features that are unaffected by sensor deletions (Sensor-masking invariant training; S-MIT). This extends the modeling of acoustic signals through the reconstruction of masked acoustic features (see, for example, Reference 1) to the modeling of sequences of multiple sensors (i.e., at a higher level).

[0017] (Reference 1) D. Niizumi, et al., “Masked Modeling DUO: Learning Representations by Encouraging Both Networks to Model the Input”, ICASSP, 2023 <Multi-Head Attention: MHA> Multi-Head Attention (MHA) is a mechanism originally proposed in the field of natural language processing (see, for example, Reference 2).

[0018] (Reference 2) A. Vaswaniet al., ”Attention is All you Need”, NIPS, 2017 Attention is a mechanism that derives value V from memory corresponding to a query by utilizing the similarity between key K and query Q. Attention is formulated, for example, as follows. Here, d K is the dimension of K.

[0019] MHA is an extension of Equation (1) and utilizes multiple attention heads as shown in the following Equations (2) and (3). Let N be a predetermined integer of 2 or more, i = 1, …, N, and X i be an N Xi - dimensional vector. Then, Concat(X1, …, X N ) means a process of combining vectors X1, …, X N to generate a Σ i=1 N N Xi - dimensional vector. For example, if the 3 - dimensional vector A is A = (a1, a2, a3) and the 3 - dimensional vector B is B = (b1, b2, b3), then Concat(A, B) = (a1, a2, a3, b1, b2, b3).

[0020] Here, W O is a model parameter that is a matrix. W Q i , W K i , W V iThese are model parameters, which are vectors. During training, these model parameters are set to predetermined initial values ​​and are updated through learning. During detection, the learned values ​​of these model parameters are used as the model parameters.

[0021] h is a predetermined positive integer.

[0022] <Multi-Head Self-Attention:MHSA> In particular, MHA in the case of Q=K=V is called Multi-Head Self-Attention (MHSA). MHSA is used in a model called Transformer, which has brought breakthroughs in many tasks in natural language processing, to model the relationships between words in a sentence. MHSA is excellent at modeling internal relationships in sequences of various modalities.

[0023] Consider a task of event detection using information input from M microphones and N cameras distributed in space, where M and N are predetermined positive integers.

[0024] The input for this task is a time series of acoustic features obtained from M microphones and video features obtained from N cameras. This input is given by Ψ=(ψ1,...,ψ T ) is expressed as . Let τ∈{1,...,T}, ψ τ =( φ 1,τ ,...,φ S,τ ) tr φ represents the input feature at time τ. The "tr" above and to the right of the vector indicates transpose. s,τ ∈R Ds These are features corresponding to the sensor index s∈{1,...,S=M+N} and the time index τ. s This is the feature quantity φ corresponding to the sensor index s. s,τ D is the number of dimensions and is a predetermined positive integer. Below, all features are embedded in a common feature space of dimension D. s Let's assume that it can be simplified to D. That is, D s Let =D.

[0025] As will be described later, in the embodiment, the feature quantity φ s,τ As an example, the sensor identification information embedded feature quantity φ, as exemplified by equation (6), is given by equation (6). s,τ ∈R D+S This is used.

[0026] The output of this task is the activation of time-series events A=(a1,a2,...,a T ) is the case. Let τ∈{1,...,T}, a τ =( a 1,τ ,...,a C,τ ) tr ∈{0,1} C represents the event at time τ. C is the number of event classes and is a predetermined positive integer. The so-called strong labels, which are the correct labels corresponding to the activation A of the event, are given by G=(g1,...,g T )

[0027] Instead of strong labels like this, bags labels bag =(g1 bag ,...,g C bag )∈{0,1} C A weak label called may be used as the correct label. Let c ∈ {1, ..., C}, g c bag It can be defined, for example, as follows: That is, in τ∈{1, ..., T} g c,τ g = 1 c,τ If g exists c bag = 1, and for all τ ∈ {1, ..., T} g c,τ If = 0 then g c bag = 0. Here, g c,τ This value is 1 if class c exists at time τ, and 0 if it does not exist.

[0028] <selfatt-msf>To successfully integrate multiple sensors for multi-view, cross-modal event detection, it is necessary to combine and integrate appropriate sensors that hold sufficient information about the events to be detected.

[0029] However, the appropriate sensor for capturing an event varies depending on the circumstances, such as "when, where, and what kind of event occurred." For example, in video modalities, it is impossible to observe events hidden by occlusion, and it is difficult to track distant events in detail. In audio modalities, noise and distance attenuation may render some sensors unusable.

[0030] Furthermore, the most effective viewpoint and modality varies depending on the type of event occurring. For example, a camera that films a person from above is better suited to capturing hand movements than body movements. To address these changing circumstances, SelfAtt-MSF has been proposed as a mechanism for sensor integration that focuses attention on the appropriate sensor depending on the situation, such as "when, where, and what kind of event occurred."

[0031] SelfAtt-MSF is implemented as an MHSA with at least one layer, as described below. Here, the output of MHSA is obtained as S integrated features corresponding to the number of sensors. That is, ψ att τ =( ψ att 1,τ ,...,ψ att S,τ ) Furthermore, as will be discussed later, the final integrated characteristics may be obtained by applying maximum integration, average integration, etc., between these steps.

[0032] Furthermore, if the MHSA is multilayered, the Multihead operation is performed at least twice on the right-hand side of equation (5). Let k be a positive integer greater than or equal to 2, and the result of the k-1th Multihead operation on the right-hand side of equation (5) is ψ τ k-1 Therefore, the k-th Multihead operation on the right-hand side of equation (5) is (ψ τ k-1 ,ψ τ k-1 ,ψ τ k-1 ) is performed against.

[0033] By adopting a self-attention-based model that determines attention weights based on the input itself, it is expected that appropriate modalities and perspectives will be emphasized and combined in response to changing situations. Furthermore, by employing multiple heads, it is expected that multiple heads will work complementaryly for inputs with different modalities.

[0034] Furthermore, since MHSA does not distinguish between input feature sensors s, it is necessary to embed information about each sensor from an external source. For this purpose, we introduce a sensor encoding, which is expressed as follows, based on the position encoding introduced in Transformer (see reference 4 below). Here, ~φ s,τ ∈R D This is a vector representing the features of the s-th sensor. Onehot S (s) is a so-called one-hot vector, an S-dimensional vector where the s-th element is 1 and the other elements are 0.

[0035] This operation is expected to cause each linear layer of SelfAtt-MSF to function as a sensor-conditional layer.

[0036] <Sensor-masking invariant training:S-MIT> As described above, S-MIT extends the concept of using the task of restoring masked signals for representation learning to the modeling of sequences of multiple sensors. In this learning process, the embedded ψ = [φ １ , . . . ,φ ｓ For ], a mask M = [μ1, ..., μ S Apply ] as follows: Here, μ ｓ is φ ｓ This is a real vector of the same dimension, where all elements have a certain probability of being either 0 or 1. "〇" represents the Hadamard product. The masked sequence of sensor features is fused using a module such as reference 5.

[0037] (Reference 5) International Publication No. 2023 / 013043 In S-MIT, learning is performed so that the fused feature z obtained from this masked sequence of sensor features approaches the fused feature ^z obtained from the original sequence of sensor features. Loss function L S-MIT For example, the following can be used: Here, equation (8) is the result of calculating ^z - z using the mean squared error. Two networks, "Online" and "Target," are used to extract z and ^z (see, for example, reference 1).

[0038] The model parameters ξ of the "Target" network are updated using the exponential moving average of the model parameters θ of the "Online" network. Examples of exponential moving averages that can be used include the following: Here, τ represents the decay rate, where 0 < τ < 1. Equation (9) smooths out the fluctuations of ^z during the learning process, stabilizing the system's learning. The "Online" and "Target" networks output sequences of fused features z and ^z, respectively.

[0039] To minimize the loss function in equation (8) and ensure that the sequence of fused features is invariant with respect to the sensor mask, it is necessary to be able to infer missing sensor information from observations by other sensors. This is equivalent to learning the correspondence between sensors, such as how an action captured by a pixel in one camera appears on a different pixel in another camera, or how it sounds to a microphone. Therefore, this mask reconstruction task is considered suitable for modeling the relationships between sensors. Furthermore, since S-MIT models the relationships between sensors while taking into account missing sensor information, it is expected to be robust to sensor information loss due to sensor reduction or occlusion addition during inference.

[0040] Furthermore, the disclosed technology performs learning in two stages. Specifically, the disclosed technology performs time-series modeling learning and inter-sensor relationship modeling learning.

[0041] The first stage of time series model learning is self-supervised learning based on mask reconstruction for time series modeling. This involves taking sensor observation signals from the past as input and using that information to reconstruct masked features from features at a predetermined time, thereby self-supervising the encoder for time series modeling of distributed multimodal sensors.

[0042] The second stage of sensor-to-sensor relationship modeling learning is weakly supervised learning for event detection. In this learning, the encoder parameters for time series modeling learned in the previous stage are fixed, and weak label event detection is performed using the resulting embeddings as input. In the newly learned part in the second stage, there are no dependencies between each time point (instance), so it is expected that weak label learning can be performed without breaking the assumptions of MIL.

[0043] The encoder for time series modeling learned in the previous section models the characteristics of a sensor at the current time, taking into account its connection to the situation at past times, and thus models the relationships between sensors while considering the connections in the time series, which is the objective.

[0044] Events such as human actions can be interpreted differently depending on past circumstances, not just in terms of each individual moment. For example, consider a situation where a customer is standing in front of a cash register and their hands are not visible. Based solely on information at this time, it is impossible to determine what the customer is doing. However, if we know that they picked up an item at a previous time, we can use that information to determine the event at the time they were standing in front of the register. Disclosure technology makes it possible to detect specific events only by considering such time-series connections that could not be detected by conventional methods.

[0045] [Learning device and method] As shown in Figure 1, the learning device 1 according to this embodiment includes, for example, an information acquisition unit 10, a time series modeling unit 20, an inter-sensor relationship modeling unit 30, an event classification unit 40, a first learning unit 50, and a second learning unit 60.

[0046] The learning method is implemented, for example, by having each component of the learning device perform the processing shown in steps S10 to S60 in Figure 5.

[0047] The following describes each component of the learning device.

[0048] <Information Acquisition Unit 10> The information acquisition unit 10 acquires information from multiple sensors, including at least two types of sensors (step S10). The acquired information from the multiple sensors is output to the time-series modeling unit 20.

[0049] The information acquisition unit 10 is a plurality of sensors, including, for example, at least two types of sensors. The at least two types of sensors are, for example, a microphone (not shown) and a camera (not shown). The following explanation will use the case where the plurality of sensors consists of M microphones and N cameras as an example. M and N are predetermined positive integers.

[0050] Multiple sensors are distributed and arranged in space. These multiple sensors can be located anywhere in real space.

[0051] Here, "sensor information" refers to the data acquired by the sensor. For example, if the sensor is a microphone, "sensor information" refers to the sound signal acquired by the microphone. Similarly, if the sensor is a camera, "sensor information" refers to the video signal acquired by the camera.

[0052] The information acquisition unit 10 acquires information from multiple sensors at randomly selected fixed time lengths from input clips, which are input learning data.

[0053] <Time Series Modeling Unit 20> The time series modeling unit 20 uses a first parameter set to (1) extract feature quantities from each sensor based on information from multiple sensors, (2) generate a first feature quantity from time-masked feature quantities, which are feature quantities obtained by masking the feature quantities at a predetermined time from the extracted feature quantities, and (3) generate a second feature quantity from the first feature quantity using a second parameter set updated with the first parameter set (step S20). The generated first and second feature quantities are output to the first learning unit 50.

[0054] The following describes an example of the processing performed by the time-series modeling unit 20.

[0055] As shown in Figure 2, the time-series modeling unit 20 includes, for example, a frequency domain conversion unit 21, an audio signal feature extraction unit 22, an image signal feature extraction unit 23, a mask unit 24, a first sensor identification information embedding unit 25, a first self-attention sensor integration unit 26, a second sensor identification information embedding unit 27, a parameter update unit 28, and a second self-attention sensor integration unit 29.

[0056] The first parameter set mentioned above is a parameter set that includes the model parameters of the sound signal feature extraction unit 22, the model parameters of the video signal feature extraction unit 23, and the model parameters of the first self-attention sensor integration unit 26.

[0057] The frequency domain conversion unit 21 converts the M input sound signals into M frequency domain signals using frequency domain transformations such as short-time Fourier transforms. The converted M frequency domain signals are output to the sound signal feature extraction unit 22.

[0058] The frequency domain conversion unit 21 generates a frequency domain signal, which is a logarithmic Mel spectrogram, by, for example, short-time Fourier transform and filter bank processing.

[0059] The sound signal feature extraction unit 22 extracts M feature quantities ~φ corresponding to each of the M input frequency domain signals. s,τ Generate (s=1,...,M). The M generated features ~φ s,τ (s=1,...,M) is output to the mask section 24 and the second sensor identification information embedding section 27.

[0060] The sound signal feature extraction unit 22 extracts features using, for example, a DNN (Deep Neural Network) as described in Reference 3. For example, the model parameters of this DNN are pre-trained using the method described in Reference 3. These DNN model parameters are assumed to be common to M frequency domain signals. That is, each of the M frequency domain signals is input to a DNN having the same model parameters.

[0061] (Reference 3) S. Hershey et al., "CNN Architectures for Large-Scale Audio Classification", ICASSP, 2017. The video signal feature extraction unit 23 extracts N feature quantities ~φ corresponding to each of the N input video signals. s,τ Generate (s=M+1,...,S=M+N). The generated N features ~φ s,τ (s=M+1,...,S=M+N) is output to the mask section 24 and the second sensor identification information embedding section 27.

[0062] The video signal feature extraction unit 23 extracts features using, for example, a DNN described in Reference 4. For example, the model parameters of this DNN are pre-trained using the method described in Reference 4. These DNN model parameters are assumed to be common to all N video signals. That is, each of the N video signals is input to a pre-trained DNN with the same model parameters.

[0063] (Reference 4) K. He et al., "Deep Residual Learning for Image Recognition", arXiv:1512.03385, 2015 Here, let s=1,...,S=M+N, and the feature vector ~φ s,τ ∈R D This is a feature quantity that is a D-dimensional vector corresponding to time τ corresponding to sensor s.

[0064] In this way, the frequency-domain conversion unit 21, the audio signal feature amount extraction unit 22, and the video signal feature amount extraction unit 23 extract the feature amounts of each sensor based on the information of a plurality of sensors using the first parameter set.

[0065] The masking unit 24 generates a time-masked feature amount ψ that is a feature amount obtained by masking the feature amount φ of each sensor obtained from the audio signal feature amount extraction unit 22 and the video signal feature amount extraction unit 23 at a predetermined time for the series of φ. s,τ That is, the masking unit 24 generates a time-masked feature amount ψ that is a feature amount obtained by masking the feature amount at a predetermined time within the feature amount φ of the sensor. s,τm The time-masked feature amount ψ (s = 1,..., S = M + N) is output to the first sensor identification information embedding unit 25. s,τ That is, the masking unit 24 generates a time-masked feature amount ψ that is a feature amount obtained by masking the feature amount at a predetermined time within the feature amount φ of the sensor. s,τm The time-masked feature amount ψ s,τm (s = 1,..., S = M + N) is output to the first sensor identification information embedding unit 25.

[0066] τm indicates a time included in the set of times not masked at time τ ∈ {1,..., T}.

[0067] The first sensor identification information embedding unit 25 embeds the identification information of the sensor corresponding to each of the input time-masked feature amounts ψ s,τm to generate a feature amount after embedding sensor identification information corresponding to each sensor (hereinafter, also referred to as "feature amount after embedding first sensor identification information"). φ s,τ The feature amount φ after embedding sensor identification information s,τ (s = 1,..., S = M + N) is output to the first self-attention sensor integration unit 26.

[0068] The first sensor identification information embedding unit 25 generates a feature amount φ after embedding sensor identification information, for example, by embedding the identification information of the sensor according to Equation (6). s,τ In the example of Equation (6), Onehot S (s) is the identification information of sensor s. The first sensor identification information embedding unit 25 uses the time-masked feature amount ψ s,τm as ~φ in Equation (6) and performs calculations based on Equation (6). Here, even for the masked feature amount (which has become a 0 vector), embedding is performed for the 0 vector. s,τ

[0069] The first self-attention sensor integration unit 26 integrates a plurality of feature amounts φ after embedding a plurality of sensor identification information corresponding to the plurality of sensors respectively s,τ (s = 1, ..., S = M + N) to generate one feature amount (fused feature amount) ψ’ τ ∈R D . The one feature amount generated by this first self-attention sensor integration unit 26 is called the first feature amount ψ’ τ . The first feature amount ψ’ τ is output to the first learning unit 50. The first self-attention sensor integration unit 26 corresponds to Online in Reference 1. The first feature amount ψ’ τ output from the first self-attention sensor integration unit 26 corresponds to the above-described fused feature amount z

[0070] The first self-attention sensor integration unit 26 first integrates the plurality of feature amounts φ after embedding a plurality of sensor identification information according to, for example, Equation (5) s,τ (s = 1, ..., S = M + N) to generate S integrated feature amounts ψ att τ =(ψ att 1,τ , ..., ψ att S,τ ) corresponding to the number of sensors. Here, with s = 1, ..., M + N, ψ att s,τ ∈R D .

[0071] Here, ψ τ on the right side of Equation (5) is ψ τ =(φ 1,τ , ..., φ S,τ ). The Multihead in Equation (5) is defined, for example, by Equations (2) and (3). In the first processing of the Multihead in Equation (5), ψ tr is input as Q, K, V in Equations (2) and (3). That is, with Q = K = V = ψ τ , the operations in Equations (2) and (3) are performed τ .

[0072] Although the operation is performed with Q = K = V = ψ τ , the model parameter W which is a vector Q i ,W K i ,W V i Different model parameters are used for each.

[0073] The first self-awareness sensor integration unit 26 then processes S integrated feature quantities ψ att τ =( ψ att 1,τ ,...,ψ att S,τ By performing processes such as maximum pooling or average pooling on ) a single feature ψ' τ ∈R D Generates.

[0074] When performing maximum pooling processing, the first self-attention sensor integration unit 26 uses the feature quantity ψ att τ Constituting D-dimensional vector ψ att 1,τ ,...,ψ att S,τ The largest vector among them is ψ' τ Let's assume that.

[0075] Also, ψ is a D-dimensional vector. att s,τ ψ att s,τ =( ψ att s,τ,1 ,...,ψ att s,τ,D ) When performing average pooling processing, the first self-attention sensor integration unit 26 performs ψ' τ =((1 / S)Σ s=1 S ψ att s,τ,1 ,...,(1 / S)Σ s=1 S ψ att s,τ,D ψ' τ Calculate.

[0076] The model parameters of the first self-awareness sensor integration unit 26 are output to the parameter update unit 28.

[0077] In this way, the mask unit 24, the first sensor identification information embedding unit 25, and the first self-attention sensor integration unit 26 use the first parameter set to generate a first feature, which is a single feature, from time mask feature quantities, which are feature quantities obtained by masking the feature quantities of the sensor at a predetermined time.

[0078] The parameter update unit 28 updates the model parameters of the second self-attention sensor integration unit 29 (hereinafter also referred to as the "second parameter set") based on the moving exponential average of the model parameters of the first self-attention sensor integration unit 26 (see, for example, Reference 1). The updated second parameter set is output to the second self-attention sensor integration unit 29.

[0079] The second sensor identification information embedding unit 27 receives M+N feature quantities ~φ from the sound signal feature quantity extraction unit 22 and the video signal feature quantity extraction unit 23. s,τ By embedding the sensor identification information corresponding to each of (s=1,...,S=M+N), the sensor identification information embedded feature vector corresponding to each sensor (hereinafter also referred to as the "second sensor identification information embedded feature vector") φ s,τ Generates the following: Sensor identification information embedded feature quantity φ s,τ (s=1,...,S=M+N) is output to the second self-awareness sensor integration unit 29.

[0080] The second self-aware sensor integration unit 29 uses the updated parameters to embed multiple sensor identification information feature quantities φ corresponding to each of the multiple sensors. s,τ By integrating (s=1,...,S=M+N), we obtain a single feature (fused feature) ψ'. τ ∈R D This generates one feature quantity generated by the second self-awareness sensor integration unit 29, which is then converted into the second feature quantity ψ'. τ We will call it the second feature ψ'. τ This is output to the first learning unit 50. The second self-attention sensor integration unit 29 corresponds to the Target in Reference 1. The second feature quantity ψ' is output from the second self-attention sensor integration unit 29. τ This corresponds to the aforementioned fused feature ^z.

[0081] In this way, the second sensor identification information embedding unit 27, the parameter update unit 28, and the second self-attention sensor integration unit 29 generate a second feature, which is a single feature, from the first feature using the second parameter set, which is updated using the first parameter set.

[0082] <First Learning Unit 50> The first learning unit 50 learns the first parameter set using the first feature and the second feature. The learned first parameter set is output to the time series modeling unit 20 (step S50).

[0083] For example, the first learning unit 50 calculates the first loss cost using the first feature and the second feature. Equation (8) is used for the calculation. Then, the first learning unit 50 uses the first loss cost to update the first parameter set used in the time series modeling unit 20 (in the above example, the model parameters of the sound signal feature extraction unit 22, the model parameters of the video signal feature extraction unit 23, and the model parameters of the first self-attention sensor integration unit 26) for example, by backpropagation or gradient descent.

[0084] Assuming there are multiple sets of training data and corresponding correct labels, the processes in steps S10 and S50 described above are performed for each of these sets. Through this process, each parameter set, which is the model parameter, is updated, and the learned parameter set is finally obtained.

[0085] <Sensor-to-Sensor Relationship Modeling Unit 30> The sensor-to-sensor relationship modeling unit 30 generates a third feature, which is a single feature, from sensor mask features, which are features obtained by masking the features of a predetermined sensor within the restored features obtained by restoring the time mask features using the first parameter set learned by the first learning unit 50, using the third parameter set, and generates a fourth feature, which is a single feature, from the third feature, using the fourth parameter set which is updated using the third parameter set (step S30).

[0086] The features of a predetermined sensor within the reconstructed features, which are features obtained by reconstructing the time mask features using the first parameter set learned by the first learning unit 50, are S integrated features ψ generated by the processing of the frequency domain conversion unit 21, sound signal feature extraction unit 22, video signal feature extraction unit 23, mask unit 24, first sensor identification information embedding unit 25, and first self-attention sensor integration unit 26 of the time series modeling unit 20, which is set with the first parameter set learned by the first learning unit 50. att τ =( ψ att 1,τ ,...,ψ att S,τ The processing of the frequency domain conversion unit 21, the sound signal feature extraction unit 22, the video signal feature extraction unit 23, the mask unit 24, the first sensor identification information embedding unit 25, and the first self-attention sensor integration unit 26 has already been explained, so a redundant explanation will be omitted here. Note that S integrated feature quantities ψ att τ =( ψ att 1,τ ,...,ψ att S,τ ) The first self-attention sensor integration unit 26 uses one feature quantity ψ' τ ∈R D These are feature quantities generated during the process of creating [something].

[0087] The following describes an example of the processing performed by the sensor relationship modeling unit 30.

[0088] As shown in Figure 3, the inter-sensor relationship modeling unit 30 includes, for example, a sensor mask unit 34, a first sensor identification information embedding unit 35, a first self-aware sensor integration unit 36, a second sensor identification information embedding unit 37, a parameter update unit 38, and a second self-aware sensor integration unit 39.

[0089] The third parameter set mentioned above is a parameter set that includes the model parameters of the first self-awareness sensor integration unit 36.

[0090] S integrated features ψ att τ =( ψ att 1,τ ,...,ψ att S,τ The data is input to the sensor mask section 34 and the second sensor identification information embedding section 37.

[0091] The sensor mask section 34 contains S integrated feature quantities ψ att τ =( ψ att 1,τ ,...,ψ att S,τ A mask is applied to the sequence of ) according to equation (7). The M+N sensor mask features ~ψ are the feature quantities of each sensor after masking. ｍ This is output to the first sensor identification information embedding unit 35.

[0092] The first sensor identification information embedding unit 35 embeds sensor identification information, for example, using formula (6), thereby creating a feature quantity φ after sensor identification information embedding. s,τ This generates the following. In the example of expression (6), Onehot S (s) is the identification information of sensor s. The first sensor identification information embedding section 35 contains the sensor mask feature quantity ~ψ ｍ ~φ in equation (6) s,τ Then, calculations based on equation (6) are performed. Here, even the masked features (those that are 0 vectors) are embedded into the 0 vectors.

[0093] The first self-aware sensor integration unit 36 ​​embeds multiple sensor identification information corresponding to multiple sensors into a feature quantity φ s,τ By integrating (s=1,...,S=M+N), we obtain a single feature (fused feature) ψ'. τ ∈R D This generates one feature quantity generated by the first self-awareness sensor integration unit 36, which then becomes the third feature quantity ψ'. τ We will call it the first feature ψ'. τ This is output to the second learning unit 60 and the event classification unit 40. The first self-attention sensor integration unit 36 ​​corresponds to Online in Reference 1. The third feature quantity ψ' is output from the first self-attention sensor integration unit 36. τ This corresponds to the fused feature z mentioned above.

[0094] The first self-aware sensor integration unit 36 ​​first embeds a feature quantity φ of multiple sensor identification information according to, for example, equation (5). s,τ By integrating (s=1,...,S=M+N), we obtain S integrated features ψ corresponding to the number of sensors. att τ =( ψ att 1,τ ,...,ψ att S,τ ) is generated. Here, let s=1,...,M+N, ψ att s,τ ∈R D That is the case.

[0095] Here, the ψ on the right side of equation (5) τ is, ψ τ =( φ 1,τ ,...,φ S,τ ) tr Therefore, the Multihead in equation (5) is defined, for example, by equations (2) and (3). In the first processing of the Multihead in equation (5), Q, K, and V in equations (2) and (3) are ψ τ The following is input: Q=K=V=ψ τ The operations in equations (2) and (3) are then performed.

[0096] Note that Q=K=V=ψ τ The calculation is performed as follows, but the model parameter W is a vector. Q i ,W K i ,W V i Different model parameters are used for each.

[0097] The first self-awareness sensor integration unit 36 ​​then processes S integrated feature quantities ψ att τ =( ψ att 1,τ ,...,ψ att S,τ By performing processes such as maximum pooling or average pooling on ) a single feature ψ' τ ∈R D Generates.

[0098] When performing maximum pooling processing, the first self-attention sensor integration unit 36 ​​uses the feature quantity ψ att τ Constituting D-dimensional vector ψ att 1,τ ,...,ψ att S,τ The largest vector among them is ψ' τ Let's assume that.

[0099] Also, ψ is a D-dimensional vector. att s,τ ψ att s,τ =( ψ att s,τ,1 ,...,ψ att s,τ,D ) When performing average pooling processing, the first self-attention sensor integration unit 36 ​​is ψ' τ =((1 / S)Σ s=1 S ψ att s,τ,1 ,...,(1 / S)Σ s=1 S ψ att s,τ,D ψ' τ Calculate.

[0100] The model parameters of the first self-awareness sensor integration unit 36 ​​are output to the parameter update unit 28.

[0101] In this way, the sensor mask unit 34, the first sensor identification information embedding unit 35, and the first self-attention sensor integration unit 36 ​​generate a third feature, which is a single feature, from the sensor mask feature, which is a feature obtained by masking the feature of a predetermined sensor within the restored feature, which is a feature obtained by restoring the time mask feature using the first parameter set learned by the first learning unit 50, using the third parameter set.

[0102] The parameter update unit 38 updates the model parameters of the second self-attention sensor integration unit 39 (hereinafter also referred to as the "fourth parameter set") based on the moving exponential average of the model parameters of the first self-attention sensor integration unit 36 ​​(see, for example, Reference 1). The updated fourth parameter set is output to the second self-attention sensor integration unit 39.

[0103] The second sensor identification information embedding unit 37 contains S integrated feature quantities ψ att τ =( ψ att 1,τ ,...,ψ att S,τ By embedding the sensor identification information corresponding to each of the sensors, the sensor identification information embedded feature quantity corresponding to each sensor (hereinafter also referred to as the "second sensor identification information embedded feature quantity") φ s,τ Generates the following: Sensor identification information embedded feature quantity φ s,τ (s=1,...,S=M+N) is output to the second self-awareness sensor integration unit 39.

[0104] The second self-aware sensor integration unit 39 uses the updated parameters to embed multiple sensor identification information feature quantities φ corresponding to each of the multiple sensors. s,τ By integrating (s=1,...,S=M+N), we obtain a single feature (fused feature) ψ'. τ ∈R D This generates the fourth feature quantity ψ' generated by the second self-aware sensor integration unit 29. τ We will call it the fourth feature ψ'. τ This is output to the second learning unit 60. The second self-attention sensor integration unit 39 corresponds to the Target in Reference 1. The fourth feature quantity ψ' is output from the second self-attention sensor integration unit 39. τ This corresponds to the aforementioned fused feature ^z.

[0105] In this way, the second sensor identification information embedding unit 37, the parameter update unit 38, and the second self-attention sensor integration unit 39 generate a fourth feature, which is a single feature, from the third feature using the fourth parameter set, which is updated using the third parameter set.

[0106] <Event Classification Unit 40> The event classification unit 40 receives one feature quantity ψ' from the first self-awareness sensor integration unit 36. τ ∈R D Based on this, event detection is performed and event detection results are generated (step S40). The generated event detection results are output to the second learning unit 60.

[0107] As shown in Figure 4, the event classification unit 40 includes, for example, an event detection unit 41 and a time compression unit 42.

[0108] The following describes an example of the processing performed by the event classification unit 40.

[0109] The event detection unit 41 receives a third feature quantity, ψ', which is one of the features input from the first self-attention sensor integration unit 36. τ ∈R D Based on this, event detection is performed, and a C-dimensional vector a represents the event detection result. τ =( a 1,τ ,...,a C,τ ) tr This generates the event detection result a at time τ. τ =( a 1,τ ,...,a C,τ ) tr This is output to the time compression unit 42.

[0110] For example, the event detection unit 41 receives the input third feature quantity ψ' τ ∈R D By multiplying this by the linear transformation matrix M1, which is a model parameter, and then performing a sigmoid function operation on the result of the multiplication, a C-dimensional vector a is obtained. τ =( a 1,τ ,...,a C,τ ) tr This generates the following: Here, let c ∈ {1, ..., C}, where 0 ≤ a c,τ The value is ≤ 1.

[0111] The time compression unit 42 processes the event detection results a1,...,a corresponding to time 1,...,T respectively. T Based on this, an integrated event detection result a is generated for the time interval from time 1 to time T. The integrated event detection result a is output to the second learning unit 60.

[0112] The time compression unit 42 generates integrated event detection results by performing operations such as averaging, maximizing, and weighted averaging.

[0113] When performing an averaging operation, the time compression unit 42 performs, for example, a = ((1 / T)Σ τ=1 T a 1,τ ,...,(1 / T)Σ τ=1 T a C,τ The integrated event detection result a is calculated as follows.

[0114] When performing a maximum value operation, a = (max τ a 1,τ ,..., max τ a C,τ The integrated event detection result a is calculated as follows.

[0115] When performing a weighted averaging operation, the time compression unit 42 performs, for example, a = ((1 / T)Σ τ=1 T r τ a 1,τ ,...,(1 / T)Σ τ=1 T r τ a C,τ The integrated event detection result a is calculated as r. τ is the weighting coefficient, where 0 ≤ r τ ≤ 1, Σ τ=1 T r τ = 1

[0116] <Second Learning Unit 60> The second learning unit 60 learns the third parameter set using the third and fourth features (step S60).

[0117] The following describes an example of the processing performed by the second learning unit 60.

[0118] In this example, the second learning unit 60 is further input with weak labels, which are the correct labels corresponding to the input learning data, and the event detection results generated by the event classification unit 40. Below, we will explain using the case where the event detection result is the integrated event detection result a as an example. The second learning unit 60 performs the following processes (1) to (3).

[0119] The second learning unit 60 calculates the first loss cost using the third and fourth feature quantities generated by the inter-sensor relationship modeling unit 30. Equation (8) is used for the calculation.

[0120] The second learning unit 60 calculates the second loss cost, which is the binary cross-entropy loss between the integrated event detection result a and the input ground truth label, which is the weak label, using (2) the integrated event detection result a and the input ground truth label.

[0121] The second learning unit 60 calculates a third loss cost using a weighted sum from the first loss cost and the second loss cost. Using the calculated third loss cost, the second learning unit 60 updates the third parameter set used in the inter-sensor relationship modeling unit 30 (in the above example, the model parameters of the first self-aware sensor integration unit 36) and the model parameters of the event detection unit 41, for example, by backpropagation or gradient descent.

[0122] As in this example, the second learning unit 60 may also update the model parameters used for event estimation in addition to the third parameter set.

[0123] Assuming there are multiple sets of training data and corresponding correct labels, the processes from steps S10 to S60 described above are performed for each of these sets. Through this process, each model parameter is updated, and the learned model parameters are finally obtained.

[0124] Furthermore, training may be performed using so-called strong labels (labels that include event occurrence time information) as training data. In this case, the training data consists of strong labels and the event detection result for time τ generated by the event detection unit 41, a τ =( a 1,τ ,...,a C,τ ) tr The result is output to the second learning unit 60. In this case, the event classification unit 40 does not need to include the time compression unit 42. The second learning unit 60 outputs the event detection result a of the input time τ. τ Using the input correct label, the strong label, the loss between them is calculated and used as the second loss cost described above. The third parameter set used in the inter-sensor relationship modeling unit 30 (in the example above, the model parameters of the first self-aware sensor integration unit 36) and the model parameters of the event detection unit 41 are updated, for example, by backpropagation or gradient descent.

[0125] [Estimation device and method] As shown in Figure 6, the estimation device 2 according to this embodiment includes, for example, an information acquisition unit 10, a time series modeling unit 20, an inter-sensor relationship modeling unit 30, and an event classification unit 40.

[0126] The estimation method is implemented, for example, by having each component of the estimation device perform the processing shown in steps S10 to S40 in Figure 9.

[0127] The following describes each component of the estimation device. Note that the following explanation will focus on the parts that differ from the processing of learning device 1. Parts that perform the same processing as learning device 1 will not be explained again.

[0128] <Information Acquisition Unit 10> The information acquisition unit 10 of the estimation device 2 acquires information from multiple sensors, including at least two types of sensors (step S10). The acquired information from the multiple sensors is output to the time series modeling unit 20.

[0129] The information acquisition unit 10 of the estimation device 2 performs the same processing as the information acquisition unit 10 of the learning device 1.

[0130] <Time Series Modeling Unit 20> The time series modeling unit 20 of the estimation device 2 uses the first parameter set learned by the first learning unit 50 of the learning device 1 to (1) extract feature quantities for each sensor based on information from multiple sensors, and (2) generate restored feature quantities, which are feature quantities obtained by restoring time mask feature quantities, which are feature quantities obtained by masking the feature quantities for a predetermined time from the extracted feature quantities (step S20). The restored feature quantities are output to the inter-sensor relationship modeling unit 30.

[0131] The time-series modeling unit 20 of the estimation device 2 performs the same processing as the frequency domain conversion unit 21, sound signal feature extraction unit 22, video signal feature extraction unit 23, mask unit 24, first sensor identification information embedding unit 25, and first self-attention sensor integration unit 26 of the time-series modeling unit 20 of the learning device 1.

[0132] Therefore, the time-series modeling unit 20 of the estimation device 2 may consist only of a frequency domain conversion unit 21, an audio signal feature extraction unit 22, an image signal feature extraction unit 23, a mask unit 24, a first sensor identification information embedding unit 25, and a first self-attention sensor integration unit 26, as illustrated in Figure 7.

[0133] It is assumed that the model parameters of the sound signal feature extraction unit 22, the video signal feature extraction unit 23, and the first self-attention sensor integration unit 26 of the estimation device 2 are set to model parameters determined by the first parameter set learned by the first learning unit 50 of the learning device 1.

[0134] The first self-attention sensor integration unit 26 of the time series modeling unit 20 of the estimation device 2 performs the same processing as the first self-attention sensor integration unit 26 of the time series modeling unit 20 of the learning device 1, and integrates S integrated feature quantities ψ corresponding to the number of sensors. att τ =( ψ att 1,τ ,...,ψ att S,τ ) generates S integrated feature quantities ψ corresponding to the number of sensors generated. att τ =( ψ att 1,τ ,...,ψ att S,τ ) is output to the inter-sensor relationship modeling unit 30. That is, the first self-attention sensor integration unit 26 of the time series modeling unit 20 of the estimation device 2 outputs S integrated feature quantities ψ att τ =( ψ att 1,τ ,...,ψ att S,τ By performing processes such as maximum pooling or average pooling on ) a single feature ψ' τ ∈R D The process of generating it will not be performed.

[0135] <Sensor Relationship Modeling Unit 30> The sensor relationship modeling unit 30 of the estimation device 2 generates a third feature, which is a single feature, from sensor mask features, which are features obtained by masking the features of a predetermined sensor within the restored features, which are features obtained by restoring the time mask features using the first parameter set learned by the first learning unit 50 (step S30). The third feature is output to the event classification unit 40.

[0136] The sensor-to-sensor relationship modeling unit 30 of the estimation device 2 performs the same processing as the sensor mask unit 34, the first sensor identification information embedding unit 35, and the first self-attention sensor integration unit 36 ​​of the sensor-to-sensor relationship modeling unit 30 of the learning device 1.

[0137] Therefore, the sensor-to-sensor relationship modeling unit 30 of the estimation device 2 may consist only of a sensor mask unit 34, a first sensor identification information embedding unit 35, and a first self-aware sensor integration unit 36, as illustrated in Figure 8.

[0138] It is assumed that the model parameters of the first self-attention sensor integration unit 36 ​​of the estimation device 2 are set to model parameters determined by the third parameter set learned by the second learning unit 60 of the learning device 1.

[0139] <Event Classification Unit 40> The event classification unit 40 of the estimation device 2 receives one feature quantity ψ' from the first self-attention sensor integration unit 36. τ ∈R D Based on this, event detection is performed and event detection results are generated (step S40).

[0140] As shown in Figure 4, the event classification unit 40 of the estimation device 2 includes, for example, an event detection unit 41 and a time compression unit 42.

[0141] The event classification unit 40 of the estimation device 2 performs the same processing as the event classification unit 40 of the learning device 1.

[0142] In this way, by using the parameter set (trained model) obtained by the learning device 1, it becomes possible to detect events while considering the connections in time series.

[0143] [Variations] The specific configuration of the embodiments of the disclosed technology is not limited to the configuration described above. The specific configuration of the embodiments of the disclosed technology can be modified as appropriate, without departing from the spirit of the embodiments of the disclosed technology.

[0144] The various processes described in the embodiments of the disclosed technology may be performed not only in chronological order according to the order described, but also in parallel or individually as required by the processing capacity of the device performing the processes.

[0145] For example, data exchange between components of the learning device may occur directly, or it may occur via a storage unit (not shown).

[0146] Furthermore, the present invention may also include a device (terminal) for using the apparatus, system, or method of the present invention via a network (telecommunication line). The "device (terminal) for use" may be equipped with functions necessary to obtain the effects of implementing the apparatus, system, or method of the present invention (for example, control functions, decoding functions, restoration functions, input / output functions, etc.).

[0147] It goes without saying that the invention may be modified as appropriate without departing from its spirit.

[0148] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually described as being incorporated by reference.

[0149] [Programs, Recording Media] The functions realized by the components described herein may be implemented in a circuitry or processing circuitry, including a general-purpose processor, an application-specific processor, an integrated circuit, an ASIC (Application Specific Integrated Circuit), a CPU (a Central Processing Unit), conventional circuits, and / or a combination thereof, programmed to realize the functions described herein. A processor includes transistors and other circuits and is considered a circuitry or processing circuitry. A processor may be a programmed processor that executes a program stored in memory.

[0150] In this specification, circuitry, unit, and means are hardware programmed to perform or execute the functions described herein. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to perform or execute the functions described herein.

[0151] If the hardware is a processor that is considered to be a type of circuitry, then the circuitry, means, or unit is a combination of hardware and software used to constitute the hardware and / or processor.

[0152] The various processes described above can be carried out by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in Figure 10, and then causing the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc. to operate.

[0153] The program describing this process can be recorded on a computer-readable recording medium. Any computer-readable recording medium can be used, such as a magnetic recording device, optical disc, magneto-optical recording medium, or semiconductor memory.

[0154] Furthermore, this program may be distributed, for example, by selling, transferring, or lending portable recording media such as DVDs or CD-ROMs on which the program is recorded. Alternatively, the program may be stored in the storage device of a server computer and distributed by transferring the program from the server computer to other computers via a network.

[0155] A computer executing such a program may, for example, first store the program recorded on a portable storage medium or a program transferred from a server computer in its own storage device. Then, when processing is to be executed, the computer reads the program stored on its own storage medium and executes the processing according to the read program. Alternatively, the computer may directly read the program from the portable storage medium and execute the processing according to that program, or it may sequentially execute the processing according to the received program each time a program is transferred to it from a server computer. Furthermore, the processing may be executed using a so-called ASP (Application Service Provider) type service, where the processing function is realized only by issuing execution instructions and obtaining results, without transferring the program from the server computer to this computer.In addition, the processing may be executed using a so-called SaaS (Software as a Service) type service, where a part of the server computer is made available to the user along with the program. Furthermore, the term "program" in this form includes information used for processing by an electronic computer that is equivalent to a program (data, etc., that is not a direct instruction to the computer but has the property of defining the processing of the computer).

[0156] Furthermore, in this configuration, the device is configured by executing a predetermined program on a computer, but at least a part of these processes may be implemented in hardware. ​

Claims

An information acquisition unit that acquires information from multiple sensors, including at least two types of sensors, A time-series modeling unit that, using a first parameter set, (1) extracts feature quantities from each sensor based on information from the multiple sensors, (2) generates a first feature quantity from time-masked feature quantities which are feature quantities obtained by masking the feature quantities at a predetermined time from the extracted feature quantities, and (3) generates a second feature quantity which is a single feature quantity from the first feature quantity using a second parameter set which is updated using the first parameter set, A first learning unit that learns the first parameter set using the first feature and the second feature, Sensor relationship modeling unit generates a third feature, which is a single feature, from a sensor mask feature, which is a feature obtained by masking a predetermined sensor feature within the restored feature, which is a feature obtained by restoring the time mask feature using the first parameter set learned by the first learning unit, using the third parameter set, and generates a fourth feature, which is a single feature, from the third feature, using the fourth parameter set updated using the third parameter set, A second learning unit that learns the third parameter set using the third feature and the fourth feature, A learning device that includes [this]. 　 A learning device according to claim 1, The second learning unit (1) calculates a first loss cost using the first feature and the second feature, (2) calculates a second loss cost using the event estimated using the first feature and the correct label for the event, (3) calculates a third loss cost using the first loss cost and the second loss cost, and (4) updates the first parameter set using the third loss cost. Learning device. 　 The information acquisition unit performs an information acquisition step in which it acquires information from multiple sensors, including at least two types of sensors. The time series modeling unit performs the following time series modeling steps: (1) extracts feature quantities from each sensor based on information from the multiple sensors using a first parameter set; (2) generates a first feature quantity from time-masked feature quantities, which are feature quantities obtained by masking the feature quantities at a predetermined time from the extracted feature quantities; and (3) generates a second feature quantity from the first feature quantity using a second parameter set updated using the first parameter set. The first learning unit performs a first learning step in which it learns the first parameter set using the first feature and the second feature, Sensor relationship modeling step: The sensor relationship modeling unit generates a third feature, which is a single feature, from a sensor mask feature, which is a feature obtained by masking a predetermined sensor feature within a restored feature, which is a feature obtained by restoring a time mask feature using the first parameter set learned by the first learning unit, using a third parameter set, and generates a fourth feature, which is a single feature, from the third feature, using a fourth parameter set updated using the third parameter set, The second learning unit performs a second learning step in which it learns the third parameter set using the third feature and the fourth feature, Learning methods that include this. 　 A program for causing a computer to perform each step of the learning method of claim 3.

Images