Information processing device, trained model generation device, method, and program

Abstract

This method extracts useful time-level features from different types of time-series data measured within the same time interval. [Solution] The information processing device acquires first time series data measured by a first sensor and second time series data measured by a second sensor in the same time interval as the first time series data. The information processing device inputs the first time series data and the second time series data into a pre-trained feature extraction model that has been trained in advance to maximize the similarity between a first feature extracted from the first data in the first time series data and a second feature extracted from the second data in the second time series data measured at the same time as the first data. The information processing device then extracts the first feature from the first time series data and the second feature from the second time series data. The information processing device then outputs a combination of the first feature and the second feature.

Description

[Technical Field]

[0001] This disclosure relates to an information processing device, a trained model generation device, a method, and a program. [Background technology]

[0002] Conventionally, self-supervised learning approaches are known that learn the representation of both frame images contained in a video and the audio data associated with that video (see, for example, Non-Patent Document 1). In addition, a technique is known for collecting video data of table tennis and detecting and classifying strokes (see, for example, Non-Patent Document 2). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Jenni, S., Black, A., & Collomosse, J. (2023, June). Audio-visual contrastive learning with temporal self-supervision. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 7, pp. 7996-8004). [Non-Patent Document 2] Kulkarni, KM, & Shenoy, S. (2021). Table tennis stroke recognition using two-dimensional human pose estimation. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition (pp. 4576-4584). [Overview of the project] [Problems that the invention aims to solve]

[0004] Now, let's consider a case where a certain time-series data (for example, video data) and another time-series data associated with that time-series data (for example, audio data) are acquired. For example, suppose the video data is time-series data of a table tennis match, and the audio data is time-series data of sounds collected during the table tennis match.

[0005] In this case, it is preferable that the features extracted from a single frame image that makes up the video data are similar to the features extracted from the sound collected at the time that single frame image was taken. Specifically, it is preferable that the features of a single frame image are similar to the features of the sound collected at the time that frame image was taken, rather than the features of the sound collected at a different time. Also, it is preferable that the features of a sound collected at a certain time are similar to the features of a frame image taken at that time, rather than the features of a frame image taken at a different time.

[0006] For example, in a table tennis match, the frame image capturing the moment a player hits the ball and the sound of the ball being hit at the moment the frame image was taken are sensor information measured at the same time. Therefore, even if the video data and audio data have different data formats, it is preferable that the feature quantities extracted from them be represented in the same way.

[0007] For example, features extracted from video and audio data can be useful for estimating whether or not a player is hitting the ball in a table tennis match. Therefore, if the features extracted from the frame image captured at the time a player hits the ball in a table tennis match differ significantly from the features extracted from the sound collected at the same time, it may not be possible to accurately estimate whether or not the player is hitting the ball.

[0008] Furthermore, for example, if a sound collected at a certain time within audio data corresponds to the sound of a player hitting the ball in a table tennis match, it is possible to search for frame images of other scenes in a table tennis match where the player is hitting the ball, based on the features extracted from that sound. In this case, since the features extracted from the sound of the player hitting the ball are similar to the features extracted from the frame images of the scenes where the player is hitting the ball, it becomes possible to search for frame images of other scenes where the player is hitting the ball based on the features extracted from the sound of the player hitting the ball.

[0009] Thus, the features extracted from video and audio data can be used for various purposes such as prediction, estimation, retrieval, or adjustment.

[0010] However, the technologies disclosed in Non-Patent Documents 1 and 2 are not intended to extract time-unit features from different types of time-series data. In particular, the technology disclosed in Non-Patent Document 1 is a technology that learns features from both video data and audio data in segment units representing predetermined time intervals within time-series data.

[0011] This disclosure is made in view of the above points and aims to extract useful time-unit features from different types of time-series data measured within the same time interval. [Means for solving the problem]

[0012] To achieve the above objective, the information processing device according to this disclosure includes: an acquisition unit that acquires first time-series data, which is time-series data measured by a first sensor, and second time-series data, which is time-series data measured by a second sensor in the same time interval as the first time-series data; an extraction unit that extracts the first feature of the first time-series data and the second feature of the second time-series data by inputting the first time-series data and the second time-series data to a pre-trained feature extraction model that has been trained in advance to increase the similarity between a first feature extracted from the first data in the first time-series data and a second feature extracted from the second data in the second time-series data measured at the same time as the first data; and an output unit that outputs a combination of the first feature and the second feature.

[0013] Furthermore, the trained model generation device of this disclosure includes: a training acquisition unit that acquires first training time series data, which is training time series data measured by a first sensor, and second training time series data, which is training time series data measured by a second sensor in the same time interval as the first training time series data; and a learning unit that trains a training model so that the similarity between a first feature extracted from the first training data in the first training time series data and a second feature extracted from the second training data in the second training time series data measured at the same time as the first training data is high, thereby generating a trained feature extraction model that outputs the first feature of the first time series data and the second feature of the second time series data when the first time series data and the second time series data are input.

[0014] Furthermore, the information processing method disclosed herein is an information processing method in which a computer performs the processing, acquiring first time series data which is time series data measured by a first sensor and second time series data which is time series data measured by a second sensor in the same time interval as the first time series data, inputting the first time series data and the second time series data into a pre-trained feature extraction model which has been trained in advance to increase the similarity between a first feature extracted from the first data in the first time series data and a second feature extracted from the second data in the second time series data measured at the same time as the first data, extracting the first feature from the first time series data and the second feature from the second time series data, and outputting a combination of the first feature and the second feature.

[0015] Furthermore, the method for generating a trained model according to this disclosure is a method for generating a trained model in which a computer performs the processing, which involves acquiring first training time series data, which is training time series data measured by a first sensor, and second training time series data, which is training time series data measured by a second sensor in the same time interval as the first training time series data, and training a training model so that the similarity between a first feature extracted from the first training data in the first training time series data and a second feature extracted from the second training data in the second training time series data measured at the same time as the first training data is high, thereby generating a trained feature extraction model that outputs the first feature of the first time series data and the second feature of the second time series data when the first time series data and the second time series data are input.

[0016] Furthermore, the information processing program disclosed herein is an information processing program that causes a computer to perform the following processing: acquiring first time series data, which is time series data measured by a first sensor, and second time series data, which is time series data measured by a second sensor in the same time interval as the first time series data; inputting the first time series data and the second time series data into a pre-trained feature extraction model that has been trained in advance to increase the similarity between a first feature extracted from the first data in the first time series data and a second feature extracted from the second data in the second time series data measured at the same time as the first data; extracting the first feature from the first time series data and the second feature from the second time series data; and outputting a combination of the first feature and the second feature.

[0017] Furthermore, the pre-trained model generation program of this disclosure is a pre-trained model generation program that causes a computer to execute a process that acquires first training time series data, which is training time series data measured by a first sensor, and second training time series data, which is training time series data measured by a second sensor in the same time interval as the first training time series data, and trains a training model so that the similarity between a first feature extracted from the first training data in the first training time series data and a second feature extracted from the second training data in the second training time series data measured at the same time as the first training data is high, thereby generating a pre-trained feature extraction model that outputs the first feature of the first time series data and the second feature of the second time series data when the first time series data and the second time series data are input. [Effects of the Invention]

[0018] According to the information processing device, trained model generation device, method, and program of this disclosure, useful time-unit features can be extracted from different types of time-series data measured within the same time interval. [Brief explanation of the drawing]

[0019] [Figure 1] This is a diagram illustrating the architecture of a neural network used for feature extraction. [Figure 2] This is a diagram illustrating a neural network used for classification. [Figure 3] This block diagram shows the hardware configuration of the trained model generation device according to this embodiment. [Figure 4] This block diagram shows the schematic configuration of the trained model generation device according to this embodiment. [Figure 5] This is a block diagram showing the hardware configuration of the information processing device according to this embodiment. [Figure 6] This is a block diagram showing the schematic configuration of the information processing device according to this embodiment. [Figure 7] This flowchart shows the flow of the trained model generation process in this embodiment. [Figure 8] This is a flowchart showing the information processing flow in this embodiment. [Modes for carrying out the invention]

[0020] Hereinafter, an example of an embodiment of this disclosure will be described with reference to the drawings. In each drawing, identical or equivalent components and parts are given the same reference numerals. Also, the dimensions and proportions in the drawings are exaggerated for illustrative purposes and may differ from actual proportions.

[0021] <1. Overview of the Embodiment> In this embodiment, a neural network, which is an example of a known machine learning model, is used to extract time-dependent features from different types of time-series data. The neural network for feature extraction in this embodiment is configured to output features of time-series data when different types of time-series data are input. In this embodiment, the case where the video data and audio data of a table tennis match are time-series data will be explained as an example.

[0022] <2. Method in the Embodiment> [2.1 Problem Form] v shown in the following formula represents video data composed of T frame images. x in the following formula t v represents the t-th (or time t) frame image in the video data.

[0023]

Equation

[0024] Similarly, a shown in the following formula represents audio data. x in the following formula t a represents the t-th (or time t) spectrogram data in the audio data. Note that the video data and the audio data are synchronized, and the t-th frame image x t v and the t-th spectrogram data x t a are data measured at the same time.

[0025]

Equation

[0026] In this embodiment, based on the pair (v, a) of video data and audio data, y shown in the following formula is estimated.

[0027]

Equation

[0028] (y in the above formula t s , y t p , y t o ) is a three-tuple binary label. y t s is a label indicating whether the player hit the ball at time t. y tp is the label representing the player who hit the ball at time t. t o This is a label indicating whether or not the player was aggressive at time t. Therefore, the estimation task in this embodiment is achieved by identifying a function h as expressed by the following equation.

[0029]

number

[0030] To identify the function h as shown in the above equation, table tennis match data is prepared in advance. This data is for self-supervised learning. The data set represented by the following equation is a set of video data and audio data.

[0031]

number

[0032] V i This is the video data for the i-th match, and A i is the audio data for the i-th match. Therefore, the data set in the above equation contains video and audio data for M matches. Here, v is expressed by the following equation i,j and a i,j We will introduce v i,j This is the data corresponding to the j-th segment within the video data of the i-th match, and a i,j This data corresponds to the j-th segment within the audio data of the i-th match. A segment represents a predetermined time interval within video or audio data, and the length of one segment is T.

[0033]

number

[0034] Using the above data set, the following data set S M The following N is defined. i M i This is the number of segments included.

[0035]

number

[0036] [2.2 Neural Networks for Feature Extraction] In this embodiment, a neural network for feature extraction is used. Figure 1 is a diagram illustrating the architecture of the neural network for feature extraction. As shown in Figure 1, the neural network FNN for feature extraction is the first model f v and the second model f a It is composed of the following. Note that the first model f v and the second model f a By training the trained first model f v and the trained second model f a and are generated.

[0037] As shown in Figure 1, the first model f v Video data v is input to it. Then, the first model f v When video data v is input, it outputs video feature vector Z1, which is a feature of the video data v. (g,v) and vector sequence Z v It is composed of including and .

[0038] Furthermore, as shown in Figure 1, the second model f a Audio data a is input to it. Then, the second model f a When audio data a is input, it outputs audio feature vector Z2, which is a feature of audio data a. (g,a) and vector sequence Z a It is composed of including and .

[0039] Furthermore, as shown in Figure 1, the first model f v and the second model f a Each of these consists of two types of neural networks. Specifically, the first model f is constructed using Transformer and ConvNext, which are examples of known neural networks. v and the second model f a It is composed of the following.

[0040] Specifically, as shown in Figure 1, the first model f v This is a ConvNext neural network f for extracting spatial features. s v And, a Transformer neural network f for extracting temporal features t v It is composed of the following: ConvNext neural network f s v When video data v is input to it, the ConvNext neural network f s v is a vector sequence g v,s It outputs the following. In the example shown in Figure 1, the vector sequence g v,s For example, it is composed of five vectors (each square in Figure 1 represents one vector). Vector sequence g v,s The single vector contained within is a feature vector extracted from a single frame image. In the example shown in Figure 1, the video data v consists of five frame images, and for each of these five frame images, one feature vector is extracted from one frame image.

[0041] And, as shown in Figure 1, Transformer neural network f t v For this, a ConvNext neural network f s v The vector sequence g output from v,s and vector g vThe following is input: vector g v For example, this is a vector whose components are set randomly. v,s Each of the feature vectors contained in is a feature vector of the frame image, whereas vector g v This is a vector corresponding to one segment (video data v consisting of 5 frame images). Transformer neural network f t v When each vector is input to the Transformer neural network f t v is the vector z (g,v) and vector sequence Z v Outputs video feature vector Z1 that includes [the specified element].

[0042] Furthermore, as shown in Figure 1, the second model f a This is a ConvNext neural network f for extracting spatial features. s a And, a Transformer neural network f for extracting temporal features t a It is composed of the following: ConvNext neural network f s a When audio data a is input to the ConvNext neural network f s a is a vector sequence g a,s It outputs the following. In the example shown in Figure 1, the vector sequence g a,s For example, it is composed of five vectors (each square in Figure 1 represents one vector). Vector sequence g a,sOne of the vectors contained in is a feature vector extracted from the audio data of one frame (hereinafter, also simply referred to as frame audio). In the example shown in FIG. 1, the audio data v is composed of five frame audios, and for each of these five frame audios, one feature vector is extracted from the frame audio. Note that the frame audio is the data within a predetermined time interval centered on a certain time for the audio data. By extracting the data within the predetermined time interval while shifting the central time for the audio data, the frame audio is obtained.

[0043] And, as shown in FIG. 1, for the Transformer neural network f t a For the ConvNext neural network f s a The vector sequence g output from a,s And the vector g a Are input. The vector g a Is, for example, a vector whose components are randomly set. Each of the feature vectors included in the vector sequence g a,s Is the feature vector of each audio data, while the vector g a Is a vector corresponding to one segment (the audio data a composed of five audio data). When each vector is input to the Transformer neural network f t a The Transformer neural network f t a Outputs the audio feature quantity Z2 including the vector z (g,a) And the vector sequence Z a

[0044] In this embodiment, the feature vector z t v Related to the segment output from the Transformer neural network f (g,v) And the feature vector sequence Z vTo separate them, a vector g corresponding to the learnable token is used. v Transformer neural network f t v This is included in the input. This results in the feature vector z associated with the segment. (g,v) and the feature vector sequence Z associated with the frame image v This is expressed by the following equation (1). Note that the feature vector sequence Z v This is an example of the first feature of this disclosure.

[0045]

number

[0046] θ in equation (1) above t v This is a Transformer neural network f t v This represents the model parameters. Also, θ s v This is a ConvNext neural network f s v This represents the model parameters. Also, `cat` represents the concatenation of vector sequences.

[0047] Similarly, for audio data, the Transformer neural network f t v Feature vector z related to the segment output from (g,a) and the feature vector sequence Z related to frame sounds a To separate them, a vector g corresponding to the learnable token is used. a Transformer neural network f t a This is included in the input. This results in the feature vector z associated with the segment. (g,a) and the feature vector sequence Z related to frame sounds a This is expressed by the following equation (2). Note that the feature vector sequence Z a This is an example of the second feature of this disclosure.

[0048]

number

[0049] θ in equation (2) above t a This is a Transformer neural network f t a This represents the model parameters. Also, θ s a This is a ConvNext neural network f s a This represents the model parameters.

[0050] Note: Model 2 f a Audio data x input to t a This is converted into a spectrogram image. Also, the feature vector sequence Z v and the feature vector sequence Z a Since the pair is a combination of the features of the frame image and the features of the frame sound, it can also be called a cross-modal feature vector sequence. v This refers to features extracted from data in a different format (audio data) than the video data, which is an example of the first time series data, and related features. Also, the feature vector sequence Z a These are features extracted from data in a different format (video data) than audio data, which is an example of second-generation time-series data, and related features.

[0051] [2.3 Local Contrastive Learning] In this embodiment, Local Contrastive Learning (hereinafter also simply referred to as local contrast learning) is introduced. Local contrast learning in this embodiment is introduced to extract features related to frames. This makes it possible to extract features of scenes such as when a player hits the ball in a table tennis match. For this reason, in this embodiment, the local contrast loss function L shown in equation (3) below is used. l Set it.

[0052]

number

[0053] In equation (3) above, only when t=t', the pair of video data and audio data associated with the frame (x t v ,x t’ a ) is a positive example. Also, in the case of t≠t', the pair of video data and audio data associated with the frame (x t v ,x t’ a ) is a negative example. Also, θ = {θ s v ,θ t v ,θ s a ,θ t a}. BCEwL is a binary cross-entropy function with a sigmoid function incorporated, and it can be realized using known techniques. Also, z t v The feature vector sequence Z v It is the t-th element of z. t a The feature vector sequence Z a This is the t-th element. δ represents a known delta function.

[0054] The frame rate of the video data and the sampling rate of the audio data play important roles in equation (3) above. In this embodiment, for example, we assume that the frame rate is 30 FPS and the sampling rate is 44100 Hz.

[0055] [2.4 Global Contrastive Learning] In this embodiment, Global Contrastive Learning (hereinafter also simply referred to as global contrast learning) is introduced. Global contrast learning in this embodiment is introduced to extract features related to the segment. In this embodiment, the global contrast loss function L is as shown in equation (4) below. g Set it.

[0056]

number

[0057] In equation (4) above, the feature vector z associated with the segment i,j (g,v) and feature vector z i,j (g,a) To extract positive examples, a pair of video and audio data segments within a segment is considered a positive example only if the segments in the video data and audio data match. If the segments in the video data and audio data do not match, the pair of video and audio data segments within that segment is considered a negative example.

[0058] Therefore, as shown in Figure 1, the feature vector z is a feature quantity extracted from the video data contained within a predetermined segment. t v and the feature vector z, which is a feature extracted from the frame sound at time t. t a The greater the similarity, the better the local contrast loss function L in equation (3) above. l It becomes smaller. Therefore, the first learning model f v and the second learning model f a This refers to the feature vector z, which is a feature extracted from the frame image at time t. t v and the feature vector z, which is a feature extracted from the frame sound at time t. t a The parameters are learned in such a way that the similarity to the given model increases.

[0059] Furthermore, as shown in Figure 1, the feature vector z is a feature extracted from the frame image belonging to the j-th segment within the i-th video data. i,j (g,v) And the feature vector z is a feature quantity extracted from the frame sound belonging to the j-th segment within the i-th audio data. i,j (g,a) The greater the similarity, the greater the global comparative loss function L in equation (4) above. g It becomes smaller. Therefore, the first learning model f v and the second learning model f a This refers to the feature vector z i,j (g,v) and feature vector z i,j (g,a) The parameters are learned in such a way that the similarity to the given model increases.

[0060] [2.5 Soft Label] In local contrast learning, the constraint t=t' exists in equation (3) above. However, such a constraint is strict for training the learning model, and it may not be possible to train the learning model properly. Therefore, in this embodiment, the Kronecker delta function δ with t=t' above is used. t,t’ δ' shown in the following equation t,t’ This relaxes the constraint, allowing non-zero values ​​set around t=t' to guide the learning model to the correct solution and reducing the constraint that does not accept deviations from t=t'.

[0061]

number

[0062] [2.6 Uncertainty Loss] In this embodiment, the local contrast loss function L is represented by equation (3) above. l and the global relative loss function L expressed by equation (4) above g By combining these, the loss function L shown in equation (5) below is set. Note that in equation (5) below, σ pThese are learnable parameters that are adjusted when training a learning model.

[0063]

number

[0064] [2.7 Downstream Classification] In this embodiment, the features extracted using the neural network FNN described above (z (g,v) ,Z v ,z (g,a) ,Z a The numbers ) are combined into a single vector, and this combined vector (hereinafter also simply referred to as the combined vector) is input to a neural network, which is a classification model. Figure 2 is a diagram illustrating the neural network for classification. As shown in Figure 2, in this embodiment, a first classification neural network MLP1 and a second classification neural network MLP2 are used. The first classification neural network MLP1 and the second classification neural network MLP2 are implemented, for example, by a known neural network called MLP (Multi-layer Perceptron). As shown in Figure 2, the combined vector cat(z (g,v) ,Z v ,z (g,a) ,Z a When the input ) is given to the first-classification neural network MLP1, the first-classification neural network MLP1 receives a label y that indicates whether or not the player hit the ball at time t. t s It outputs the combined vector cat(z (g,v) ,Z v ,z (g,a) ,Z a When the following is input to the Class II neural network MLP2, the Class II neural network MLP2 will input the label y representing the player who hit the ball at time t. t p and label y indicating whether the player was aggressive or not. t oThe label y represents both of the above. t po Outputs.

[0065] Furthermore, for each frame of the video data and each frame of the audio data, a label y is assigned to indicate whether or not the player hit the ball at time t. t s Since the correct values ​​are pre-set by the user, the first classification neural network MLP1 can be trained using known supervised learning. Similarly, for each frame of video data and each frame of audio data, a label y representing the player who hit the ball at time t is assigned. t p The correct value and the label y indicating whether the player was aggressive or not. t o Since the correct values ​​are pre-set by the user, the MLP2 second-class neural network can also be trained using known supervised learning methods.

[0066] Note that the labels estimated by the first classification neural network MLP1 and the second classification neural network MLP2 are (y t s ^,y t po ^) is expressed as. Also, (y t p ,y t o ) = (0,0), (0,1), (1,0), (1,0). In this embodiment, the loss function L when training the first classification neural network MLP1 and the second classification neural network MLP2 is pos The following loss function L pos This is defined.

[0067]

number

[0068] Note that θ in equation (6) above cThese are the parameters of the first-classification neural network MLP1 and the second-classification neural network MLP2. Furthermore, CE in equation (6) above is a known cross-entropy function. Also, the label y mentioned above... t po is (y t p ,y t o It is represented as a one-hot vector of ).

[0069] The following provides a more detailed explanation.

[0070] [Trained Model Generator] Figure 3 is a block diagram showing the hardware configuration of the trained model generation device 10 according to this embodiment. As shown in Figure 3, the trained model generation device 10 includes a CPU (Central Processing Unit) 42, memory 44, storage device 46, input / output I / F (Interface) 48, storage medium reader 50, and communication I / F 52. Each component is connected to the others via a bus 54 so as to be able to communicate with each other.

[0071] The memory device 46 stores a trained model generation program for executing the processes described later. The CPU 42 is a central processing unit that executes various programs and controls each component. Specifically, the CPU 42 reads a program from the memory device 46 and executes the program using memory 44 as its workspace. The CPU 42 controls each component and performs various calculations according to the program stored in the memory device 46.

[0072] Memory 44 consists of RAM (Random Access Memory) and temporarily stores programs and data as a working area. Storage device 46 consists of ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), etc., and stores various programs including the operating system and various data.

[0073] The I / F48 is an interface for inputting and outputting data. Input devices such as keyboards and mice, and output devices such as displays and printers, may also be connected. A touch panel display may be used as an output device, thereby functioning as an input device.

[0074] The storage medium reader 50 reads data stored on various storage media such as CD (Compact Disc)-ROM, DVD (Digital Versatile Disc)-ROM, Blu-ray disc, and USB (Universal Serial Bus) memory, and writes data to the storage media.

[0075] Communication I / F52 is an interface for communicating with other devices, and standards such as Ethernet®, FDDI, and Wi-Fi® are used.

[0076] Next, the functional configuration of the trained model generation device 10 will be described. As shown in Figure 4, the trained model generation device 10 functionally includes a training acquisition unit 12 and a training unit 16. In addition, a training data storage unit 14 and a trained model storage unit 18 are provided in a predetermined storage area of ​​the trained model generation device 10. Each functional configuration is realized by the CPU 42 reading each program stored in the storage device 46, expanding it into memory 44, and executing it.

[0077] The learning data storage unit 14 stores the learning data. In this embodiment, the learning data is a combination of the first learning time series data, which is learning time series data measured by the first sensor, and the second learning time series data, which is learning time series data measured by the second sensor in the same time interval as the first learning time series data.

[0078] In this embodiment, we will describe the case where the first sensor is a camera and the second sensor is a microphone. Therefore, the first time-series data for learning in this embodiment is video data, and the second time-series data for learning is audio data. The video data and audio data are data measured within the same time interval.

[0079] The trained model memory unit 18 stores the trained model generated by the process described later. Specifically, the trained model memory unit 18 stores the FNN neural network for feature extraction as described above, the first classification neural network MLP1, and the second classification neural network MLP2.

[0080] The learning data acquisition unit 12 acquires learning data. Specifically, the learning data acquisition unit 12 reads the learning data stored in the learning data storage unit 14.

[0081] The learning unit 16 generates a trained feature extraction neural network FNN by self-supervising training the FNN based on the training data acquired by the training data acquisition unit 12. The learning unit 16 also generates a trained first-classification neural network MLP1 and a trained second-classification neural network MLP2 by supervising training the first-classification neural network MLP1 and the second-classification neural network MLP2 based on the training data acquired by the training data acquisition unit 12. The machine learning algorithm used to generate the above trained models may be any type. The trained feature extraction neural network FNN is an example of a trained feature extraction model.

[0082] Furthermore, when training the FNN neural network for feature extraction, self-supervised learning is performed based on training data consisting of pairs of video and audio data. On the other hand, when training the first classification neural network MLP1 and the second classification neural network MLP2, a label y representing whether or not the player hit the ball at time t is used. t s Training data with the correct answer value attached, and labels y representing the player who hit the ball at time t. t p Training data with the correct values ​​attached, and labels y indicating whether the player was aggressive or not. t o Supervised learning is performed based on training data that has the correct answers attached.

[0083] The learning unit 16 extracts a feature vector sequence Z from the training frame images within the training video data. v The feature vector sequence Z is extracted from the frame sound in the training audio data measured at the same time as the training frame image. a A neural network (FNN) for feature extraction is trained to achieve high similarity between the two. This allows the feature vector sequence Z to be generated when video and audio data are input. v and feature vector sequence Z a A trained feature extraction neural network FNN is generated that outputs the following. Specifically, the learning unit 16 uses the local contrast loss function L of equation (3) above. l The neural network FNN for feature extraction is trained to minimize this value.

[0084] Furthermore, the learning unit 16 extracts a feature vector z from the data within a segment that is within a predetermined time interval in the video data. (g,v) And, a feature vector z extracted from data within the same segment in the audio data. (g,a) The neural network FNN for feature extraction is trained to increase the similarity between the two. Specifically, the learning unit 16 trains the global comparison loss function L in equation (4) above. gThe neural network FNN for feature extraction is trained to minimize this value.

[0085] In practice, the learning unit 16 trains the FNN neural network for feature extraction so that the loss function L in equation (5) above is minimized.

[0086] Furthermore, the learning unit 16 uses the loss function L in equation (6) above. pos The first classification neural network MLP1 and the second classification neural network MLP2 are trained to minimize the following.

[0087] The learning unit 16 then stores the various trained models it has generated into the trained model storage unit 18.

[0088] [Information Processing Device] Figure 5 is a block diagram showing the hardware configuration of the information processing device 24 according to this embodiment. As shown in Figure 5, the information processing device 24 includes a CPU (Central Processing Unit) 62, memory 64, storage device 66, input / output I / F (Interface) 68, storage medium reader 70, and communication I / F 72. Each component is connected to the others via a bus 74 so as to be able to communicate with each other.

[0089] The storage device 66 stores information processing programs for executing the processes described later. The CPU 62 is a central processing unit that executes various programs and controls each component. Specifically, the CPU 62 reads programs from the storage device 66 and executes them using memory 64 as a workspace. The CPU 62 controls each component and performs various calculations according to the programs stored in the storage device 66.

[0090] Memory 64 consists of RAM (Random Access Memory) and temporarily stores programs and data as a working area. Storage device 66 consists of ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), etc., and stores various programs including the operating system and various data.

[0091] The I / F68 is an interface for inputting and outputting data. It may also be connected to various input devices, such as a keyboard or mouse, and output devices, such as a display or printer, for outputting various types of information. A touch panel display may be used as an output device, allowing it to function as an input device as well.

[0092] The storage medium reader 70 reads data stored on various storage media such as CD (Compact Disc)-ROM, DVD (Digital Versatile Disc)-ROM, Blu-ray disc, and USB (Universal Serial Bus) memory, and writes data to the storage media.

[0093] Communication I / F72 is an interface for communicating with other devices, and standards such as Ethernet®, FDDI, and Wi-Fi® are used.

[0094] Next, the functional configuration of the information processing device 24 will be described. As shown in Figure 6, the information processing device 24 functionally includes an acquisition unit 28, an extraction unit 30, a processing unit 31, and an output unit 32. In addition, a data storage unit 26 and a trained model storage unit 27 are provided in a predetermined storage area of ​​the information processing device 24. Each functional configuration is realized by the CPU 62 reading each program stored in the storage device 66, expanding it into memory 64, and executing it.

[0095] The data storage unit 26 stores the video data and audio data that are the subject of the estimation task. In this embodiment, since the estimation task described above is performed, video data and audio data of a table tennis match are stored.

[0096] The trained model storage unit 27 stores various trained models generated by the trained model generation device 10. Specifically, the trained model storage unit 27 stores a trained feature extraction neural network FNN, a trained first classification neural network MLP1, and a trained second classification neural network MLP2. As described above, the trained feature extraction neural network FNN is a feature vector sequence Z extracted from frame images in video data. v The feature vector sequence Z is extracted from the frame sound in the audio data measured at the same time as the frame image. a This is an example of a pre-trained feature extraction model that has been trained to achieve high similarity between the given data and the given data.

[0097] The acquisition unit 28 acquires video data and audio data stored in the data storage unit 26. The video data and audio data are data measured within the same time interval.

[0098] The extraction unit 30 inputs the video data and audio data acquired by the acquisition unit 28 into the trained feature extraction neural network FNN stored in the trained model storage unit 27, thereby extracting a feature vector sequence Z, which is an example of the first feature of the video data. v And, as an example of the second feature of audio data, feature vector sequence Z a Extract the following: Feature vector sequence Z v and feature vector sequence Z a This is a feature vector associated with the frame.

[0099] Furthermore, the extraction unit 30 inputs the video data and audio data acquired by the acquisition unit 28 into a trained feature extraction neural network FNN, thereby generating the feature vector z of the video data. (g,v) and the feature vector z of the audio data (g,a) Also extract the feature vector sequence z. (g,v) and feature vector sequence z (g,a) This is a feature vector associated with the segment.

[0100] The processing unit 31 controls the feature vector sequence Z v and feature vector sequence Z a Based on this, the state of the athlete, who is an example of a person appearing in the video data, is estimated. Specifically, the processing unit 31 uses the z obtained by the extraction unit 30 to estimate the state of the athlete. (g,v) ,Z v ,z (g,a) ,Z a The combined vector (z (g,v) ,Z v ,z (g,a) ,Z a The trained first-classification neural network MLP1 and the trained second-classification neural network MLP2 stored in the trained model memory unit 27 are input to the trained first-classification neural network MLP1 and the trained second-classification neural network MLP2. The trained first-classification neural network MLP1 inputs the label y t s ^ is output. Also, from the trained second-class neural network MLP2, label y t po ^ is output. These labels indicate whether a player hit the ball at time t, which player hit the ball at time t, and whether the player who hit the ball at time t was offensive.

[0101] The output unit 32 outputs the estimation results obtained by the processing unit 31. The user checks the results output from the output unit 32.

[0102] Next, the operation of the trained model generation device according to this embodiment will be described.

[0103] First, a table tennis expert watches multiple videos of table tennis matches in progress and labels the frame images and frame sounds at each point in time. This generates labeled training data. The labeled training data is used for supervised learning. The unlabeled training data is used for self-supervised learning.

[0104] The user inputs the generated training data into the trained model generation device 10. Once the training data is input into the trained model generation device 10, it is stored in the training data storage unit 14.

[0105] Then, when the trained model generation device 10 receives a predetermined instruction signal, the CPU 42 of the trained model generation device 10 reads the trained model generation program from the storage device 46, loads it into memory 44, and executes it. As a result, the CPU 42 functions as each of the functional configurations of the trained model generation device 10, and the trained model generation process shown in Figure 7 is executed.

[0106] In step S100, the learning data acquisition unit 12 acquires labeled learning data and unlabeled learning data stored in the learning data storage unit 14.

[0107] In step S102, the learning unit 16 generates a trained model by machine learning a known training model based on the training data acquired in step S100. Specifically, the learning unit 16 generates a trained feature extraction neural network FNN by self-supervising training the FNN based on the unlabeled training data acquired in step S100 so that the loss function L in equation (5) above is minimized. The learning unit 16 also generates a trained feature extraction neural network FNN based on the labeled training data acquired in step S100 so that the loss function L in equation (6) above is minimized. posBy supervising the first-classification neural network MLP1 and the second-classification neural network MLP2 to minimize the coefficient of error, trained first-classification neural network MLP1 and trained second-classification neural network MLP2 are generated.

[0108] In step S104, the learning unit 16 stores the trained feature extraction neural network FNN, the trained first classification neural network MLP1, and the trained second classification neural network MLP2, which were generated in step S102, into the trained model storage unit 18.

[0109] Next, the operation of the information processing device 24 according to this embodiment will be described.

[0110] When various trained models generated by the trained model generation device 10 are input to the information processing device 24, these trained models are stored in the trained model storage unit 27 of the information processing device 24.

[0111] Next, video and audio data of the table tennis match to be estimated are input to the information processing device 24. The information processing device 24 stores the input video and audio data in the data storage unit 26.

[0112] Then, when the information processing device 24 receives a predetermined instruction signal, the CPU 62 of the information processing device 24 reads the information processing program from the storage device 66, loads it into memory 64, and executes it. As a result, the CPU 62 functions as each of the functional configurations of the information processing device 24, and the information processing shown in Figure 8 is executed.

[0113] In step S200, the acquisition unit 28 acquires the video data and audio data stored in the data storage unit 26.

[0114] In step S202, the extraction unit 30 reads out the trained feature extraction neural network FNN stored in the trained model memory unit 27.

[0115] In step S204, the extraction unit 30 inputs the video data and audio data acquired in step S200 into the trained feature extraction neural network FNN read out in step S202, thereby generating the feature vector z (g,v) ,Z v ,z (g,a) ,Z a Obtain it.

[0116] In step S206, the processing unit 31 reads out the trained first classification neural network MLP1 and the trained second classification neural network MLP2 stored in the trained model memory unit 27.

[0117] In step S208, the processing unit 31 applies the feature vector z obtained in step S204 to the trained first classification neural network MLP1 and the trained second classification neural network MLP2 read out in step S206. (g,v) ,Z v ,z (g,a) ,Z a By inputting the combined vector, the estimated label y t s ^ and label y t po Get the ^ symbol.

[0118] In step S210, the processing unit 31 outputs the prediction result obtained in step S208.

[0119] The user checks the estimation results output from the output unit 32.

[0120] As described above, the information processing device according to this embodiment acquires first time-series data, which is time-series data measured by the first sensor, and second time-series data, which is time-series data measured by the second sensor in the same time interval as the first time-series data. The information processing device inputs the first time-series data and the second time-series data into a pre-trained feature extraction model that has been trained in advance to maximize the similarity between a first feature extracted from the first data in the first time-series data and a second feature extracted from the second data in the second time-series data measured at the same time as the first data. The information processing device then extracts the first feature from the first time-series data and the second feature from the second time-series data. The information processing device then outputs a combination of the first feature and the second feature. This makes it possible to extract useful time-unit features from different types of time-series data measured in the same time interval.

[0121] Furthermore, the trained model generation device according to this embodiment acquires first training time series data, which is training time series data measured by the first sensor, and second training time series data, which is training time series data measured by the second sensor in the same time interval as the first training time series data. The trained model generation device trains a training model so that the similarity between the first feature extracted from the first training data in the first training time series data and the second feature extracted from the second training data in the second training time series data, which is measured at the same time as the first training data, is high. As a result, when the first time series data and the second time series data are input, the device generates a trained feature extraction model that outputs the first feature of the first time series data and the second feature of the second time series data. This makes it possible to obtain a training model for extracting useful time-unit features from different types of time series data measured in the same time interval. This makes it possible to recognize timing-sensitive human actions such as clapping hands and making noises.

[0122] Furthermore, the pre-trained model for feature extraction is also a model that has been pre-trained to maximize the similarity between features extracted from data within a predetermined time interval in the first time series data and features extracted from data within a predetermined time interval in the second time series data. Specifically, an example of a pre-trained model for feature extraction, the neural network FNN for feature extraction, has the local contrast loss function L in equation (3) above. l In addition, the global relative loss function L in equation (4) above g The system is trained to minimize the local contrast loss function L. l Minimizing this corresponds to increasing the similarity between the feature quantities of the frame image and the feature quantities of the frame sound at each time unit. However, it is difficult to find the parameters of a feature extraction neural network that can extract such features. In contrast, in this embodiment, the global contrast loss function L at the segment level is used. g By introducing this approach, it becomes possible to first increase the similarity between features at the segment level, and then explore parameters that further increase the similarity between features at the frame level (or time level). This allows us to obtain a training model for extracting useful features at the frame level (or time level).

[0123] This disclosure is not limited to the embodiments and examples described above, and various modifications and applications are possible without departing from the spirit of the invention.

[0124] For example, the above embodiment was described using a table tennis match as an example, but it is not limited to this. For example, this embodiment may be applied to recognizing human behavior. For example, this embodiment may be applied to monitoring suspicious persons, human behavior in the workplace, customer purchasing behavior, behavior in sports, consumer behavior in entertainment, behavior during work in a factory, etc. According to this embodiment, it is possible to extract useful features from data representing the timing-sensitive behaviors described above.

[0125] Furthermore, although the above embodiment was described using the example of a case where the first time-series data is video data and the second time-series data is audio data, it is not limited to this. For example, this embodiment can be applied to any pair of synchronized time-series data. For example, the first time-series data may be video data and the second time-series data may be vibration data or acceleration data, etc.

[0126] Furthermore, although the above embodiments described examples where the machine learning model is a Transformer, ConvNext, and MLP, the invention is not limited to these. Any machine learning model may be used.

[0127] Furthermore, in the above embodiment, the neural network FNN for feature extraction, which is an example of a pre-trained model for feature extraction, is the first pre-trained model f v and the second pre-trained model f a The example given was one in which the model includes the above, but it is not limited to this. For example, a pre-trained model for feature extraction may consist of a single machine learning model.

[0128] Furthermore, in the above embodiment, the feature vector z extracted from the FNN, a neural network for feature extraction which is an example of a pre-trained model for feature extraction. (g,v) ,Z v ,z (g,a) ,Z a The example given was using the estimation task in a table tennis match, but it is not limited to this. The processing unit of the information processing device is a feature vector sequence Z, which is an example of the first feature. v And an example of a second feature is the feature vector sequence Z. a Based on this, at least one of the following processes may be performed on the first and second time series data: prediction, estimation, search, and adjustment. For example, feature vector sequence Z v ,Z aThese are features related to frame images and frame sounds, and these features can be useful when performing other tasks. For example, as mentioned above, if it corresponds to the sound of a player hitting the ball in a table tennis match, it is possible to search for frame images of other scenes in a table tennis match where a player is hitting the ball, based on the features extracted from that sound. Alternatively, the feature vector sequence Z v ,Z a These can also serve as useful features for recognizing, predicting, or estimating human behavior as described above. Alternatively, the feature vector sequence Z v ,Z a This can also be used for processing that adjusts the discrepancy between video data and audio data. Therefore, the feature vector z (g,v) ,Z v ,z (g,a) ,Z a Instead of using it to perform some task, simply use the feature vector z (g,v) ,Z v ,z (g,a) ,Z a The output unit 32 may output at least one of the following: v and feature vector sequence Z a You could also output the combination of the two.

[0129] Furthermore, in the above embodiment, a neural network FNN for feature extraction, which is an example of a pre-trained model for feature extraction, was trained so that the similarity between the features extracted from the frame images in the video data and the features extracted from the frame sounds in the audio data measured at the same time as the frame images is high. Furthermore, in the above embodiment, a neural network FNN for feature extraction, which is an example of a pre-trained model for feature extraction, was trained so that the similarity between the features extracted from the frame images in the segments of the video data and the features extracted from the frame sounds in the segments of the audio data measured in the same time interval as the segments is high. Specifically, a neural network FNN for feature extraction, which is an example of a pre-trained model for feature extraction, was trained so that the similarity between features extracted from data where the timing of the video and audio are close is higher than the similarity between features extracted from data where the timing is far apart. In this case, the similarity is the dot product (z) of the feature vectors, as shown in equations (3) and (4) above. t v ·z t’ a or z i,j (g,v) ·z i,j’ (g,a) This corresponds to ), but other metrics may be used to indicate similarity between feature vectors. For example, the reciprocal of the distance between feature vectors may be used as the similarity.

[0130] Furthermore, in the above embodiment, the feature vector z (g,v) ,Z v ,z (g,a) ,Z a The example given was using to perform some task, but it is not limited to this. (g,v) and z (g,a) It is not necessary. For example, feature vector sequence Z is an example of a first feature. v And Z is an example of a second feature. a You may also use this to execute the task.

[0131] Furthermore, in the above embodiment, the example described was the case in which the neural network FNN for feature extraction is trained so that equation (5), which is a combination of equation (3) and equation (4), becomes small, but it is not limited to this. For example, the neural network FNN for feature extraction may be trained so that only equation (3) becomes small.

[0132] Furthermore, in the above embodiment, each process that the CPU reads and executes software (programs) may be executed by various processors other than the CPU. Examples of such processors include PLDs (Programmable Logic Devices) such as FPGAs (Field-Programmable Gate Arrays) whose circuit configuration can be changed after manufacturing, ASICs (Application Specific Integrated Circuits), and dedicated electrical circuits that have a circuit configuration specifically designed to execute a particular process, such as GPUs (Graphics Processing Units). Each process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (for example, multiple FPGAs, and a combination of a CPU or GPU and an FPGA). More specifically, the hardware structure of these various processors is an electrical circuit that combines circuit elements such as semiconductor elements.

[0133] Furthermore, although the above embodiment describes a configuration in which each program is pre-stored (installed) on a storage device, the invention is not limited to this configuration. Programs may be provided in a form stored on a storage medium such as a CD-ROM, DVD-ROM, Blu-ray disc, or USB memory. Programs may also be provided in a form that can be downloaded from an external device via a network. [Examples]

[0134] In this embodiment, the label estimation (y ts ^,y t p ^,y t o Performed ^).

[0135] (1. Assignment of Correct Labels) First, correct labels were assigned to the players at each time of the video data and audio data. This correct label is a label indicating whether the player's action is offensive or not, whether the player hit the ball or not, and which player hit the ball. The correct label in this embodiment was assigned by a table tennis expert (a player belonging to a university table tennis club).

[0136] (2. Evaluation Method) In this embodiment, the learned model as described above was evaluated by the known Area Under the Receiver Operating Characteristic (AUC). Note that the label y t s ,y t p ,y t o Separate from that, the label y t os was set. The label y t os is represented by the following formula.

[0137] [Number]

[0138] (3. Evaluation Results) The following table shows the experimental results of this embodiment. The numerical values described in the following table are AUC.

[0139] [Table 1]

[0140] Note that each item in the above table is as follows.

[0141] Baseline 1: This shows the results when the estimation task was performed using a first-classification neural network and a second-classification neural network without retraining the FNN neural network used for feature extraction. Baseline 2: Global relative loss function L of equation (4) above g This shows the results when the FNN neural network for feature extraction was retrained using [a specific method / tool], and then the estimation task was performed using the first-classification neural network and the second-classification neural network. Ours (l): Local contrast loss function L of equation (3) above l This is the result when the FNN neural network for feature extraction was retrained using [a specific method / tool]. Ours (ls): Local contrast loss function L in equation (3) above l This is the result when using [2.5 Soft Label] and retraining the FNN neural network for feature extraction. Ours (gl): Local contrast loss function L of equation (3) above l and the global relative loss function L of equation (4) above g Functions that include (L l +L g This is the result when the FNN neural network for feature extraction was retrained using ) / 2. Ours (gls): Local contrast loss function L of equation (3) above l and the global relative loss function L of equation (4) above g Functions that include (L l +L g This is the result when the neural network FNN for feature extraction is retrained using ) / 2 and with the introduction of [2.5 Soft Label]. Ours (glu): This is the result when the neural network FNN for feature extraction is retrained using the loss function L from equation (5) above, and without introducing [2.5 Soft Label]. Ours (glsu): This is the result of retraining the FNN neural network for feature extraction using all the elements in the above embodiment.

[0142] As shown in the table above, it was confirmed that introducing the method of this embodiment has the potential to improve estimation accuracy.

[0143] (Note) The following is an addendum regarding the nature of this disclosure.

[0144] (Note 1) An acquisition unit that acquires first time-series data, which is time-series data measured by a first sensor, and second time-series data, which is time-series data measured by a second sensor in the same time interval as the first time-series data. An extraction unit extracts the first feature from the first time series data and the second feature from the second time series data by inputting the first time series data and the second time series data to a pre-trained feature extraction model that has been trained in advance to maximize the similarity between the first feature extracted from the first data in the first time series data and the second feature extracted from the second data in the second time series data measured at the same time as the first data. An output unit that outputs a combination of the first feature and the second feature, Information processing device including (Note 2) The first time-series data is video data, The second time-series data is audio data measured during the same time interval as the video data. The information processing device described in Appendix 1. (Note 3) The aforementioned pre-trained model for feature extraction includes a first pre-trained model and a second pre-trained model. The extraction unit inputs the first time series data into the first trained model to obtain the first feature output from the first trained model, and inputs the second time series data into the second trained model to obtain the second feature output from the second trained model. The information processing device described in Appendix 1 or Appendix 2. (Note 4) The aforementioned trained model for feature extraction further, This is a model that has been pre-trained to increase the similarity between features extracted from data within a predetermined time interval in the first time series data and features extracted from data within the same predetermined time interval in the second time series data. An information processing device as described in any one of the items in Appendix 1 to Appendix 3. (Note 5) The system further includes a processing unit that performs at least one of the following processes related to the first time series data and the second time series data, based on the first and second features: The output unit outputs the processing result obtained by the processing unit. An information processing device as described in any one of the items in Appendix 1 to Appendix 3. (Note 6) The system further includes a processing unit that estimates the state of a person appearing in the video data based on the first and second features, The output unit outputs the estimation result obtained by the processing unit. The information processing device described in Appendix 2. (Note 7) A learning acquisition unit that acquires learning time series data, specifically a first learning time series data measured by a first sensor, and learning time series data, specifically a second learning time series data measured by a second sensor during the same time interval as the first learning time series data. A learning unit generates a trained feature extraction model that outputs the first feature of the first time series data and the second feature of the second time series data when the first time series data and the second time series data are input, by training the learning model so that the similarity between the first feature extracted from the first time series data in the first time series data for training and the second feature extracted from the second time series data in the second time series data for training, which is measured at the same time as the first time series data, is high. A pre-trained model generator that includes this feature. (Note 8) First time series data, which is time series data measured by the first sensor, and second time series data, which is time series data measured by the second sensor in the same time interval as the first time series data, are acquired. By inputting the first time series data and the second time series data into a pre-trained feature extraction model that has been trained to maximize the similarity between the first feature extracted from the first data in the first time series data and the second feature extracted from the second data in the second time series data measured at the same time as the first data, the first feature of the first time series data and the second feature of the second time series data are extracted. Output the combination of the aforementioned first feature and the aforementioned second feature. An information processing method in which a computer performs the processing. (Note 9) The system acquires a first learning time series data, which is learning time series data measured by the first sensor, and a second learning time series data, which is learning time series data measured by the second sensor during the same time interval as the first learning time series data. By training the training model so that the similarity between the first feature extracted from the first training data within the first training time series data and the second feature extracted from the second training data within the second training time series data measured at the same time as the first training data is high, a trained feature extraction model is generated that outputs the first feature of the first time series data and the second feature of the second time series data when the first time series data and the second time series data are input. A method for generating a trained model that is executed by a computer. (Note 10) First time series data, which is time series data measured by the first sensor, and second time series data, which is time series data measured by the second sensor in the same time interval as the first time series data, are acquired. By inputting the first time-series data and the second time-series data into a pre-trained learned model for feature extraction, such that the similarity between the first feature quantity extracted from the first data within the first time-series data and the second feature quantity extracted from the second data within the second time-series data measured at the same time as the first data is increased, the first feature quantity of the first time-series data and the second feature quantity of the second time-series data are extracted. Output the combination of the first feature quantity and the second feature quantity. An information processing program for causing a computer to execute the processing. (Appendix 11) Obtain learning first time-series data, which is time-series data for learning measured by a first sensor, and learning second time-series data, which is time-series data for learning measured by a second sensor in the same time interval as the learning first time-series data. By training a learning model such that the similarity between the first feature quantity extracted from the first learning data within the learning first time-series data and the second feature quantity extracted from the second learning data within the learning second time-series data measured at the same time as the first learning data is increased, a learned model for feature extraction that outputs the first feature quantity of the first time-series data and the second feature quantity of the second time-series data when the first time-series data and the second time-series data are input is generated. A learned model generation program for causing a computer to execute the processing.

Explanation of Signs

[0145] 10 Learned model generation device 12 Learning acquisition unit 14 Learning data storage unit 16 Learning unit 18 Learned model storage unit 24 Information processing device 26 Data storage unit 27 Learned model storage unit 28 Acquisition unit 30 Extraction unit 31 Processing unit 32 Output section

Claims

1. An acquisition unit that acquires first time-series data, which is time-series data measured by a first sensor, and second time-series data, which is time-series data measured by a second sensor in the same time interval as the first time-series data. An extraction unit extracts the first feature from the first time series data and the second feature from the second time series data by inputting the first time series data and the second time series data to a pre-trained feature extraction model that has been trained in advance to maximize the similarity between the first feature extracted from the first data in the first time series data and the second feature extracted from the second data in the second time series data measured at the same time as the first data. An output unit that outputs a combination of the first feature and the second feature, Information processing device including

2. The first time-series data is video data, The second time-series data is audio data measured during the same time interval as the video data. The information processing apparatus according to claim 1.

3. The aforementioned pre-trained model for feature extraction includes a first pre-trained model and a second pre-trained model. The extraction unit inputs the first time series data into the first trained model to obtain the first feature output from the first trained model, and inputs the second time series data into the second trained model to obtain the second feature output from the second trained model. The information processing apparatus according to claim 1 or claim 2.

4. The aforementioned trained model for feature extraction further, This is a model that has been pre-trained to increase the similarity between the features extracted from data within a predetermined time interval in the first time series data and the features extracted from data within the predetermined time interval in the second time series data. The information processing apparatus according to claim 1 or claim 2.

5. The system further includes a processing unit that performs at least one of the following processes related to the first time series data and the second time series data, based on the first and second features: The output unit outputs the processing result obtained by the processing unit. The information processing apparatus according to claim 1 or claim 2.

6. The system further includes a processing unit that estimates the state of a person appearing in the video data based on the first and second feature quantities, The output unit outputs the estimation result obtained by the processing unit. The information processing apparatus according to claim 2.

7. A learning acquisition unit that acquires learning time series data, which is learning time series data measured by a first sensor, and learning time series data, which is learning time series data measured by a second sensor in the same time interval as the learning first time series data, A learning unit generates a trained feature extraction model that outputs the first feature of the first time series data and the second feature of the second time series data when the first time series data and the second time series data are input, by training the learning model so that the similarity between the first feature extracted from the first training data in the first training time series data and the second feature extracted from the second training data in the second training time series data measured at the same time as the first training data is high. A pre-trained model generator that includes this feature.

8. The system acquires first time-series data, which is time-series data measured by the first sensor, and second time-series data, which is time-series data measured by the second sensor during the same time interval as the first time-series data. By inputting the first time series data and the second time series data into a pre-trained feature extraction model that has been trained to maximize the similarity between the first feature extracted from the first data in the first time series data and the second feature extracted from the second data in the second time series data measured at the same time as the first data, the first feature of the first time series data and the second feature of the second time series data are extracted. Output the combination of the first feature and the second feature. An information processing method in which a computer performs the processing.

9. The system acquires a first learning time series data, which is learning time series data measured by the first sensor, and a second learning time series data, which is learning time series data measured by the second sensor during the same time interval as the first learning time series data. By training the training model so that the similarity between the first feature extracted from the first training data within the first training time series data and the second feature extracted from the second training data within the second training time series data measured at the same time as the first training data is high, a trained feature extraction model is generated that outputs the first feature of the first time series data and the second feature of the second time series data when the first time series data and the second time series data are input. A method for generating a trained model that is executed by a computer.

10. The system acquires first time-series data, which is time-series data measured by the first sensor, and second time-series data, which is time-series data measured by the second sensor during the same time interval as the first time-series data. By inputting the first time series data and the second time series data into a pre-trained feature extraction model that has been trained to maximize the similarity between the first feature extracted from the first data in the first time series data and the second feature extracted from the second data in the second time series data measured at the same time as the first data, the first feature of the first time series data and the second feature of the second time series data are extracted. Output the combination of the first feature and the second feature. An information processing program that causes a computer to perform a task.

11. The system acquires a first learning time series data, which is learning time series data measured by the first sensor, and a second learning time series data, which is learning time series data measured by the second sensor during the same time interval as the first learning time series data. By training the training model so that the similarity between the first feature extracted from the first training data within the first training time series data and the second feature extracted from the second training data within the second training time series data measured at the same time as the first training data is high, a trained feature extraction model is generated that outputs the first feature of the first time series data and the second feature of the second time series data when the first time series data and the second time series data are input. A program that generates pre-trained models for a computer to perform a process.

Images