Program, information processing device, and information processing method

Abstract

To suppress calculation amounts from increasing, while improving accuracy of processing using a machine learning model.SOLUTION: A program makes a computer execute each step of acquisition, calculation, selection and learning inference. An acquisition step acquires one or more first time-series data of mutually different kinds, and one or more second time-series data which are different from the first time-series data in the kinds and have mutually different kinds. A calculation step calculates an attention degree which shows a degree focused in inference based on a first model which inputs the first time-series data, about each of a plurality of first frames included in the first time-series data. A selection step selects N-number of second frames from the second time-series data, by using the attention degree. A learning inference step executes learning or inference of a second model which executes the inference, by using information based on the first time-series data, and the information based on the selected N-number of second frames.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] Embodiments of the present invention relate to a program, an information processing apparatus, and an information processing method.

Background Art

[0002] Techniques have been proposed for inputting time-series data (sequence data) to a machine learning model to execute various types of inference. The machine learning model is, for example, a deep learning model. For example, a model has been proposed that inputs video data capturing a moving object such as a person, or skeleton sequence data obtained by analyzing the video data, to infer the motion of the moving object (such as the action of a person).

[0003] If inference is executed using multiple types of time-series data, it is expected that the accuracy of inference will be improved. On the other hand, since time-series data is data including data (frames) at a plurality of timestamps, in a configuration using a plurality of pieces of time-series data, the computational complexity of processes such as inference and learning by the machine learning model may increase.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Non-Patent Documents

[0005]

Non-Patent Document 1

Summary of Invention

Problems to be Solved by the Invention

[0006] The present invention aims to provide a program, an information processing device, and an information processing method that can improve the accuracy of processing using machine learning models while suppressing an increase in computational load. [Means for solving the problem]

[0007] The program of the embodiment is a program that causes a computer to perform the steps of acquisition, calculation, selection, and learning inference. The acquisition step acquires one or more first time series data that are of different types from each other, and one or more second time series data that are of different types from the first time series data and are also of different types from each other. The calculation step calculates an attention score for each of the multiple first frames contained in the first time series data, which represents the degree to which it is given attention in inference by a first model that takes the first time series data as input. The selection step selects N second frames from the second time series data using the attention scores. The learning inference step uses information based on the first time series data and information based on the selected N second frames to train or perform inference on a second model that performs inference. [Brief explanation of the drawing]

[0008] [Figure 1] Block diagram of the information processing device of the first embodiment. [Figure 2] A diagram showing an example of the acquired time-series data. [Figure 3] This diagram illustrates an example where there is only one frame that attracts the most attention. [Figure 4] A diagram illustrating an example of applying the selection method. [Figure 5] A diagram illustrating an example of applying the selection method. [Figure 6] Block diagram of the learning inference unit of the first embodiment. [Figure 7] Flowchart of information processing in the first embodiment. [Figure 8] This figure shows an example of the output of the inference result from the output control unit. [Figure 9] Block diagram of the information processing device of the second embodiment. [Figure 10]Block diagram of the learning inference unit according to the second embodiment. [Figure 11] Block diagram of the information processing apparatus according to the third embodiment. [Figure 12] Block diagram of the learning inference unit according to the third embodiment. [Figure 13] Hardware configuration diagram of the information processing apparatuses according to the first to third embodiments. [Mode for Carrying Out the Invention]

[0009] Hereinafter, preferred embodiments of the information processing apparatus according to the present invention will be described in detail with reference to the accompanying drawings.

[0010] In the following, an example of a configuration using a model that inputs a plurality of types of time-series data (sequence data) to estimate (infer) the behavior of a person will be mainly described. The inference to which the following embodiments are applicable is not limited to such human behavior estimation, and may be any inference using time-series data. For example, the configuration may be configured to estimate the motion of a moving object other than a person, such as an animal and a robot.

[0011] The time-series data is data including a plurality of pieces of data (hereinafter referred to as frames) that are consecutive in time series. Each of the plurality of frames included in the time-series data is, for example, the following types of data, but is not limited thereto. · Color image data (for example, RGB image data) · Skeleton data · Optical flow data · Depth image data · Segmented image data · Infrared image data · X-ray image data · Audio data

[0012] Note that the type of data (time-series data) may be referred to as a data modality, a data format, or the like in some cases. The following embodiments can also be interpreted as examples of performing inference using multi-modal time-series data.

[0013] As described above, by performing inference using a plurality of types of time-series data, it is possible to improve the accuracy of the inference. On the other hand, depending on the type of time-series data, there are cases where it is not necessary to use all frames included in the time-series data for the intended inference.

[0014] For example, consider a case of estimating an action of a certain person eating a certain food using time-series data of skeleton data (hereinafter referred to as skeleton sequence data) and time-series data of color image data (hereinafter referred to as moving image data). In such a case, the action of eating food is estimated using, for example, skeleton sequence data including a plurality of frames. On the other hand, the food being eaten may be estimable using only the frame in which the food is captured, among the frames included in the moving image data, for example.

[0015] Therefore, when performing inference using a plurality of types of time-series data, the accuracy of the inference can be improved even if the inference is performed using some frames, instead of all frames included in the time-series data, for some types of time-series data.

[0016] (First Embodiment) The information processing apparatus according to the first embodiment executes predetermined inference using a plurality of types of time-series data. At this time, in the present embodiment, using the attention level of each frame calculated from a certain type of time-series data (hereinafter referred to as time-series data SDA), N frames (N is an integer of 1 or more, hereinafter referred to as frame data FDB) are selected from another type of time-series data (hereinafter referred to as time-series data SDB), and the inference is executed using the selected frame data FDB. This makes it possible to reduce the amount of calculation compared to a configuration that performs inference using the time-series data SDB including all frames.

[0017] The "attention level" represents the degree to which a frame is given attention in inference using time-series data SDA. For example, the frame data FDB corresponding to the time frame with the highest attention level is selected. Therefore, it is possible to improve the accuracy of processing using machine learning models without using time-series data SDB that contains all frames.

[0018] Figure 1 is a block diagram showing an example of the configuration of an information processing device 100 according to the first embodiment. As shown in Figure 1, the information processing device 100 includes an acquisition unit 101, a calculation unit 102, a selection unit 103, a learning and inference unit 104, an output control unit 105, and a storage unit 121.

[0019] The acquisition unit 101 acquires various types of information used by the information processing device 100. For example, the acquisition unit 101 acquires one or more time-series data SDA (first time-series data) of different types from each other, and one or more time-series data SDB (second time-series data) of different types from the time-series data SDA and of different types from each other.

[0020] The following describes an example in the first embodiment in which one time-series data SDA and one time-series data SDB are acquired. An example using two time-series data SDBs will be described in the second embodiment. An example using two time-series data SDAs will be described in the third embodiment. By combining the second and third embodiments, a configuration using two or more time-series data SDAs and two or more time-series data SDBs can be realized.

[0021] As will be described later, the time series data SDA corresponds to time series data in which all frames are used for behavior estimation. On the other hand, the time series data SDB corresponds to time series data in which a selection of N frames are used for behavior estimation.

[0022] The time series data SDA and SDB may be data where the start and end times match. This eliminates the need to associate times between multiple time series data.

[0023] If the frame rates differ among multiple time-series data, for example, the acquisition unit 101 performs a process to interpolate frames for the time-series data with the lower frame rate, thereby matching the frame rates of the multiple time-series data.

[0024] The time series data with a larger data volume may be acquired as the time series data from which the frame is selected. In other words, the time series data SDB may have a larger data volume than the time series data SDA. This can further increase the amount of data reduction.

[0025] For example, the acquisition unit 101 may acquire skeletal sequence data as time-series data SDA and video data as time-series data SDB. Skeletal sequence data is, for example, time-series coordinate data representing the joint points of a person. Skeletal sequence data can be acquired by any method, but can be obtained by, for example, a skeletal detection method such as Openpose, or a method using sensors.

[0026] In the following, it is assumed that there is only one person to be estimated. If the time series data contains data for multiple people, the acquisition unit 101 may generate and acquire time series data that includes data extracted for each person using an object detector or the like. When performing inference other than estimating a person's behavior, it is not necessary to perform processing such as extracting data for each person using object detection or the like.

[0027] Figure 2 shows examples of acquired time-series data. Figure 2 shows examples of time-series data SDA, which includes skeletal data as frames, and time-series data SDB, which includes RGB image data as frames. Note that while Figure 2 shows only one frame for each time-series data, each time-series data contains multiple frames.

[0028] The calculation unit 102 calculates the degree of attention for each of the multiple frames (first frame) included in the time series data SDA. The degree of attention represents the degree to which it is given attention in the inference performed by the model MA (first model), which takes the time series data SDA as input and performs inference.

[0029] Frames that receive significant attention can be interpreted as being important frames in the inference process using Model MA, frames that form the basis for the inference process using Model MA, or frames that make a significant contribution to the inference process using Model MA. Therefore, attention can be rephrased as importance or contribution.

[0030] When skeletal sequence data is used as the time series data SDA, the model MA may be a deep learning model such as ST-GCN (Spatial Temporal Graph Convolional Network) that outputs output data including the temporal attention level (temporal attention) for each of the multiple frames included in the skeletal sequence data as the inference result.

[0031] For example, ST-GCN takes skeletal sequence data as input and outputs a feature vector (an example of output data) that includes three-dimensional elements (skeletal points, number of people, and attention level in the time direction). The calculation unit 102 may use such a model MA to calculate the attention level in the time direction as the attention level for each frame.

[0032] For example, the calculation unit 102 compresses elements of dimensions other than the temporal attention level among the elements included in the feature quantity using a dimensionality reduction method such as Global Average pooling, and calculates a feature quantity that includes only the temporal attention level. This feature quantity corresponds to the attention level (temporal attention level) of each frame. The value of the attention level is set to be larger the more attention it receives in inference. Alternatively, instead of performing dimensionality reduction, the system may be configured to calculate the temporal attention level for each skeletal point, and use the attention level with the highest value among the multiple attention levels for each skeletal point.

[0033] Furthermore, the method for calculating attention is not limited to the above; any method that can calculate the degree of attention for each of the multiple frames included in the time-series data SDA using inference by the MA model is acceptable.

[0034] The selection unit 103 uses the attention score calculated by the calculation unit 102 to select frame data FDB, which consists of N frames (second frames) from the time series data SDB.

[0035] For example, the selection unit 103 selects a frame in the time-series data SDB at the same time as the frame with the highest level of attention as the frame data FDB. If there is only one frame with the highest level of attention, the selection unit 103 selects one (N=1) frame data FDB.

[0036] Figure 3 shows an example where there is one frame with the highest level of attention. Figure 3 is an example of a graph showing the level of attention values ​​for each of the multiple frames included in the time series data SDA. In the example in Figure 3, level of attention 301 represents the level of attention with the highest value. In such a case, the selection unit 103 selects one frame from the time series data SDB as frame data FDB, which is at the same time as the frame with level of attention 301.

[0037] Furthermore, if there are two or more frames that attract the most attention, the selection unit 103 may select two or more frames from the time-series data SDB as frame data FDB for each of the two or more frames that attract the most attention, at the same time.

[0038] The method of frame selection by the selection unit 103 is not limited to selecting the frame corresponding to the frame with the highest level of attention. For example, the selection unit 103 may apply the following selection method. • Selection Method 1-1: Select N frames from the multiple frames included in the time-series data SDB that have a maximum level of attention, and N frames at the same time, as the frame data FDB. • Selection Method 1-2: Select a certain number of frames, or a certain percentage (e.g., top M%) of N frames from the multiple frames included in the time-series data SDB, in order of importance, and N frames from the same time period, as the frame data FDB. • Selection Method 1-3: Select N frames from the multiple frames included in the time-series data SDB that have a level of interest greater than the threshold, and N frames at the same time, as the frame data FDB.

[0039] Figure 4 shows an example of applying selection method 1-1. In the example in Figure 4, attention values ​​401 and 402 represent attention values ​​that reach a local maximum. The selection unit 103 can, for example, identify the attention value corresponding to the point where the derivative of the function representing the change in attention value with respect to the frame changes from positive to negative as the attention value that reaches a local maximum.

[0040] Figure 5 shows an example of applying selection method 1-2. In the example in Figure 5, multiple frames included in range 501 containing M% of frames, in descending order of importance, are selected as the frame data FDB. Figure 5 can also be interpreted as an example of applying selection method 1-3. For example, multiple frames included in range 501 containing N frames with an importance greater than the threshold 511 are selected as the frame data FDB.

[0041] Returning to the explanation of Figure 1, the learning and inference unit 104 uses information based on the time series data SDA and information based on the selected N frame data FDB to perform learning or inference on the inference model MB (second model).

[0042] Figure 6 is a block diagram showing an example of the configuration of the learning inference unit 104. As shown in Figure 6, the learning inference unit 104 includes model 601, model 602, and model 603 (third model).

[0043] Model 601 is a model that takes time-series data SDA as input, performs inference, and outputs the inference result. In the example in Figure 6, Model 601 outputs the feature FA (first feature) as the inference result. Model 601 is the same model as Model MA, which is used by the calculation unit 102 when calculating the attention score.

[0044] Model 603 is a model that takes frame data FDB as input, performs inference, and outputs the inference result. In the example in Figure 6, Model 603 outputs the feature vector FB (secondary feature vector) as the inference result. Model 603 could also be a deep learning model such as EfficientNet, which takes frame data FDB as input and outputs the inference result.

[0045] Model 603 may be a model that batches N frames, that is, a model that takes N frames as input and outputs N inference results. Model 603 may also be a model that takes N frames as input one by one and outputs one inference result. In the latter case, the inference by Model 603 is repeated N times, and N inference results are obtained. If multiple inference results are obtained (when N ≥ 2), the statistical values ​​(mean, median, etc.) of the multiple inference results may be used as the final inference result of Model 603. Model 603 may also be a model that takes N frames as input and outputs one inference result.

[0046] Model 602 is a model that uses the inference results of Models 601 and 603 to output the final inference result. In the example in Figure 6, Model 602 corresponds to Model MB, which performs inference using Feature FA, which is an example of information based on time-series data SDA, and Feature FB, which is an example of information based on N selected frame data FDB. For example, Model 602 (Model MB) is a model that is trained to take Feature FA and Feature FB as input and output the inference result.

[0047] When performing inference, the learning inference unit 104 outputs the inference result from model 602 as a processing result.

[0048] When performing training, the learning inference unit 104 trains each model using training data. The training method used by the learning inference unit 104 may be any method applicable to each model. For example, the learning inference unit 104 calculates the value of the loss function using the output (inference result) of model 602 and trains model 602 to optimize the value of the loss function.

[0049] The learning inference unit 104 may learn at least one of model 601 (model MA) and model 603 together with model 602 (model MB). For example, the learning inference unit 104 may learn the parameters of model 601, model 602, and model 603 in order to optimize the value of the loss function. The processing results when performing learning are, for example, parameters representing each learned model.

[0050] If model 601 is trained together with model 602 by the learning inference unit 104, the calculation unit 102 calculates the attention score using model 601 that is currently being trained. If model 601 is not trained together with model 602 by the learning inference unit 104, the calculation unit 102 calculates the attention score using the trained model 601.

[0051] The configuration of the learning inference unit 104 is not limited to the configuration shown in Figure 6. For example, model 601 (model MA) may be a model that outputs probability information PA as the inference result (first inference result) instead of feature FA. Also, model 603 may be a model that outputs probability information PB as the inference result (second inference result) instead of feature FB. Probability information is information that includes probability values ​​representing the likelihood of each of a predetermined set of inference results (such as the actions of multiple people).

[0052] In this case, Model 602 takes probability information PA (another example of information based on time series data SDA) and probability information PB (another example of information based on N selected frame data FDB) as input and outputs the inference result. For example, Model 602 outputs probability information obtained by adding probability information PA and probability information PB as the inference result.

[0053] Returning to the explanation of Figure 1, the output control unit 105 controls the output of various types of information used by the information processing device 100. For example, the output control unit 105 outputs the inference results from the learning inference unit 104. The inference results may include N selected frame data FDBs.

[0054] The selected frame data FDB corresponds to frames of high importance, i.e., frames that were highlighted in the inference, frames that formed the basis of the inference, or frames that were important in the inference. Therefore, by outputting the selected frame data FDB, it becomes possible to understand the basis of the inference, etc.

[0055] The output information from the output control unit 105 can be output by any method, but for example, it can be displayed on a display device such as a display, or transmitted to an external device connected via a network.

[0056] At least a portion of each of the above components (acquisition unit 101, calculation unit 102, selection unit 103, learning inference unit 104, and output control unit 105) may be implemented by one or more processing units. Each of the above components may be implemented by, for example, one or more processors. For example, each of the above components may be implemented by having a processor such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit) execute a program, i.e., by software. Each of the above components may be implemented by a processor such as a dedicated IC (Integrated Circuit), i.e., by hardware. Each of the above components may be implemented by using a combination of software and hardware. When multiple processors are used, each processor may implement one of the above components, or two or more of the above components.

[0057] The memory unit 121 stores various types of information used in the information processing device. For example, the memory unit 121 stores information acquired by the acquisition unit 101 (time series data SDA, time series data SDB), frame data FDB selected by the selection unit 103, and inference results from the learning inference unit 104.

[0058] The storage unit 121 can be composed of any commonly used storage medium, such as flash memory, memory card, RAM (Random Access Memory), HDD (Hard Disk Drive), and optical disc.

[0059] The information processing device 100 may be composed of one physical device or multiple physical devices. For example, the information processing device 100 may be built on a cloud environment. Furthermore, each part of the information processing device 100 may be distributed and provided on multiple devices.

[0060] Next, we will describe the information processing performed by the information processing device 100 of the first embodiment. Figure 7 is a flowchart showing an example of information processing in the first embodiment.

[0061] The acquisition unit 101 acquires time series data SDA and time series data SDB (step S101). The calculation unit 102 uses the time series data SDA to calculate the attention level of each frame included in the time series data SDA (step S102). The selection unit 103 uses the attention levels to select N frames (frame data FDB) from the time series data SDB (step S103). For example, the selection unit 103 selects the frame data FDB corresponding to the frame with the highest attention level from the time series data SDB. The learning and inference unit 104 performs processing (inference or learning) using the time series data SDA and frame data FDB (step S104). The output control unit 105 outputs the processing result (inference result or learning result) (step S105) and terminates the information processing.

[0062] Figure 8 shows an example of the output of the inference result by the output control unit 105. In the example in Figure 8, the output control unit 105 outputs a frame 802 corresponding to the frame data FDB selected by the selection unit 103, along with the skeletal sequence data 801, which is the input time series data SDA. If the output control unit 105 saves the time series data SDA as, for example, a video file, it may save the selected frame data FDB as a thumbnail of the video file as the basis for its decision.

[0063] Thus, in the first embodiment, inference is performed using one type of time series data and frames of other types of time series data selected using the attention level calculated from that time series data. This improves the accuracy of the processing compared to inference using a single time series data. Furthermore, for some time series data, selected frames are used for inference rather than the entire time series data. This reduces the computational load compared to a configuration that performs inference using a time series data SDB containing all frames.

[0064] Furthermore, by using selected frames for inference, it becomes possible to reduce the burden of building the model for inference. For example, for models that perform inference on a portion of the time series data (e.g., one frame) rather than the entire time series data, it is possible to configure the model to use a model that has been pre-trained on publicly available large-scale training data.

[0065] (Second embodiment) In the first embodiment, there was only one type of time-series data used to select a frame. There may be two or more types of time-series data used to select a frame. In the second embodiment, an example of selecting a frame from two different types of time-series data is described.

[0066] Figure 9 is a block diagram showing an example of the configuration of the information processing device 100-2 according to the second embodiment. As shown in Figure 9, the information processing device 100-2 includes an acquisition unit 101-2, a calculation unit 102, a selection unit 103-2, a learning and inference unit 104-2, an output control unit 105, and a storage unit 121.

[0067] In the second embodiment, the functions of the acquisition unit 101-2, the selection unit 103-2, and the learning inference unit 104-2 differ from those of the first embodiment. The other configurations and functions are the same as those in Figure 1, which is a block diagram of the information processing device 100 of the first embodiment, so they are denoted by the same reference numerals and their description is omitted here.

[0068] The acquisition unit 101-2 acquires one time-series data SDA and two time-series data SDB-1 and SDB-2, which are of different types. For example, time-series data SDA is skeletal sequence data, time-series data SDB-1 is video data, and time-series data SDB-2 is sequence data of optical flow calculated from the video data (time-series data SDB-1).

[0069] The selection unit 103-2 selects N frames from each of the two time-series data sets, SDB-1 and SDB-2. Hereafter, the frames selected from time-series data SDB-1 will be referred to as frame data FDB-1, and the frames selected from time-series data SDB-2 will be referred to as frame data FDB-2.

[0070] The method for selecting frames for each time series data (time series data SDB-1, time series data SDB-2) is the same as that of the selection unit 103 in the first embodiment. For example, the selection unit 103-2 selects the frame in time series data SDB-1 (SDB-2) at the same time as the frame with the highest level of attention as frame data FDB-1 (FDB-2).

[0071] The learning and inference unit 104-2 uses information based on the time series data SDA and information based on N frame data FDB-1 and FDB-2 to train or infer the model MB.

[0072] Figure 10 is a block diagram showing an example of the configuration of the learning inference unit 104-2. As shown in Figure 10, the learning inference unit 104-2 includes model 601, model 602-2, model 603, and model 604-2. Note that components identical to those of the learning inference unit 104 in the first embodiment are denoted by the same reference numerals and their descriptions may be omitted.

[0073] Model 603 is the same as in Figure 6, except that the frame data FDB and feature vector FB are replaced with frame data FDB-1 and feature vector FB-1, respectively.

[0074] Model 604-2 is a model that takes frame data FDB-2 as input, performs inference, and outputs the inference result. In the example in Figure 10, Model 604-2 outputs the feature FB-2 (second feature) as the inference result. Model 604-2, like Model 603, can be implemented using a deep learning model such as EfficientNet.

[0075] Model 602-2 is a model that uses the inference results of Models 601, 603, and 604-2 to output the final inference result. In the example in Figure 10, Model 602-2 is a model that is trained to perform inference using features FA, FB-1, and FB-2.

[0076] Similar to the first embodiment, models 601, 603, and 604-2 may output probabilistic information as the inference result instead of features. In this case, model 602-2 outputs the inference result using each piece of probabilistic information. When using three or more types of time series data SDBs, the system can be configured to use a model corresponding to each of the multiple types, as described above.

[0077] The information processing flow by the information processing device 100-2 of the second embodiment is the same as that shown in Figure 7, which illustrates the information processing of the first embodiment, so its explanation will be omitted.

[0078] Thus, in the second embodiment, a configuration can be realized in which frames are selected from each of the two types of time-series data.

[0079] (Third embodiment) In the first embodiment, there was only one type of time-series data used to calculate attention. There may be two or more types of time-series data used to calculate attention. In the third embodiment, an example of calculating attention from two different types of time-series data will be described.

[0080] Figure 11 is a block diagram showing an example of the configuration of an information processing device 100-3 according to the third embodiment. As shown in Figure 11, the information processing device 100-3 includes an acquisition unit 101-3, a calculation unit 102-3, a selection unit 103-3, a learning and inference unit 104-3, an output control unit 105, and a storage unit 121.

[0081] In the third embodiment, the functions of the acquisition unit 101-3, the calculation unit 102-3, the selection unit 103-3, and the learning inference unit 104-3 differ from those of the first embodiment. The other configurations and functions are the same as those in Figure 1, which is a block diagram of the information processing device 100 of the first embodiment, so they are denoted by the same reference numerals and their description is omitted here.

[0082] The acquisition unit 101-3 acquires two time-series data sets SDA-1 and SDA-2, which are of different types, and one time-series data set SDB. For example, time-series data set SDA-1 is skeletal sequence data, time-series data set SDB is video data, and time-series data set SDA-2 is sequence data of optical flow calculated from the video data (time-series data set SDB).

[0083] The calculation unit 102-3 calculates the attention level of each frame for time series data SDA-1 and SDA-2, respectively. The method for calculating the attention level for each time series data (time series data SDA-1 and SDA-2) is the same as that of the calculation unit 102 in the first embodiment. For example, the calculation unit 102-2 uses model MA-1 (an example of the first model), which takes time series data SDA-1 as input and performs inference, to calculate the attention level of each of the multiple frames included in time series data SDA-1 (hereinafter referred to as attention level AD-1). The calculation unit 102-2 also uses model MA-2 (an example of the first model), which takes time series data SDA-2 as input and performs inference, to calculate the attention level of each of the multiple frames included in time series data SDA-2 (hereinafter referred to as attention level AD-2).

[0084] The selection unit 103-3 selects N frames from a single time-series data SDB using attention levels AD-1 and AD-2. The selection unit 103-3 may also apply the following selection method. • Selection Method 2-1: Select the frame at the same time as the frame corresponding to the higher attention level among attention levels AD-1 and AD-2, as the frame data FDB. • Selection Method 2-2: Select the frame data FDB from the frame at the same time as the frame with the highest combined attention score, calculated based on attention scores AD-1 and AD-2. The combined attention score is, for example, the sum of attention scores AD-1 and AD-2. The method for calculating the combined attention score is not limited to this, and any other method may be used. For example, the combined attention score may be a weighted sum of attention scores AD-1 and AD-2. When calculating the combined attention score, attention scores AD-1 and AD-2 may be normalized by the maximum or average value of attention scores AD-1 and AD-2.

[0085] The learning and inference unit 104-3 uses information based on two time series data SDA-1 and SDA-2, and information based on N frame data FDB to train or infer the model MB.

[0086] Figure 12 is a block diagram showing an example of the configuration of the learning inference unit 104-3. As shown in Figure 12, the learning inference unit 104-3 includes model 601, model 602-3, model 603, and model 605-3. Note that components identical to those of the learning inference unit 104 in the first embodiment are denoted by the same reference numerals and their descriptions may be omitted.

[0087] Model 601 is the same as in Figure 6, except that the time series data SDA and feature FA are replaced with time series data SDA-1 and feature FA-1, respectively.

[0088] Model 605-2 is a model that takes time-series data SDA-2 as input, performs inference, and outputs the inference result. In the example in Figure 12, Model 605-2 takes time-series data SDA-2 as input and outputs feature FA-2 (first feature) as the inference result. Model 605-2, like Model 601, can be implemented using a deep learning model such as ST-GCN.

[0089] Model 602-3 is a model that uses the inference results of Models 601, 603, and 605-3 to output the final inference result. In the example in Figure 12, Model 602-3 is a model that is trained to perform inference using features FA-1, FB, and FA-2.

[0090] Similar to the first embodiment, models 601, 603, and 605-3 may output probabilistic information as the inference result instead of features. In this case, model 602-3 outputs the inference result using each piece of probabilistic information. When using three or more types of time series data SDA, the system can be configured to use a model corresponding to each of the multiple types, as described above.

[0091] In the example shown in Figure 12, both time series data SDA-1 and SDA-2 are used for inference, but only a portion of them may be used for inference.

[0092] The information processing flow by the information processing device 100-2 of the second embodiment is the same as that shown in Figure 7, which illustrates the information processing of the first embodiment, so its explanation will be omitted.

[0093] Thus, in the third embodiment, a configuration can be realized in which the level of attention is calculated from each of the two types of time-series data.

[0094] As explained above, according to the first to third embodiments, it is possible to improve the accuracy of processing using machine learning models while suppressing the increase in computational load.

[0095] Next, the hardware configuration of the information processing device according to the first to third embodiments will be described using Figure 13. Figure 13 is an explanatory diagram showing examples of the hardware configuration of the information processing device according to the first to third embodiments.

[0096] The information processing devices of the first to third embodiments include a control device such as a CPU (Central Processing Unit) 51, a storage device such as a ROM (Read Only Memory) 52 and a RAM (Random Access Memory) 53, a communication interface 54 for communication via a network, and a bus 61 for connecting the various parts.

[0097] The programs to be executed in the information processing devices of the first to third embodiments are provided pre-installed in a ROM 52 or the like.

[0098] The programs executed by the information processing devices of the first to third embodiments may be configured to be provided as computer program products by recording them in an installable or executable file format onto a computer-readable recording medium such as a CD-ROM (Compact Disk Read Only Memory), a flexible disk (FD), a CD-R (Compact Disk Recordable), or a DVD (Digital Versatile Disk).

[0099] Furthermore, the information processing device of the first to third embodiments may be configured to store the program on a computer connected to a network such as the Internet and provide it by allowing download via the network. Alternatively, the information processing device of the first to third embodiments may be configured to provide or distribute the program via a network such as the Internet.

[0100] The programs executed in the information processing apparatus of the first to third embodiments can cause the computer to function as a part of the information processing apparatus described above. This computer can read the program from a computer-readable storage medium onto the main memory and execute it using the CPU 51.

[0101] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of Symbols]

[0102] 100, 100-2, 100-3 Information Processing Devices 101, 101-2, 101-3 Acquisition section 102, 102-3 Calculation section 103, 103-2, 103-3 Selection Section 104, 104-2, 104-3 Learning Inference Unit 105 Output control unit 121 Storage section

Claims

1. On the computer, An acquisition step of acquiring one or more first time series data sets of different types from each other, and one or more second time series data sets of different types from the first time series data sets and of different types from each other, A calculation step for each of the multiple first frames included in the first time series data, which is used to perform inference by a first model that takes the first time series data as input and calculates the degree of attention it is given to; A selection step in which N (where N is an integer greater than or equal to 1) second frames are selected from the second time series data using the attention level based on the first time series data, A learning inference step that performs the inference using information based on the first time series data and information based on the selected N second frames, A program to execute.

2. The first model takes the first time series data as input and outputs a first feature quantity representing the characteristics of the first time series data as the result of the inference. The second model takes the first feature and a second feature representing N features of the second frame as input and outputs the result of the inference. The program according to claim 1.

3. The second model receives a first inference result, which is the result of the inference output by the first model, and a second inference result, which is the result of the inference performed by the third model, which takes N selected second frames as input and performs the inference, and outputs the result of the inference. The program according to claim 1.

4. The first time series data and the second time series data have the same start time and end time. The program according to claim 1.

5. The first model is a model that outputs the inference result, including the degree of attention in the time direction, for each of the multiple first frames. The calculation step involves calculating the degree of interest in the time direction using the first model. The program according to claim 1.

6. The learning inference step performs the learning of the first model along with the learning of the second model, The calculation step, when learning is performed by the learning inference step, calculates the time-direction attention using the first model being learned together with the second model. The program according to claim 5.

7. The second time series data has a larger data volume than the first time series data. The program according to claim 1.

8. The selection step involves selecting a second frame from among a plurality of first frames that is at the same time as the first frame which has the greatest level of attention. The program according to claim 1.

9. The acquisition step involves acquiring two or more of the second time-series data, The selection step involves selecting N second frames from each of two or more second time series data. The program according to claim 1.

10. The acquisition step involves acquiring two or more of the first time series data, The calculation step involves calculating the degree of attention for each of the two or more first time series data, The aforementioned selection step is, Select the second frame at the same time as the first frame, which corresponds to the largest of the two or more attention values ​​calculated for the two or more first time series data. or Select the second frame at the same time as the first frame in which the combined attention value calculated based on two or more attention values ​​calculated for two or more of the first time series data is the highest. The program according to claim 1.

11. The selection step involves selecting N second frames from among the plurality of first frames that have the maximum attention value and that are at the same time as N first frames. The program according to claim 1.

12. The output control step further comprises outputting N selected second frames. The program according to claim 1.

13. The frames included in the first time series data and the frames included in the second time series data are any of the following: color image data, skeletal data, optical flow data, depth image data, region segmentation image data, infrared image data, audio data, and X-ray image data. The program according to claim 1.

14. An acquisition unit that acquires one or more first time series data sets of different types from each other, and one or more second time series data sets of different types from the first time series data sets and of different types from each other, For each of the multiple first frames included in the first time series data, a calculation unit calculates a degree of attention, which represents the degree to which the first frame is given attention in the inference performed by the first model that takes the first time series data as input. A selection unit that selects N (where N is an integer greater than or equal to 1) second frames from the second time series data using the attention level based on the first time series data, A learning inference unit that performs the inference or learns a second model that performs the inference using information based on the first time series data and information based on N selected second frames, An information processing device equipped with the following features.

15. An information processing method performed by an information processing device, An acquisition step of acquiring one or more first time series data sets of different types from each other, and one or more second time series data sets of different types from the first time series data sets and of different types from each other, A calculation step for each of the multiple first frames included in the first time series data, which is used to perform inference by a first model that takes the first time series data as input and calculates the degree of attention it is given to; A selection step in which N (where N is an integer greater than or equal to 1) second frames are selected from the second time series data using the attention level based on the first time series data, A learning inference step that performs the inference using information based on the first time series data and information based on the selected N second frames, Information processing methods including

Images