Program, information processing device, and information processing method

Abstract

To suppress calculation amounts from increasing, while improving accuracy of processing using a machine learning model.SOLUTION: A program makes a computer execute each step of acquisition, calculation, selection and learning inference. An acquisition step acquires one or more first time-series data of mutually different kinds, and one or more second time-series data which are different from the first time-series data in the kinds and have mutually different kinds. A calculation step calculates an attention degree which shows a degree focused in inference based on a first model which inputs the first time-series data, about each of a plurality of first frames included in the first time-series data. A selection step selects N-number of second frames from the second time-series data, by using the attention degree. A learning inference step executes learning or inference of a second model which executes the inference, by using information based on the first time-series data, and the information based on the selected N-number of second frames.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] Embodiments of the present invention relate to a program, an information processing apparatus, and an information processing method.

Background Art

[0002] Techniques have been proposed for inputting time series data (sequence data) into a machine learning model and performing various inferences. The machine learning model is, for example, a deep learning model. For example, a model has been proposed that inputs video data obtained by imaging a moving object such as a person, or skeleton sequence data obtained by analyzing the video data, and infers the actions of the moving object (such as the behavior of a person).

[0003] If inferences are performed using multiple types of time series data, it is expected that the accuracy of the inferences will be improved. On the other hand, since time series data is data that includes data (frames) at multiple times, in a configuration that uses multiple time series data, the amount of computation for processes such as inference and learning by the machine learning model may increase.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Non-Patent Documents

[0005]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] An object of the present invention is to provide a program, an information processing apparatus, and an information processing method capable of improving the accuracy of processing using a machine learning model while suppressing an increase in the amount of calculation.

Means for Solving the Problems

[0007] The program according to the embodiment is a program for causing a computer to execute each step of acquisition, calculation, selection, and learning inference. The acquisition step acquires one or more first time-series data of different types from each other, and one or more second time-series data of different types from the first time-series data and different types from each other. The calculation step calculates, for each of a plurality of first frames included in the first time-series data, an attention degree representing the degree of attention focused by inference using a first model that inputs the first time-series data. The selection step selects N second frames from the second time-series data using the attention degree. The learning inference step performs learning or inference of a second model that executes inference using information based on the first time-series data and information based on the selected N second frames.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Embodiments for Carrying Out the Invention

[0009] Hereinafter, with reference to the accompanying drawings, preferred embodiments of the information processing apparatus according to the present invention will be described in detail.

[0010] Hereinafter, an example of a configuration using a model that inputs a plurality of types of time series data (sequence data) and estimates (infers) human behavior will be mainly described. The inference to which the following embodiments are applicable is not limited to such human behavior estimation, but may be any inference using time series data. For example, a configuration for estimating the actions of moving objects other than humans, such as animals and robots, may also be used.

[0011] Time series data is data including a plurality of data (hereinafter, frames) continuous in time series. Each of the plurality of frames included in the time series data is, for example, data of the following types, but is not limited thereto. · Color image data (e.g., RGB image data) · Skeleton data · Optical flow data · Depth image data · Region division image data · Infrared image data · X-ray image data · Audio data

[0012] Note that the type of data (time series data) may sometimes be referred to as the modality of the data or the format of the data. The following embodiments can also be interpreted as examples of performing inference using multimodal time series data.

[0013] As described above, by performing inference using multiple types of time series data, it becomes possible to improve the accuracy of the inference. On the other hand, depending on the type of time series data, it may not be necessary to use all the frames included in the time series data for the target inference.

[0014] For example, consider a case where an action of a certain person eating a certain food is estimated using time series data of skeletal data (hereinafter referred to as skeletal sequence data) and time series data of color image data (hereinafter referred to as video data). In such a case, the action of eating the food is estimated using, for example, skeletal sequence data including a plurality of frames. On the other hand, the food being eaten can potentially be estimated using only the frames in which the food is imaged among the frames included in the video data, for example.

[0015] Therefore, when performing inference using multiple types of time series data, for some types of time series data, even if the inference is performed using some frames instead of all the frames included in the time series data, the accuracy of the inference can be improved.

[0016] (First Embodiment) The information processing apparatus according to the first embodiment performs a predetermined inference using multiple types of time series data. At this time, in this embodiment, using the attention degree of each frame calculated from a certain type of time series data (hereinafter referred to as time series data SDA), N (N is an integer of 1 or more) frames (hereinafter referred to as frame data FDB) are selected from another type of time series data (hereinafter referred to as time series data SDB), and the inference is performed using the selected frame data FDB. Thereby, compared with a configuration in which the inference is performed using the time series data SDB including all the frames, the amount of calculation can be reduced.

[0017] The degree of attention is information representing the degree of attention in inference using time-series data SDA. And, for example, the frame data FDB at the time corresponding to the frame with the maximum degree of attention is selected. Therefore, even without using the time-series data SDB including all frames, the accuracy of the processing using the machine learning model can be improved.

[0018] FIG. 1 is a block diagram showing an example of the configuration of the information processing apparatus 100 according to the first embodiment. As shown in FIG. 1, the information processing apparatus 100 includes an acquisition unit 101, a calculation unit 102, a selection unit 103, a learning inference unit 104, an output control unit 105, and a storage unit 121.

[0019] The acquisition unit 101 acquires various information used in the information processing apparatus 100. For example, the acquisition unit 101 acquires one or more time-series data SDA (first time-series data) of different types from each other, and time-series data SDB (second time-series data) of a different type from the time-series data SDA and of different types from each other.

[0020] Hereinafter, in the first embodiment, an example in which one time-series data SDA and one time-series data SDB are acquired will be described. An example using two time-series data SDB will be described in the second embodiment. An example using two time-series data SDA will be described in the third embodiment. By combining the second embodiment and the third embodiment, a configuration using two or more time-series data SDA and two or more time-series data SDB can be realized.

[0021] As will be described later, the time-series data SDA corresponds to time-series data in which all frames are used for action estimation. On the other hand, the time-series data SDB corresponds to time-series data in which a part of N frames are selected and the selected frames are used for action estimation.

[0022] The time-series data SDA and the time-series data SDB may be data with the same start time and end time. Thereby, the process of associating time among a plurality of time-series data becomes unnecessary.

[0023] When the frame rates are different among a plurality of time-series data, for example, the acquisition unit 101 performs a process of interpolating frames for the time-series data with a lower frame rate to make the frame rates of the plurality of time-series data match.

[0024] Time-series data with a larger data volume may be acquired as the time-series data for which frames are selected. That is, the time-series data SDB may have a larger data volume than the time-series data SDA. Thereby, the reduction amount of the data volume can be further increased.

[0025] For example, the acquisition unit 101 may acquire skeleton sequence data as the time-series data SDA and acquire video data as the time-series data SDB. The skeleton sequence data is, for example, time-series coordinate data representing the joint points of a person. The skeleton sequence data may be acquired by any method, but can be obtained, for example, by a skeleton detection method such as Openpose and a method using a sensor.

[0026] Hereinafter, it is assumed that there is one person to be estimated. When the time-series data includes data of a plurality of persons, the acquisition unit 101 may generate and acquire time-series data including data cut out for each person using an object detector or the like. When performing inferences other than the action estimation of a person, a process of cutting out data for each person by object detection or the like may not be executed.

[0027] FIG. 2 is a diagram showing an example of the acquired time-series data. In FIG. 2, examples of the time-series data SDA including skeleton data as frames and the time-series data SDB including RGB image data as frames are shown. Note that in FIG. 2, only one frame is shown for each time-series data, but each time-series data includes a plurality of frames.

[0028] The calculation unit 102 calculates the attention level for each of the plurality of frames (first frame) included in the time-series data SDA. The attention level represents the degree of attention in the inference by the model MA (first model) that inputs the time-series data SDA and executes the inference.

[0029] Frames with a high attention level can be interpreted as frames that are important in the inference by the model MA, frames that are the basis for the inference by the model MA, or frames that make a large contribution to the inference by the model MA. Therefore, the attention level can be rephrased as importance level or contribution level, etc.

[0030] When skeleton sequence data is used as the time-series data SDA, the model MA may be a deep learning model such as ST-GCN (Spatial Temporal Graph Convolional Network) that outputs, as the result of inference (inference result), output data including the attention level in the time direction (Attention in the time direction) for each of the plurality of frames included in the skeleton sequence data.

[0031] For example, ST-GCN inputs the skeleton sequence data and outputs a feature amount (an example of output data) including three-dimensional elements of (skeleton points, number of people, attention level in the time direction). The calculation unit 102 may calculate the attention level in the time direction as the attention level for each frame using such a model MA.

[0032] For example, the calculation unit 102 compresses the elements in dimensions other than the attention level in the time direction among the elements included in the feature amount using a dimension compression method such as Global Average pooling, and calculates a feature amount including only the attention level in the time direction. This feature amount corresponds to the attention level (attention level in the time direction) for each frame. It is assumed that the value of the attention level increases as the degree of attention in the inference increases. Note that it may be configured not to perform dimension compression, for example, to calculate the attention level in the time direction for each skeleton point and use the attention level with the maximum value among the plurality of attention levels for each skeleton point.

[0033] Note that the method for calculating the attention degree is not limited to the above. Any method may be used as long as it can calculate the degree of attention noticed by the inference using the model MA for each of the plurality of frames included in the time series data SDA.

[0034] The selection unit 103 selects frame data FDB, which is N frames (second frames), from the time series data SDB using the attention degree calculated by the calculation unit 102.

[0035] For example, the selection unit 103 selects, as the frame data FDB, the frame in the time series data SDB at the same time as the time of the frame with the maximum attention degree. When there is one frame with the maximum attention degree, the selection unit 103 selects one (N = 1) frame data FDB.

[0036] FIG. 3 is a diagram showing an example when there is one frame with the maximum attention degree. FIG. 3 is an example of a graph showing the values of the attention degree for each of the plurality of frames included in the time series data SDA. In the example of FIG. 3, the attention degree 301 represents the attention degree with the maximum value. In such a case, the selection unit 103 selects one frame at the same time as the frame with the attention degree 301 from the time series data SDB as the frame data FDB.

[0037] Note that when there are two or more frames with the maximum attention degree, the selection unit 103 may select, as the frame data FDB, two or more frames at the same time as each of the two or more frames with the maximum attention degree from the time series data SDB.

[0038] The method for selecting frames by the selection unit 103 is not limited to the method of selecting the frame corresponding to the frame with the maximum attention degree. For example, the selection unit 103 may apply the following selection method. · Selection method 1-1: Select, as the frame data FDB, N frames at the same time as the N frames with the maximum attention degree among the plurality of frames included in the time series data SDB. · Selection method 1-2: Select N frames at the same time as N frames with a certain number or a certain ratio (e.g., top M%) in descending order of attention among the multiple frames included in the time-series data SDB as the frame data FDB. · Selection method 1-3: Select N frames at the same time as N frames with attention greater than the threshold among the multiple frames included in the time-series data SDB as the frame data FDB.

[0039] Figure 4 is a diagram showing an example of applying the selection method 1-1. In the example of Figure 4, the attentions 401 and 402 represent the attentions that become the maximum values. The selection unit 103 can specify, for example, the attention corresponding to the point where the differential value of the function representing the change in attention with respect to the frame changes from positive to negative as the attention that becomes the maximum value.

[0040] Figure 5 is a diagram showing an example of applying the selection method 1-2. In the example of Figure 5, a plurality of frames included in the range 501 containing M% of the frames in descending order of attention are selected as the frame data FDB. Figure 5 can also be interpreted as an example of applying the selection method 1-3. For example, a plurality of frames included in the range 501 containing N frames with attention greater than the threshold 511 are selected as the frame data FDB.

[0041] Return to the description of Figure 1. The learning and inference unit 104 performs learning or inference of the model MB (second model) that executes inference using the information based on the time-series data SDA and the information based on the selected N frame data FDB.

[0042] Figure 6 is a block diagram showing an example of the configuration of the learning and inference unit 104. As shown in Figure 6, the learning and inference unit 104 includes a model 601, a model 602, and a model 603 (third model).

[0043] Model 601 is a model that takes in time-series data SDA, performs inference, and outputs the inference result. In the example of FIG. 6, model 601 outputs feature quantity FA (first feature quantity) as the inference result. Model 601 is, for example, the same model as model MA used when calculation unit 102 calculates the attention level.

[0044] Model 603 is a model that takes in frame data FDB, performs inference, and outputs the inference result. In the example of FIG. 6, model 603 outputs feature quantity FB (second feature quantity) as the inference result. Model 603 may be, for example, a deep learning model such as EfficientNet that takes in frame data FDB and outputs the inference result.

[0045] Model 603 may be a model that batch-processes N frames, that is, a model that inputs N frames together and outputs N inference results. Model 603 may be a model that inputs N frames one by one and outputs one inference result. In the latter case, the inference by model 603 is repeated N times, and N inference results are obtained. When a plurality of inference results are obtained (when N ≧ 2), statistical values (average value, median, etc.) of the plurality of inference results may be used as the final inference result of model 603. Model 603 may be a model that inputs N frames together and outputs one inference result.

[0046] Model 602 is a model that uses the inference results of model 601 and model 603 to output the final inference result. In the example of FIG. 6, model 602 corresponds to model MB, which is a model that performs inference using feature quantity FA, which is an example of information based on time-series data SDA, and feature quantity FB, which is an example of information based on the selected N frame data FDB. For example, model 602 (model MB) is a model that is learned to input feature quantity FA and feature quantity FB and output the inference result.

[0047] Note that when performing inference, learning inference unit 104 outputs the inference result by model 602 as a processing result.

[0048] When performing learning, the learning and inference unit 104 learns each model using learning data. The learning method by the learning and inference unit 104 may be a method applicable to each model. For example, the learning and inference unit 104 calculates the value of the loss function using the output (inference result) of the model 602, and learns the model 602 so as to optimize the value of the loss function.

[0049] The learning and inference unit 104 may learn at least one of the model 601 (model MA) and the model 603 together with the model 602 (model MB). For example, the learning and inference unit 104 may learn the parameters of the model 601, the parameters of the model 602, and the parameters of the model 603 so as to optimize the value of the loss function. The processing result when performing learning is, for example, the parameters representing each learned model.

[0050] When the model 601 is learned together with the model 602 by the learning and inference unit 104, the calculation unit 102 calculates the attention using the model 601 being learned. When the model 601 is not learned together with the model 602 by the learning and inference unit 104, the calculation unit 102 calculates the attention using the learned model 601.

[0051] The configuration of the learning and inference unit 104 is not limited to the configuration shown in FIG. 6. For example, the model 601 (model MA) may be a model that outputs probability information PA as an inference result (first inference result) instead of the feature amount FA. Also, the model 603 may be a model that outputs probability information PB as an inference result (second inference result) instead of the feature amount FB. The probability information is, for example, information including probability values representing the likelihood of each of a plurality of predetermined inference results (such as the actions of a plurality of people).

[0052] In this case, the model 602 inputs probability information PA (another example of information based on the time-series data SDA) and probability information PB (another example of information based on the selected N frame data FDB), and outputs an inference result. For example, the model 602 outputs the probability information obtained by adding the probability information PA and the probability information PB as the inference result.

[0053] Returning to the description of FIG. 1. The output control unit 105 controls the output of various types of information used in the information processing apparatus 100. For example, the output control unit 105 outputs the inference result by the learning inference unit 104. The inference result may include the selected N frame data FDB.

[0054] The selected frame data FDB corresponds to a frame with a high degree of attention, that is, a frame that is noticed in the inference, a frame that is the basis for the inference, or a frame that is important in the inference. Therefore, by outputting the selected frame data FDB, it becomes possible to grasp the basis of the inference and the like.

[0055] The output information of the information by the output control unit 105 may be in any method. For example, methods such as displaying on a display device such as a display, and transmitting to an external device connected via a network can be applied.

[0056] At least a part of the above-described respective units (acquisition unit 101, calculation unit 102, selection unit 103, learning and inference unit 104, and output control unit 105) may be realized by one or more processing units. The above-described respective units may be realized by, for example, one or a plurality of processors. For example, the above-described respective units may be realized by causing a processor such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit) to execute a program, that is, by software. The above-described respective units may be realized by a processor such as a dedicated IC (Integrated Circuit), that is, by hardware. The above-described respective units may be realized by using a combination of software and hardware. When using a plurality of processors, each processor may realize one of the respective units or may realize two or more of the respective units.

[0057] The storage unit 121 stores various information used in the information processing apparatus. For example, the storage unit 121 stores information (time-series data SDA, time-series data SDB) acquired by the acquisition unit 101, frame data FDB selected by the selection unit 103, and inference results by the learning and inference unit 104, and the like.

[0058] Note that the storage unit 121 can be configured by any generally used storage medium such as a flash memory, a memory card, a RAM (Random Access Memory), an HDD (Hard Disk Drive), and an optical disk.

[0059] The information processing apparatus 100 may be physically configured by one device or may be physically configured by a plurality of devices. For example, the information processing apparatus 100 may be constructed in a cloud environment. Also, each unit in the information processing apparatus 100 may be provided in a distributed manner among a plurality of devices.

[0060] Next, information processing by the information processing apparatus 100 of the first embodiment will be described. FIG. 7 is a flowchart showing an example of information processing in the first embodiment.

[0061] The acquisition unit 101 acquires the time-series data SDA and the time-series data SDB (step S101). The calculation unit 102 calculates the attention level of each frame included in the time-series data SDA using the time-series data SDA (step S102). The selection unit 103 selects N frames (frame data FDB) from the time-series data SDB using the attention level (step S103). For example, the selection unit 103 selects the frame data FDB corresponding to the frame with the highest attention level from the time-series data SDB. The learning and inference unit 104 executes processing (inference or learning) using the time-series data SDA and the frame data FDB (step S104). The output control unit 105 outputs the processing result (inference result or learning result) (step S105) and ends the information processing.

[0062] FIG. 8 is a diagram showing an example of the output of the inference result by the output control unit 105. In the example of FIG. 8, the output control unit 105 outputs the frame 802 corresponding to the frame data FDB selected by the selection unit 103 together with the skeleton sequence data 801 which is the input time-series data SDA. When the output control unit 105 stores the time-series data SDA as, for example, a video file, the selected frame data FDB may be stored as a thumbnail of the video file as a basis for judgment.

[0063] As described above, in the first embodiment, inference is executed using a certain type of time-series data and frames of another type of time-series data selected using the attention level calculated from the time-series data. Thereby, the accuracy of the processing can be improved as compared with the inference using one piece of time-series data. Also, for some of the time-series data, instead of the entire time-series data, the selected frames are used for inference. Thereby, the amount of calculation can be reduced as compared with a configuration in which inference is performed using the time-series data SDB including all the frames.

[0064] Also, by performing inference using the selected frame, it is possible to reduce the load of building a model for inference. For example, for a model that performs inference on a part of the frames (e.g., one frame) instead of the entire time-series data, it can be configured to use a model pre-trained with publicly available large-scale learning data.

[0065] (Second Embodiment) In the first embodiment, the time-series data for selecting a frame was of one type. The time-series data for selecting a frame may be two or more types. In the second embodiment, an example of selecting a frame from each of two types of time-series data will be described.

[0066] FIG. 9 is a block diagram showing an example of the configuration of the information processing apparatus 100-2 according to the second embodiment. As shown in FIG. 9, the information processing apparatus 100-2 includes an acquisition unit 101-2, a calculation unit 102, a selection unit 103-2, a learning and inference unit 104-2, an output control unit 105, and a storage unit 121.

[0067] In the second embodiment, the functions of the acquisition unit 101-2, the selection unit 103-2, and the learning and inference unit 104-2 are different from those in the first embodiment. Other configurations and functions are the same as those in FIG. 1, which is a block diagram of the information processing apparatus 100 according to the first embodiment, so the same reference numerals are used and the description here is omitted.

[0068] The acquisition unit 101-2 acquires one time-series data SDA, and two time-series data SDB-1 and SDB-2 that are different in type from each other. For example, the time-series data SDA is skeleton sequence data, the time-series data SDB-1 is video data, and the time-series data SDB-2 is optical flow sequence data calculated from the video data (time-series data SDB-1).

[0069] The selection unit 103-2 selects N frames from each of the two time-series data SDB-1 and SDB-2. Hereinafter, the frame selected from the time-series data SDB-1 is referred to as frame data FDB-1, and the frame selected from the time-series data SDB-2 is referred to as frame data FDB-2.

[0070] For each time-series data (time-series data SDB-1, time-series data SDB-2), the method of selecting frames is the same as that of the selection unit 103 in the first embodiment. For example, the selection unit 103-2 selects, as the frame data FDB-1 (FDB-2), the frame within the time-series data SDB-1 (SDB-2) at the same time as the time of the frame with the maximum degree of attention.

[0071] The learning and inference unit 104-2 performs learning or inference of the model MB using the information based on the time-series data SDA and the information based on the N frame data FDB-1 and FDB-2.

[0072] FIG. 10 is a block diagram showing an example of the configuration of the learning and inference unit 104-2. As shown in FIG. 10, the learning and inference unit 104-2 includes a model 601, a model 602-2, a model 603, and a model 604-2. For the same configuration as that of the learning and inference unit 104 in the first embodiment, the same reference numerals may be used and the description may be omitted.

[0073] The model 603 is the same as FIG. 6 except that the frame data FDB and the feature amount FB are replaced with the frame data FDB-1 and the feature amount FB-1, respectively.

[0074] The model 604-2 is a model that inputs the frame data FDB-2, performs inference, and outputs the inference result. In the example of FIG. 10, the model 604-2 outputs the feature amount FB-2 (second feature amount) as the inference result. The model 604-2 can be realized by a deep learning model such as EfficientNet in the same manner as the model 603.

[0075] Model 602-2 is a model that outputs the final inference result using the inference results of model 601, model 603, and model 604-2. In the example of FIG. 10, model 602-2 is a model that is learned to perform inference using feature quantity FA, feature quantity FB-1, and feature quantity FB-2.

[0076] Similar to the first embodiment, model 601, model 603, and model 604-2 may output probability information as the inference result instead of the feature quantity. In this case, model 602-2 outputs the inference result using each probability information. When using three or more types of time-series data SDB, it may be configured to use models corresponding to each of the plurality of types in the same manner as described above.

[0077] Since the information processing flow by the information processing apparatus 100-2 of the second embodiment is the same as that of FIG. 7 showing the information processing of the first embodiment, the description thereof is omitted.

[0078] As described above, in the second embodiment, a configuration for selecting frames from each of the two types of time-series data can be realized.

[0079] (Third Embodiment) In the first embodiment, there was one type of time-series data for calculating the attention degree. There may be two or more types of time-series data for calculating the attention degree. In the third embodiment, an example of calculating the attention degree from each of the two types of time-series data will be described.

[0080] FIG. 11 is a block diagram showing an example of the configuration of the information processing apparatus 100-3 of the third embodiment. As shown in FIG. 11, the information processing apparatus 100-3 includes an acquisition unit 101-3, a calculation unit 102-3, a selection unit 103-3, a learning inference unit 104-3, an output control unit 105, and a storage unit 121.

[0081] In the third embodiment, the functions of the acquisition unit 101-3, the calculation unit 102-3, the selection unit 103-3, and the learning and inference unit 104-3 are different from those in the first embodiment. Since the other configurations and functions are the same as those in FIG. 1, which is a block diagram of the information processing apparatus 100 in the first embodiment, the same reference numerals are given and the description thereof is omitted here.

[0082] The acquisition unit 101-3 acquires two time-series data SDA-1 and SDA-2 that are different in type from each other, and one time-series data SDB. For example, the time-series data SDA-1 is skeleton sequence data, the time-series data SDB is video data, and the time-series data SDA-2 is sequence data of optical flow calculated from the video data (time-series data SDB).

[0083] The calculation unit 102-3 calculates the attention degree of each frame for each of the time-series data SDA-1 and SDA-2. The method for calculating the attention degree for each time-series data (time-series data SDA-1 and SDA-2) is the same as that of the calculation unit 102 in the first embodiment. For example, the calculation unit 102-2 uses a model MA-1 (an example of the first model) that inputs the time-series data SDA-1 and executes inference to calculate the attention degree (hereinafter referred to as attention degree AD-1) of each of the plurality of frames included in the time-series data SDA-1. Further, the calculation unit 102-2 uses a model MA-2 (an example of the first model) that inputs the time-series data SDA-2 and executes inference to calculate the attention degree (hereinafter referred to as attention degree AD-2) of each of the plurality of frames included in the time-series data SDA-2.

[0084] The selection unit 103-3 selects N frames from one time-series data SDB using the attention degree AD-1 and the attention degree AD-2. The selection unit 103-3 may apply the following selection method. · Selection method 2-1: Select the frame at the same time as the frame corresponding to the attention degree with the larger value among the attention degrees AD-1 and AD-2 as the frame data FDB. · Selection method 2-2: Select the frame at the same time as the frame with the maximum integrated attention calculated based on attention AD-1 and attention AD-2 as the frame data FDB. The integrated attention is, for example, the value obtained by adding attention AD-1 and attention AD-2. The method for calculating the integrated attention is not limited to this, and any other method may be used. For example, the integrated attention may be the value obtained by weighted addition of attention AD-1 and attention AD-2. When calculating the integrated attention, attention AD-1 and attention AD-2 may be normalized by the maximum value or average value of attention AD-1 and attention AD-2, etc.

[0085] The learning and inference unit 104-3 performs learning or inference of the model MB using the information based on the two time-series data SDA-1 and SDA-2 and the information based on the N frame data FDB.

[0086] FIG. 12 is a block diagram showing an example of the configuration of the learning and inference unit 104-3. As shown in FIG. 12, the learning and inference unit 104-3 includes a model 601, a model 602-3, a model 603, and a model 605-3. For the same configuration as the learning and inference unit 104 in the first embodiment, the same reference numerals may be used and the description may be omitted.

[0087] The model 601 is the same as FIG. 6 except that the time-series data SDA and the feature amount FA are replaced with the time-series data SDA-1 and the feature amount FA-1, respectively.

[0088] The model 605-2 is a model that inputs the time-series data SDA-2, performs inference, and outputs the inference result. In the example of FIG. 12, the model 605-2 inputs the time-series data SDA-2 and outputs the feature amount FA-2 (first feature amount) as the inference result. The model 605-2 can be realized by a deep learning model such as ST-GCN, similar to the model 601.

[0089] Model 602-3 is a model that outputs the final inference result using the inference results of Model 601, Model 603, and Model 605-3. In the example of FIG. 12, Model 602-3 is a model that is learned to perform inference using feature quantity FA-1, feature quantity FB, and feature quantity FA-2.

[0090] Similar to the first embodiment, Model 601, Model 603, and Model 605-3 may output probability information as the inference result instead of the feature quantity. In this case, Model 602-3 outputs the inference result using each probability information. When using three or more types of time-series data SDA, it may be configured to use models corresponding to each of the plurality of types in the same manner as described above.

[0091] In the example of FIG. 12, both of the two time-series data SDA-1 and SDA-2 are used for inference, but only a part of these may be used for inference.

[0092] Since the information processing flow by the information processing apparatus 100-2 of the second embodiment is the same as that of FIG. 7 showing the information processing of the first embodiment, the description thereof is omitted.

[0093] As described above, in the third embodiment, a configuration for calculating the degree of attention from each of the two types of time-series data can be realized.

[0094] As described above, according to the first to third embodiments, it is possible to improve the accuracy of processing using the machine learning model while suppressing an increase in the amount of calculation.

[0095] Next, the hardware configuration of the information processing apparatus according to the first to third embodiments will be described with reference to FIG. 13. FIG. 13 is an explanatory diagram showing a hardware configuration example of the information processing apparatus according to the first to third embodiments.

[0096] The information processing apparatuses according to the first to third embodiments include a control device such as a CPU (Central Processing Unit) 51, a storage device such as a ROM (Read Only Memory) 52 and a RAM (Random Access Memory) 53, a communication I / F 54 that connects to a network and performs communication, and a bus 61 that connects each part.

[0097] The programs executed by the information processing apparatuses according to the first to third embodiments are provided by being pre - incorporated in the ROM 52 or the like.

[0098] The programs executed by the information processing apparatuses according to the first to third embodiments may be configured to be recorded on a computer - readable recording medium such as a CD - ROM (Compact Disk Read Only Memory), a flexible disk (FD), a CD - R (Compact Disk Recordable), a DVD (Digital Versatile Disk) in an installable format or an executable format file, and provided as a computer program product.

[0099] Furthermore, the programs executed by the information processing apparatuses according to the first to third embodiments may be configured to be stored on a computer connected to a network such as the Internet, and provided by being downloaded via the network. Also, the programs executed by the information processing apparatuses according to the first to third embodiments may be configured to be provided or distributed via a network such as the Internet.

[0100] The programs executed by the information processing apparatuses according to the first to third embodiments can cause a computer to function as each part of the above - described information processing apparatus. This computer can have the CPU 51 read a program from a computer - readable storage medium onto the main storage device and execute it.

[0101] Although some embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalent scope thereof.

Explanation of Reference Numerals

[0102] 100, 100-2, 100-3 Information Processing Apparatus 101, 101-2, 101-3 Acquisition Unit 102, 102-3 Calculation Unit 103, 103-2, 103-3 Selection Unit 104, 104-2, 104-3 Learning and Inference Unit 105 Output Control Unit 121 Storage Unit

Claims

1. Cause a computer to perform an acquisition step of acquiring one or more first time-series data of different types from each other, and one or more second time-series data that are different in type from the first time-series data and different in type from each other; a calculation step of calculating, for each of a plurality of first frames included in the first time-series data, an attention degree representing the degree of attention focused in the inference by a first model that inputs the first time-series data and performs the inference; a selection step of selecting N (N is an integer of 1 or more) second frames from the second time-series data using the attention degree; a learning inference step of learning a second model that performs the inference or performing the inference using information based on the first time-series data and information based on the selected N second frames; A program for causing the above to be executed.

2. The first model inputs the first time-series data and outputs, as a result of the inference, a first feature amount representing a feature of the first time-series data, The second model inputs the first feature amount and a second feature amount representing features of the N second frames, and outputs a result of the inference. The program according to claim 1.

3. The second model inputs a first inference result that is a result of the inference output by the first model and a second inference result that is a result of the inference by a third model that inputs the selected N second frames and performs the inference, and outputs a result of the inference. The program according to claim 1.

4. The start time and the end time of the first time-series data and the second time-series data match. The program according to claim 1.

5. The first model is a model that outputs, for each of the plurality of first frames, a result of the inference including attention degree in the time direction, The calculation step calculates the attention degree in the time direction using the first model. The program according to claim 1.

6. The learning inference step performs learning of the first model together with learning of the second model, When learning is performed by the learning inference step, the calculation step calculates the attention degree in the time direction using the first model being learned together with the second model. The program according to claim 5.

7. The second time-series data has a larger data volume than the second time-series data. The program according to claim 1.

8. The selection step selects the second frame at the same time as the first frame having the highest degree of attention among the plurality of first frames. The program according to claim 1.

9. The acquisition step acquires two or more pieces of the second time-series data. The selection step selects N second frames from each of the two or more pieces of the second time-series data. The program according to claim 1.

10. The acquisition step acquires two or more pieces of the first time-series data. The calculation step calculates the degree of attention for each of the two or more pieces of the first time-series data. The selection step selects the second frame at the same time as the first frame corresponding to the degree of attention having a large value among the two or more degrees of attention calculated for the two or more pieces of the first time-series data, or selects the second frame at the same time as the first frame for which the integrated degree of attention calculated based on the two or more degrees of attention calculated for the two or more pieces of the first time-series data is the maximum. The program according to claim 1.

11. The selection step selects N second frames at the same time as the N first frames having the maximum value of the degree of attention among the plurality of first frames. The program according to claim 1.

12. Further includes an output control step of outputting the selected N second frames. The program according to claim 1.

13. The frames included in the first time-series data and the frames included in the second time-series data are any one of color image data, skeleton data, optical flow data, depth image data, region division image data, infrared image data, audio data, and X-ray image data. The program according to claim 1.

14. An acquisition unit that acquires one or more first time-series data of different types from each other, and one or more second time-series data that are different in type from the first time-series data and are different in type from each other, A calculation unit that calculates a degree of attention representing the degree of being noticed in the inference by the first model that inputs the first time-series data and executes inference for each of the plurality of first frames included in the first time-series data, A selection unit that selects N (N is an integer of 1 or more) second frames from the second time-series data using the degree of attention. A learning inference unit that performs learning of a second model that executes the inference or executes the inference, using information based on the first time-series data and information based on the selected N second frames. An information processing apparatus comprising the same. **Claim 15** An information processing method executed by an information processing apparatus, the method comprising: an acquisition step of acquiring one or more first time-series data of different types from each other, and one or more second time-series data of different types from the first time-series data and different from each other; a calculation step of calculating, for each of a plurality of first frames included in the first time-series data, an attention degree representing a degree of attention noted in the inference by a first model that inputs the first time-series data and executes the inference; a selection step of selecting N (N is an integer of 1 or more) second frames from the second time-series data using the attention degree; a learning inference step of performing learning of a second model that executes the inference or executing the inference, using information based on the first time-series data and information based on the selected N second frames; An information processing method including the above.

Images