Information processing device, information processing method, and program

Abstract

To improve the efficiency of creation operation of teacher data in performing machine learning of events of data series.SOLUTION: An information processing device includes a division unit for dividing a set of feature amounts constituting feature amount sequences corresponding to a plurality of data sequences, respectively into a plurality of partial feature amount sets, a clustering unit for clustering a set of the partial feature amount sets acquired from the plurality of data sequences to a plurality of clusters, and a selection unit for selecting data to be an application object of a correct label from a set of data constituting at least a portion of any of the plurality of data sequences corresponding to at least one partial feature amount set included in each cluster.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] To train a machine learning model that makes predictions on a data series, it is known to use training data in which correct labels are assigned to only some of the data, instead of using training data in which correct labels are assigned to all of the data that make up multiple data series.

[0003] To generate such training data, it is important to select data to which a correct label should be assigned from among data constituting multiple data series. For example, Patent Document 1 describes a technique for classifying frames constituting a video into multiple clusters based on the feature amount of each frame. It is conceivable to select frames to which a correct label should be assigned from each cluster into which frames are classified using such a technique. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-60413 Summary of the Invention [Problem to be solved by the invention]

[0005] When using the technology described in Patent Document 1 to select frames to which correct labels should be assigned, there was a problem that the longer the video, the greater the number of frames, which increases the calculation cost, because clustering is performed on a frame-by-frame basis.

[0006] The present disclosure has been made in consideration of the above-mentioned problems, and one exemplary purpose thereof is to provide a technology for making it more efficient to create training data when performing machine learning on events in a data series. [Means for solving the problem]

[0007] An information processing device according to an exemplary aspect of the present disclosure includes: a division means for dividing a set of features constituting a feature sequence corresponding to each of a plurality of data sequences into a plurality of partial feature sets; a clustering means for clustering the set of partial feature sets obtained from the plurality of data sequences into a plurality of clusters; and a selection means for selecting data to be assigned a correct label from a set of data constituting at least a part of any of the plurality of data sequences, which corresponds to at least one of the partial feature sets included in each cluster.

[0008] An information processing method according to an exemplary aspect of the present disclosure includes: a division process, performed by at least one processor, dividing a set of features constituting a feature sequence corresponding to each of a plurality of data sequences into a plurality of partial feature sets; a clustering process, performed by the at least one processor, clustering the set of partial feature sets obtained from the plurality of data sequences into a plurality of clusters; and a selection process, performed by the at least one processor, selecting data to be assigned a correct label from a set of data constituting at least a part of any of the plurality of data sequences, which corresponds to at least one of the partial feature sets included in each cluster. Contains:

[0009] A program according to an exemplary aspect of the present disclosure causes at least one processor to execute a division process of dividing a set of features constituting a feature sequence corresponding to each of a plurality of data sequences into a plurality of partial feature sets; a clustering process of clustering the set of partial feature sets obtained from the plurality of data sequences into a plurality of clusters; and a selection process of selecting data to be assigned a correct label from a set of data constituting at least a part of any of the plurality of data sequences, which corresponds to at least one of the partial feature sets included in each cluster. [Effects of the Invention]

[0010] According to an exemplary aspect of the present disclosure, an exemplary effect is provided in that a technology can be provided that improves the efficiency of the creation of training data when machine learning events in a data series is performed. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram illustrating a configuration of an information processing device according to the present disclosure. [Figure 2] FIG. 1 is a flow diagram showing the flow of an information processing method according to the present disclosure. [Figure 3] 1 is a block diagram illustrating a configuration of an information processing device according to the present disclosure. [Figure 4] FIG. 1 is a flow diagram showing the flow of an information processing method according to the present disclosure. [Figure 5] FIG. 10 is a flowchart showing a detailed flow of a division process according to the present disclosure. [Figure 6] FIG. 10 is a flowchart showing a detailed flow of a division process according to the present disclosure. [Figure 7] FIG. 10 is a diagram schematically illustrating an example of a screen according to the present disclosure. [Figure 8] FIG. 2 is a block diagram showing the hardware configuration of a computer that functions as each device according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technologies (part or all of the products or methods) employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technologies employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, the effects mentioned in the exemplary embodiments shown below are examples of effects expected in the exemplary embodiments, and do not define the scope of the present invention. In other words, embodiments that do not exhibit the effects mentioned in the exemplary embodiments shown below may also be included in the scope of the present invention.

[0013] First Exemplary Embodiment A first exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is a basic form for each of the exemplary embodiments described below. Note that the scope of application of each technique employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technique employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technique shown in the drawings referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0014] (Overview of information processing device 1) The information processing device 1 is a device that selects data to which a correct label should be assigned from a plurality of data sequences in order to generate training data used in training a machine learning model that makes predictions on data sequences.

[0015] A "data series" is a sequence of data arranged in order. For example, a video, which is a sequence of frames arranged in time order, is an example of a data series. A machine learning model that makes predictions on videos is a behavior prediction model that predicts human behavior in the video. As an example, the behavior prediction model takes a video as input and outputs the class and interval of the behavior of people included as subjects in the video. The information processing device 1 can generate, for example, multiple videos in which behavior labels are assigned to only some of the frames as training data used to train such a behavior prediction model.

[0016] Another example of a data sequence is a sequence of audio data arranged in chronological order. An example of a machine learning model that makes predictions on audio data sequences is an audio event prediction model that predicts audio events in an audio data sequence. As an example, the audio event prediction model receives an audio data sequence as input and outputs the classes and intervals of audio events contained in the audio data sequence. The information processing device 1 can generate, for example, multiple audio data sequences in which audio event labels are assigned to only some of the audio data as training data used for training such an audio event prediction model.

[0017] (Configuration of information processing device 1) The configuration of the information processing device 1 will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the information processing device 1. As shown in Fig. 1, the information processing device 1 includes a division unit 11, a clustering unit 12, and a selection unit 13. The division unit 11, the clustering unit 12, and the selection unit 13 are an example of a configuration that realizes the division means, the clustering means, and the selection means. For example, when the information processing device 1 is configured by a computer including at least one processor, the division unit 11, the clustering unit 12, and the selection unit 13 are realized by the at least one processor executing a program.

[0018] The dividing unit 11 divides a set of features constituting a feature sequence corresponding to each of a plurality of data sequences into a plurality of partial feature sets. The dividing process is performed so that all of the features included in the plurality of partial feature sets cover the features constituting the original feature sequence without excess or deficiency. Here, the feature sequence indicates the characteristics of the corresponding data sequence. As an example, the feature sequence may be a sequence in which the features of each data constituting the corresponding data sequence are arranged in the order of the original data in the data sequence. In this case, the data constituting the data sequence and the features constituting the feature sequence correspond one-to-one. As another example, the feature sequence may be a sequence of features obtained by convolution processing of the corresponding data sequence. In this case, the number of data constituting the feature sequence may be smaller than the number of data constituting the data sequence.

[0019] For example, the dividing unit 11 may divide a set of features constituting a feature sequence into a plurality of partial feature sets while maintaining the order of the features in the feature sequence. For example, a feature sequence in which a first feature, a second feature, a third feature, a fourth feature, and a fifth feature are arranged in this order will be described as an example. In this example, the dividing unit 11 may divide the set of first to fifth features in the feature sequence into a first partial feature set including the first and second feature amounts and a second partial feature set including the third to fifth feature amounts while maintaining the order.

[0020] Alternatively, the dividing unit 11 may divide a set of features constituting a feature sequence into a plurality of partial feature sets without necessarily maintaining the order of the features in the feature sequence. For example, in the above example, the dividing unit 11 may divide the set of first to fifth features constituting the feature sequence into a first partial feature set including the first, third, and fourth feature amounts and a second partial feature set including the second and fifth feature amounts, without necessarily maintaining this order.

[0021] As another example of the division process by the division unit 11, the division unit 11 may divide a set of features constituting the feature sequence randomly, or may divide the feature sequence at equal intervals.

[0022] The clustering unit 12 clusters a set of partial feature sets obtained from a plurality of data series into a plurality of clusters. The "set of partial feature sets obtained from a plurality of feature series" preferably includes all of the plurality of partial feature sets obtained by dividing each of the plurality of feature series. For example, suppose a first partial feature set and a second partial feature set are obtained from a first feature series, and a third partial feature set, a fourth partial feature set, and a fifth partial feature set are obtained from a second data series. In this case, the "set of partial feature sets obtained from a plurality of data series" includes the first to fifth partial feature sets, and the clustering unit 12 performs clustering on the first to fifth partial feature sets. However, the "set of partial feature sets" may also be a set of partial feature sets that satisfy some condition from among all of the plurality of partial feature sets.

[0023] The selection unit 13 selects data to be assigned a correct label from a set of data constituting at least a part of any of a plurality of data series corresponding to at least one partial feature set included in each cluster. For example, the selection unit 13 may select any partial feature set from the partial feature sets included in a certain cluster randomly or based on predetermined conditions. The number of partial feature sets to be selected may be one or more for each cluster. Furthermore, the number of partial feature sets to be selected may be the same for all clusters, or the number selected in at least one cluster may be different from the number selected in at least one other cluster.

[0024] The selection unit 13 also identifies a set of data constituting at least a part of any of the multiple data series corresponding to the selected partial feature set. For example, if there is a one-to-one correspondence between the data constituting the data series and the features constituting the feature series, a set of data corresponding to each feature included in the selected partial feature set is identified. If there is not a one-to-one correspondence between the data constituting the data series and the features constituting the feature series, a corresponding set of data is identified by performing the reverse of the process of converting the data series into a feature series based on each feature included in the selected partial feature set. As an example, if a feature series is obtained by a convolution process on the data series, a corresponding set of data is identified by performing the reverse of the convolution process from each feature included in the selected partial feature set.

[0025] Furthermore, the selection unit 13 may select data, for example, randomly or based on predetermined conditions, from a set of data corresponding to a selected partial feature set as data to be assigned a correct label. The number of data to be selected may be one or more for each selected partial feature set. Furthermore, the number of data to be selected may be the same for all selected partial feature sets, or the number of data selected in at least one partial feature set may be different from the number of data selected in at least one other partial feature set.

[0026] (Effects of information processing devices) As described above, the information processing device 1 employs a configuration including the above-described dividing unit 11, clustering unit 12, and selection unit 13. In this way, the computational cost of clustering a set of partial feature sets obtained from multiple data series is reduced compared to clustering all features obtained from the multiple data series. Furthermore, it can be said that the data sets corresponding to the partial feature sets included in each cluster are classified into the cluster based on the characteristics of each data. Therefore, data selected from the data sets corresponding to any of the partial feature sets included in each cluster is expected to be representative of data having similar characteristics across the multiple data series. Such multiple data series, in which only some of the data are labeled as correct, are useful as training data for accurate machine learning of events in the data series. As a result, the information processing device 1 achieves the effect of streamlining the operation of creating training data when performing machine learning of events in a data series.

[0027] (Flow of information processing method S1) Information processing method S1 is a method executed by at least one processor. For example, when the above-described information processing device 1 is configured by a computer including at least one processor, the information processing device 1 executes information processing method S1. The flow of information processing method S1 will be described with reference to FIG. 2. FIG. 2 is a flow diagram showing the flow of information processing method S1. As shown in FIG. 2, information processing method S1 includes a division process S11, a clustering process S12, and a selection process S13.

[0028] In the division process S11, at least one processor (for example, the division unit 11) divides a set of features constituting a feature sequence corresponding to each of a plurality of data sequences into a plurality of partial feature sets.

[0029] In the clustering process S12, at least one processor (for example, the clustering unit 12) clusters the set of partial feature sets obtained from the plurality of data sequences into a plurality of clusters.

[0030] In the selection process S13, at least one processor (e.g., the selection unit 13) selects data to be assigned a correct label from a set of data that constitutes at least a part of any of a plurality of data series, corresponding to at least one partial feature set included in each cluster.

[0031] The details of the division process S11, clustering process S12, and selection process S13 are the same as those described for the division unit 11, clustering unit 12, and selection unit 13, and therefore will not be described in detail again.

[0032] (Effect of information processing method S1) As described above, the information processing method S1 employs a configuration including the above-described division process S11, clustering process S12, and selection process S13. Therefore, the information processing method S1 can obtain the same effects as the information processing device 1.

[0033] Second Exemplary Embodiment A second exemplary embodiment, which is one example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be assigned the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technology employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs. Furthermore, each technology shown in each drawing referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs.

[0034] (Configuration of information processing device 1A) The information processing device 1A can be used, for example, to generate training data used in weak label learning. Weak label learning is machine learning performed using training data in which correct labels are assigned to only some of the data. However, training data generated using the information processing device 1A is not limited to being used in weak label learning. The configuration of the information processing device 1A will be described with reference to FIG. 3. FIG. 3 is a block diagram showing the configuration of the information processing device 1A. The information processing device 1A includes a control unit 110 and a storage unit 120. The control unit 110 controls each unit of the information processing device 1A in an integrated manner. When the information processing device 1A is configured using a computer, the control unit 110 is realized, for example, by at least one processor included in the computer executing a program.

[0035] The storage unit 120 stores a program for causing the control unit 110 to function and various data used by the control unit 110. When the information processing device 1A is configured by a computer, the storage unit 120 is configured by a memory provided in the computer, for example.

[0036] The storage unit 120 stores multiple data series, multiple pieces of label information, and multiple feature series. The multiple data series are targets to which correct labels are assigned for use as training data, and are referenced in an information processing method S1A described later. The multiple feature series are temporary information generated during execution of the information processing method S1A described later. The multiple label information is information indicating some data in the multiple data series and correct labels to be assigned to the data, and is generated by the information processing method S1A described later. In other words, the multiple data series and the multiple label information are examples of "training data used in weak label learning." Note that some or all of the multiple data series do not necessarily need to be stored in the storage unit 120 of the device itself, and may be read, for example, from another device, storage medium, etc. connected via a network.

[0037] For example, an example will be described in which the data series is a moving image. In this case, the plurality of data series and the plurality of label information stored in the storage unit 120 are expressed by the following equation (1).

[0038]

number

[0039]

number

[0040] The control unit 110 includes a feature extraction unit 14 and a UI (User Interface) unit 15 in addition to the division unit 11, clustering unit 12, and selection unit 13 included in the information processing device 1. The UI unit 15 is an example of a configuration that realizes a user interface means.

[0041] The feature extraction unit 14 extracts features from each of a plurality of data series to generate a feature series. A feature series is a sequence of features that indicate the characteristics of a data series. For example, the feature extraction unit 14 extracts features from a frame series shown in equation (2) corresponding to a moving image V_n in equation (1) to generate a feature series shown in the following equation (3).

[0042]

number

[0043] By obtaining equation (3) for each of the N moving images V_n, a series of N feature amounts shown in the following equation (4) is obtained.

[0044]

number

[0045] The division unit 11 is configured similarly to the division unit 11 included in the information processing device 1, and is also configured as follows: The division unit 11 may perform division processing using a first division method described below, or may perform division processing using a second division method.

[0046] In the first division method, the division unit 11 divides each of the plurality of feature sequences at minimum points where the similarity between adjacent feature sequences is minimum, and obtains each of the plurality of partial feature sequences obtained by the division as a partial feature set. A minimum point is a point where the similarity between feature sequences included in a predetermined width is minimum. For example, the division unit 11 may sequentially detect minimum points while shifting the predetermined width, starting from the beginning of the feature sequence. As the similarity, negative Euclidean distance, cosine similarity, etc. may be used.

[0047] The dividing unit 11 may detect minimum points after smoothing the feature sequence. Examples of smoothing include, but are not limited to, processing using a moving average and a window function. Smoothing is not necessarily required, but smoothing can suppress detection noise in minimum points. The width of the window used in smoothing is desirably set shorter than the duration of the correct label to be detected by the prediction model being trained (for example, if the prediction model is the above-mentioned behavior prediction model, the typical duration of behavior). The partial feature set divided by the first division method includes features that are consecutively arranged in the original feature sequence.

[0048] In the second division method, the division unit 11 divides a set of features constituting a feature sequence into a plurality of partial feature sets by clustering. The features included in the partial feature sets divided by the second division method are not necessarily arranged consecutively in the original feature sequence, and may include discontinuous features.

[0049] Regardless of whether the first division method or the second division method is used, the set of partial feature amounts obtained by division is expressed by the following formula (5).

[0050]

number

[0051] By obtaining a plurality of partial feature sets shown in formula (5) for each of the N feature sequences shown in formula (4), a set of partial feature sets shown in formula (6) is obtained. The set of partial feature sets shown in formula (6) includes partial feature sets obtained from all of the multiple videos V_n.

[0052]

number

[0053] The clustering unit 12 has the same configuration as the clustering unit 12 included in the information processing device 1, and in addition has the following configuration: The clustering unit 12 includes an aggregate feature calculation unit 121 and an aggregate feature clustering unit 122.

[0054] The aggregate feature calculation unit 121 calculates an aggregate feature representing each partial feature set in the set of partial feature sets. Here, the aggregate feature may be a feature obtained by aggregating features included in the partial feature set. "Aggregating" refers to converting one or more features into a single feature. Hereinafter, the aggregate feature will also be referred to as an "aggregated feature." As an example, the aggregate feature calculation unit 121 may calculate the average of each feature included in a certain partial feature set (e.g., z_n_σ_nij, j = 1, 2, ..., J_ni shown in Equation (5)) as the aggregate feature. As another example, the aggregate feature calculation unit 121 may calculate the aggregate feature by performing max pooling for each dimension of each feature included in a certain partial feature set. When features are expressed as vectors, the aggregate feature is also expressed as a vector.

[0055] By calculating the aggregate feature for each partial feature set included in the set of partial feature sets shown in equation (6), the set of aggregate features shown in the following equation (7) is obtained.

[0056]

number

[0057] Furthermore, the aggregate feature clustering unit 122 clusters the set of partial feature sets into multiple clusters by clustering multiple aggregate features. As an example, the aggregate feature clustering unit 122 performs clustering based on a known method such as K-Means or BIRSH. For example, it is desirable to set the number of clusters G (hereinafter also referred to as the number of clusters G) so that the execution time of clustering falls within an allowable range.

[0058] As an example, by performing clustering on the set of partial aggregated features shown in equation (7), the following equation (8) is obtained as the clustering result.

[0059]

number

[0060] The selection unit 13 has the same configuration as the selection unit 13 included in the information processing device 1, and in addition, has the following configuration: The selection unit 13 includes a representative aggregate feature selection unit 131 and a data selection unit 132.

[0061] The representative aggregate feature selection unit 131 selects, based on the aggregate feature, one of the partial feature sets included in each cluster as a partial feature set corresponding to a data set from which data to be assigned a correct label should be selected. Furthermore, the representative aggregate feature selection unit 131 may select, based on the number of clusters and the number of data to be assigned a correct label, one of the partial feature sets included in each cluster as a partial feature set corresponding to a data set from which data to be assigned a correct label should be selected.

[0062] Here, "a set of partial features included in each cluster" refers to "a set of partial features that was the basis for calculating the aggregate features included in each cluster obtained by clustering the aggregate features." In other words, the representative aggregate feature selection unit 131 selects one of the aggregate features included in each cluster, thereby selecting one of the corresponding partial feature sets. Hereinafter, the aggregate feature selected by the representative aggregate feature selection unit 131 will also be referred to as the representative aggregate feature.

[0063] As mentioned above, it is desirable that the number of clusters G is set so that the execution time of clustering falls within an allowable range. It is also desirable that the number K of data to which correct labels should be assigned is set based on the man-hours of the user who assigns the correct labels. Hereinafter, "data to which correct labels should be assigned" will also be referred to as "data to be assigned." The number K of data to be assigned will also be referred to as the number K of data to be assigned. The number K of data to be assigned is expressed, for example, by the following equation (9).

[0064]

number

[0065] For example, the representative aggregate feature selection unit 131 may select K / G representative aggregate features from each cluster. Here, "K / G" represents a natural number obtained by dividing K by G and rounding off any fractions. For example, the representative aggregate feature selection unit 131 may randomly select K / G representative aggregate features from each cluster. Furthermore, when the number of clusters G and the number of data items K to be annotated are equal, K / G=1. Therefore, the representative aggregate feature selection unit 131 selects one representative aggregate feature from each cluster. In this case, the representative aggregate feature selection unit 131 may select the aggregate feature closest to the center of the feature space in each cluster as the representative aggregate feature. Furthermore, when the number of clusters G is smaller than the number of data items K to be annotated, the representative aggregate feature selection unit 131 may cluster the aggregate features belonging to each cluster into K / G partial clusters and select the aggregate feature closest to the center of the feature space in each partial cluster as the representative aggregate feature.

[0066] The data selection unit 132 identifies a set of at least a portion of data in any of the multiple data series corresponding to the representative aggregate feature. Furthermore, the data selection unit 132 selects data to be assigned a correct label from the identified data set. For example, the data selection unit 132 identifies a partial feature set from which the representative aggregate feature is calculated. Furthermore, the data selection unit 132 identifies a set of data that constitutes at least a portion of any of the multiple data series corresponding to the identified partial feature set. A specific example of identifying a data set corresponding to a partial feature set is as described in the first exemplary embodiment, and therefore, the details will not be repeated. If the identified data set is consecutive in the original data series (i.e., a partial data series), the data selection unit 132 may select any data based on the order of the data. For example, the data selection unit 132 may select data in the center of the partial data series. Furthermore, if the identified data set is not consecutive in the original data series, the data selection unit 132 may randomly select data from the data set. The data selected by the data selection unit 132 is expressed, for example, by the following equation (10).

[0067]

number

[0068] The UI unit 15 accepts, from the user, input of a correct label for the data selected by the data selection unit 132. For example, when the data series is a video, the UI unit 15 may display, on the display device, a screen including information indicating the frame selected by the data selection unit 132 and a UI object for accepting input of a correct label for the data. The UI unit 15 also generates label information for each of the multiple data series.

[0069] For example, each of the plurality of pieces of label information includes a set of data identification information and a correct label. The label information does not have to be stored in the storage unit 120 before executing an information processing method S1A described later, but is stored in the storage unit 120 by executing the information processing method S1A. However, some or all of the plurality of pieces of label information do not necessarily have to be stored in the storage unit 120 of the device itself, and may be stored in, for example, another device connected via a network, a portable storage medium, or the like.

[0070] As an example, N pieces of label information generated for N moving images V_n are expressed by the following equation (11).

[0071]

number

[0072]

number

[0073] (Information processing method S1A) The information processing device 1A configured as above executes an information processing method S1A. The flow of the information processing method S1A will be described with reference to Fig. 4. Fig. 4 is a flow diagram showing the flow of the information processing method S1A. As shown in Fig. 4, the information processing method S1A includes steps S101 to S114. At the start of the information processing method S1A, it is assumed that a plurality of data series (for example, N moving images V_n shown in equation (1)) are stored in the storage unit 120, but the feature series and label information have not yet been stored.

[0074] In step S101, control unit 110 determines whether or not there is an unprocessed data series. An "unprocessed data series" is a data series, among the multiple data series stored in storage unit 120, for which the series of processes in steps S102 to S107 has not been completed. If there is no unprocessed data series (No in step S101), step S108, which will be described later, is executed. If there is an unprocessed data series (Yes in step S101), the next step S102 is executed.

[0075] In step S102, the control unit 110 selects one of the unprocessed data series as a target data series. A series of processes from steps S103 to S107 is executed for the target data series. The target data series is, for example, the n-th moving image V_n among N moving images V_n.

[0076] In step S103, the feature extraction unit 14 generates a feature sequence by extracting features from the target data sequence. As a result, for example, a feature sequence shown in Equation (3) corresponding to the target moving image V_n is generated.

[0077] In step S104, the division unit 11 divides the set of features constituting the feature sequence into a plurality of partial feature sets, thereby generating I_n partial feature sets shown in equation (5) corresponding to the target moving image V_n.

[0078] For example, in step S104, the dividing unit may execute either the dividing process S104a shown in FIG. 5 or the dividing process S104b shown in FIG.

[0079] Fig. 5 is a flow diagram showing the detailed flow of the division process S104a. As shown in Fig. 5, the division process S104a includes steps S201 to S203.

[0080] In step S201, the dividing unit 11 smooths the feature sequence. In step S202, the dividing unit 11 calculates the similarity between adjacent feature sequences in the smoothed feature sequence. In step S203, the dividing unit 11 divides the feature sequence at points of minimum similarity, and defines each of the multiple partial feature sequences obtained by the division as a partial feature set. Note that in step S203, the partial feature set may include feature sequences after smoothing, or may include feature sequences before smoothing. This completes the dividing process S104a.

[0081] FIG. 6 is a flow diagram showing a detailed flow of the division process S104b. As shown in FIG. 6, the division process S104b includes step S301. In step S301, the division unit 11 divides a set of features constituting a feature sequence into a plurality of partial feature sets by clustering the set of features. Note that the clustering performed by the division unit 11 in the division process S104b is performed on a feature sequence basis, and differs from the clustering performed by the clustering unit 12, which targets all of the partial feature sets obtained across a plurality of feature sequences. This completes the division process S104b.

[0082] After the division process S104a or S104b is executed in step S104, step S105 in FIG. 4 is executed.

[0083] In step S105, the control unit 110 determines whether or not there is an unprocessed partial feature set. An "unprocessed partial feature set" is a partial feature set, among the multiple partial feature sets into which a feature series corresponding to a target data series is divided, for which the series of processes in steps S106 to S107 has not been completed. If there is no unprocessed partial feature set (No in step S105), the series of processes for the target data series ends, and the processes from step S101 onwards are repeated. If there is an unprocessed partial feature set (Yes in step S105), the next step S106 is executed.

[0084] In step S106, the control unit 110 selects one of the unprocessed partial feature sets as a target partial feature set. The processing in the next step S107 is executed for the target partial feature set. The target partial feature set is, for example, the i-th partial feature set among the I_n partial feature sets corresponding to the target moving image V_n.

[0085] In step S107, the aggregate feature calculation unit 121 calculates an aggregate feature corresponding to the target partial feature set. This completes the processing for the target partial feature set. Subsequently, the processing from step S105 is repeated.

[0086] In this way, when it is determined in step S101 that there are no unprocessed data series, for example, the set of feature series shown in equation (4), the set of partial feature sets shown in equation (6), and the set of aggregated features shown in equation (7) are obtained across all N moving images V_n. In this case, the next step S108 is executed.

[0087] In step S108, the aggregate feature clustering unit 122 clusters a plurality of aggregate features. For example, clustering is performed on the set of aggregate features shown in equation (7). As a result, for example, a clustering result shown in equation (8) is obtained.

[0088] In step S109, the representative aggregate feature selection unit 131 selects a representative aggregate feature from each cluster. For example, as described above, K / G representative aggregate features may be selected for each cluster.

[0089] In step S110, the control unit 110 determines whether or not there is an unprocessed representative aggregate feature. An "unprocessed representative aggregate feature" is a representative aggregate feature, among the representative aggregate features selected from each cluster, for which the series of processes in steps S111 to S112 has not been completed. If there is no unprocessed representative aggregate feature (No in step S110), step S113, which will be described later, is executed. If there is an unprocessed representative aggregate feature (Yes in step S110), the next step S111 is executed.

[0090] In step S111, the control unit 110 selects one of the unprocessed representative aggregate features as a target representative aggregate feature. The process in the next step S112 is executed for the target representative aggregate feature.

[0091] In step S112, the data selection unit 132 identifies a partial feature set from which the target representative aggregate feature is calculated, and identifies a set of at least a portion of data in any of the multiple data series that corresponds to the identified partial feature set. For example, the data set can be identified by tracing back through equations (7), (6), (4), and (2).

[0092] In step S112, the data selection unit 132 selects target data to be labeled with a correct label from the identified data set. This completes the series of processes for the target representative aggregate feature. Then, the processes from step S110 are repeated.

[0093] In this way, when it is determined in step S110 that there are no unprocessed representative features, the data to be added, for example, selected across all N moving images V_n and shown in equation (10), is obtained. In this case, the next step S113 is executed.

[0094] In step S113, the UI unit 15 accepts input of a correct label for the data to be assigned. Fig. 7 is a diagram schematically showing an example of a screen displayed on a display device (not shown) in step S113.

[0095] As shown in FIG. 7, screen example G100 is an input screen for assigning label information to multiple videos V1, V2, and V3. Screen example G100 includes seek bars B1, B2, and B3 corresponding to the lengths of each video, V_1, V_2, V_3, and so on, and figures T11 to T14, T21 to T25, and T31 to T33 indicating the positions of frames to which labels are to be assigned. For example, in screen example G100, video V_1 has been selected by the user, and seek bar B1 is displayed more highlighted than seek bars B2 and B3 for the other videos V_2 and V_3. Figures T11 to T14 on seek bar B1 indicate the positions of the four frames to which labels are to be assigned selected for video V_1. Images G11 to G14 show thumbnails of the frames to which labels are to be assigned indicated by figures T11 to T14. Furthermore, UI objects L11 to L14 are UI objects for inputting correct labels to be assigned to the target frames indicated by figures T11 to T14. UI objects L11 to L14 may be, for example, objects that accept an operation to select one of C predetermined behavior classes. Correct labels of "BasebalPitch," "BasebalPitch," and "BG" have already been input to UI objects L11 to L13, respectively. Furthermore, figure T14 has been selected by the user for inputting the correct label. For this reason, image G14 corresponding to figure T14 is displayed larger and more highlighted than the other images G11 to G13.

[0096] Such screen example G100 is displayed by the information processing device 1A performing clustering on a set of partial feature sets obtained across all N moving images V_n. This has the effect of reducing the time it takes from when the moving images V_n are provided to the information processing device 1A until the screen example G100 is displayed, compared to when clustering is performed on sets of all feature amounts included in the feature sequence corresponding to each of the moving images V_n.

[0097] (Variation) In the above-described exemplary embodiment 2, the case where the data series is a video has been mainly described, but the present invention is also applicable to the case where training data used in weak-label learning is generated from a data series other than a video (for example, an audio data series). Specific examples of the division process, clustering process, and selection process shown in each exemplary embodiment are not limited to the above examples. Furthermore, the information processing device 1, 1A shown in each exemplary embodiment is not limited to being physically configured by one computer, but may be configured by multiple computers (for example, a server and a terminal).

[0098] (Advantages of this exemplary embodiment) As described above, this exemplary embodiment employs a configuration further including a user interface unit 15 that accepts, from the user, an input of a correct label for the data selected by the selection unit 13. This configuration can assist the user in creating training data to be used in weak-label learning.

[0099] In addition, in this exemplary embodiment, the aggregate feature calculation unit 121 calculates an aggregate feature representing each partial feature set in the set of partial feature sets, and the aggregate feature clustering unit 122 clusters the set of partial feature sets into multiple clusters by clustering the multiple aggregate features. With this configuration, the set of partial feature sets can be clustered with high accuracy.

[0100] Furthermore, in this exemplary embodiment, the representative aggregate feature selection unit 131 is configured to select, based on the aggregate feature, one of the partial feature sets included in each cluster as a partial feature set corresponding to a set of data from which data to be assigned a correct label should be selected. This configuration makes it possible to more accurately select partial feature sets included in multiple clusters.

[0101] In this exemplary embodiment, the dividing unit 11 is configured to divide a set of features constituting a feature sequence into a plurality of partial feature sets by clustering. This configuration makes it possible to divide the feature sequence with higher accuracy without being restricted by the arrangement in the original feature sequence.

[0102] Furthermore, in this exemplary embodiment, the representative aggregate feature selection unit 131 is configured to select one of the partial feature sets included in each cluster as a partial feature set corresponding to a set of data from which data to be labeled with a correct answer label should be selected, based on the number of clusters and the number of data to be labeled with a correct answer label. This configuration makes it possible to select a more appropriate number of data to be labeled, taking into consideration both the computational cost required for clustering the partial feature sets and the user's effort in assigning correct labels to the data to be labeled.

[0103] [Software implementation example] Some or all of the functions of the information processing devices 1, 1A (hereinafter also referred to as "each of the above devices") may be realized by hardware such as an integrated circuit (IC chip), or by software.

[0104] In the latter case, each of the above devices is realized by, for example, a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 8. Figure 8 is a block diagram showing the hardware configuration of computer C that functions as each of the above devices.

[0105] The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for causing the computer C to operate as each of the above-mentioned devices. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing the functions of each of the above-mentioned devices.

[0106] The processor C1 may be, for example, a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0107] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, mouse, display, and printer.

[0108] Furthermore, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0109] Furthermore, the functions of each of the devices may be realized by a single processor provided in a single computer, by multiple processors provided in a single computer working in cooperation, or by multiple processors provided in each of multiple computers working in cooperation. Furthermore, the programs for causing each of the devices to realize the functions may be stored in a single memory provided in a single computer, or may be distributed and stored in multiple memories provided in a single computer, or may be distributed and stored in multiple memories provided in each of multiple computers.

[0110] [Appendix 1] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0111] (Appendix 1) a dividing means for dividing a set of features constituting a feature sequence corresponding to each of a plurality of data sequences into a plurality of partial feature sets; a clustering means for clustering the set of partial feature quantities obtained from the plurality of data series into a plurality of clusters; a selection means for selecting data to be assigned a correct label from a set of data constituting at least a part of any of the plurality of data series, the data set corresponding to at least one of the partial feature sets included in each cluster; An information processing device comprising:

[0112] (Appendix 2) a user interface means for receiving from a user an input of a correct label for the data selected by the selection means; 2. The information processing device according to claim 1, further comprising:

[0113] (Appendix 3) the clustering means calculates, for each partial feature set in the set of partial feature sets, a collective feature representing the partial feature set, and clusters the plurality of collective features, thereby clustering the set of partial feature sets into the plurality of clusters. 3. The information processing device according to claim 1 or 2.

[0114] (Appendix 4) the selecting means selects, based on the set feature, one of the partial feature sets included in each cluster as the partial feature set corresponding to a set of data from which data to be assigned a correct label should be selected. 4. The information processing device according to claim 3.

[0115] (Appendix 5) the dividing means divides a set of features constituting the feature sequence into the plurality of partial feature sets by clustering the set of features; 5. An information processing device according to any one of appendices 1 to 4.

[0116] (Appendix 6) the selecting means selects, as the partial feature set corresponding to a set of data from which data to be assigned a correct label should be selected, one of the partial feature sets included in each cluster based on the number of the plurality of clusters and the number of data to be assigned a correct label. 6. An information processing device according to any one of appendices 1 to 5.

[0117] (Appendix 7) a division process in which at least one processor divides a set of features constituting a feature sequence corresponding to each of a plurality of data sequences into a plurality of partial feature sets; a clustering process in which the at least one processor clusters the set of partial feature sets obtained from the plurality of data series into a plurality of clusters; a selection process by the at least one processor to select data to be assigned a correct label from a set of data constituting at least a part of any of the plurality of data series, the data set corresponding to at least one of the partial feature sets included in each cluster; An information processing method comprising:

[0118] (Appendix 8) At least one processor has a division process for dividing a set of features constituting a feature sequence corresponding to each of a plurality of data sequences into a plurality of partial feature sets; a clustering process for clustering the set of partial feature quantities obtained from the plurality of data series into a plurality of clusters; a selection process of selecting data to be assigned a correct label from a set of data constituting at least a part of any of the plurality of data series, the data set corresponding to at least one of the partial feature sets included in each cluster; A program that executes.

[0119] (Appendix 9) at least one processor, a division process for dividing a set of features constituting a feature sequence corresponding to each of a plurality of data sequences into a plurality of partial feature sets; a clustering process for clustering the set of partial feature quantities obtained from the plurality of data series into a plurality of clusters; a selection process of selecting data to be assigned a correct label from a set of data constituting at least a part of any of the plurality of data series, the data set corresponding to at least one of the partial feature sets included in each cluster; An information processing device that executes the above.

[0120] The information processing device may further include a memory, and the memory may store a program for causing the at least one processor to execute each of the processes. [Explanation of symbols]

[0121] 1, 1A Information processing equipment 11 Division 12 Clustering Department 13 Selection section 14 Feature extraction unit 15 User Interface Section 110 control section 120 Storage section 121 Aggregate feature calculation unit 122 Aggregate Feature Clustering Unit 131 Representative aggregate feature selection unit 132 Data selection section

Claims

1. a dividing means for dividing a set of features constituting a feature sequence corresponding to each of a plurality of data sequences into a plurality of partial feature sets; a clustering means for clustering the set of partial feature quantities obtained from the plurality of data series into a plurality of clusters; a selection means for selecting data to be assigned a correct label from a set of data constituting at least a part of any of the plurality of data series, the data set corresponding to at least one of the partial feature sets included in each cluster; An information processing device comprising:

2. a user interface means for receiving from a user an input of a correct label for the data selected by the selection means; The information processing device according to claim 1 , further comprising:

3. the clustering means calculates, for each partial feature set in the set of partial feature sets, a collective feature representing the partial feature set, and clusters the plurality of collective features, thereby clustering the set of partial feature sets into the plurality of clusters.

3. The information processing device according to claim 1.

4. the selecting means selects, based on the set feature, one of the partial feature sets included in each cluster as the partial feature set corresponding to a set of data from which data to be assigned a correct label should be selected. The information processing device according to claim 3 .

5. the dividing means divides a set of features constituting the feature sequence into the plurality of partial feature sets by clustering the set of features; 3. The information processing device according to claim 1.

6. the selecting means selects, as the partial feature set corresponding to a set of data from which data to be assigned a correct label should be selected, one of the partial feature sets included in each cluster based on the number of the plurality of clusters and the number of data to be assigned a correct label.

3. The information processing device according to claim 1.

7. a division process in which at least one processor divides a set of features constituting a feature sequence corresponding to each of a plurality of data sequences into a plurality of partial feature sets; a clustering process in which the at least one processor clusters the set of partial feature sets obtained from the plurality of data series into a plurality of clusters; a selection process in which the at least one processor selects data to be assigned a correct label from a set of data constituting at least a part of any of the plurality of data series, the data set corresponding to at least one of the partial feature sets included in each cluster; An information processing method comprising:

8. At least one processor a division process for dividing a set of features constituting a feature sequence corresponding to each of a plurality of data sequences into a plurality of partial feature sets; a clustering process for clustering the set of partial feature quantities obtained from the plurality of data series into a plurality of clusters; a selection process of selecting data to be assigned a correct label from a set of data constituting at least a part of any of the plurality of data series, the data set corresponding to at least one of the partial feature sets included in each cluster; A program that executes.

Images