Information processing device, information processing method, and program

Abstract

To solve the problem of making it possible to provide a technology that can assign pseudo-labels independently of absence / presence of class-labeled data, an information processing device 1 comprises an inference means (11) that infers a class pertaining to each piece of partial data constituting time series data, a calculating means (12) that calculates a level of agreement among results which are obtained from the inference by the inference means and each of which relates to each of a plurality of pieces of partial data included in a temporally continuous section, and a pseudo-label assignment means (13) that assigns a pseudo-label based on the inference results in the section to at least one of the plurality of pieces of partial data in the section according to the level of agreement.

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program capable of assigning pseudo labels regardless of the presence or absence of class-labeled data. [Background technology]

[0002] When training an AI model using training data, the cost of collecting the data is a burden. Conventionally, in order to reduce the cost of collecting training data, a weak labeling method has been used in which only a portion of the data is labeled as training data.

[0003] For example, when performing behavior recognition on video data consisting of successive frames, there is a need to reduce the cost of training an AI model by labeling only a subset of the time-series data.

[0004] A method has also been proposed in which pseudo-labels are assigned to unlabeled data and an AI model is trained on them (see, for example, Non-Patent Document 1). According to Non-Patent Document 1, the class of data in the vicinity of data to which a class label has been assigned is predicted, and a pseudo-label is assigned based on the prediction result. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] SF-Net: Single-Frame Supervision for Temporal Action Localization aXiv:2003.06845 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the technology of Non-Patent Document 1, the target data to which pseudo-labels are assigned is limited to the vicinity of the time with class labels, so the improvement of inference accuracy is limited. For example, for time series data with a long duration, pseudo-labels can only be assigned to a part of the time series data. In addition, there is a problem that data near the time to which no class labels are assigned cannot be used for learning.

[0007] One aspect of the present invention has been made in consideration of the above problems, and one example of a purpose of the present invention is to provide a technology that can assign pseudo labels regardless of whether class-labeled data is available or not. [Means for solving the problem]

[0008] An information processing device according to one aspect of the present invention includes an inference means for inferring a class for each partial data constituting time series data, a calculation means for calculating a degree of similarity between inference results by the inference means for each of a plurality of partial data included in a temporally consecutive interval, and a pseudo label assignment means for assigning a pseudo label based on the inference result for the interval to at least any of the plurality of partial data in the interval in accordance with the degree of similarity.

[0009] An information processing method according to one aspect of the present invention includes inferring a class for each partial data constituting time series data, calculating a degree of consistency between inference results for each of a plurality of partial data included in a temporally consecutive interval, and assigning a pseudo label based on the inference result for the interval to at least one of the plurality of partial data in the interval in accordance with the degree of consistency.

[0010] A program according to one aspect of the present invention causes a computer to function as an information processing device having an inference means for inferring a class for each partial data constituting time series data, a calculation means for calculating a degree of similarity between inference results by the inference means for each of a plurality of partial data included in a temporally consecutive interval, and a pseudo label assignment means for assigning a pseudo label based on the inference result for the interval to at least one of the plurality of partial data in the interval in accordance with the degree of similarity. Effect of the Invention

[0011] According to one aspect of the present invention, it is possible to provide a technique that can assign pseudo labels regardless of the presence or absence of class-labeled data. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram showing an example of the configuration of an information processing apparatus according to a first exemplary embodiment of the present invention. [Diagram 2] 2 is a diagram illustrating the functions of each unit of the information processing device in FIG. 1. [Diagram 3] 1 is a flowchart showing a flow of an information processing method according to a first exemplary embodiment of the present invention. [Figure 4] FIG. 11 is a block diagram showing an example of the configuration of an information processing device according to an exemplary embodiment 2 of the present invention. [Diagram 5] 11 is a flowchart showing the flow of an information processing method according to a second exemplary embodiment of the present invention. [Figure 6] 13 is a diagram illustrating the variation in the highest confidence (maximum confidence) among the inference results of partial data. FIG. [Figure 7] 11 is a flowchart showing the flow of an information processing method according to an exemplary embodiment 3 of the present invention. [Figure 8] FIG. 1 is a diagram illustrating an example of the configuration of a computer that executes instructions of a program, which is software that realizes each function. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Example 1 A first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described later.

[0014] <Overview of information processing device 1> The information processing device 1 according to this exemplary embodiment is, generally speaking, a device that assigns pseudo labels to partial data of time-series data.

[0015] As an example, the information processing device 1 includes: An inference means for inferring a class for each piece of partial data constituting the time series data; a calculation means for calculating a degree of coincidence between inference results by the inference means for each of a plurality of partial data included in a time-consecutive section; The apparatus further comprises a pseudo label assignment means for assigning a pseudo label based on the inference result for the section to at least any of the plurality of partial data in the section according to the degree of coincidence.

[0016] <Configuration of information processing device 1> The configuration of an information processing device 1 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of the information processing device 1.

[0017] 1, the information processing device 1 includes an inference unit 11, a calculation unit 12, and a pseudo-labeling unit 13. The inference unit 11 is configured to realize an inference means in this exemplary embodiment. The calculation unit 12 is configured to realize a calculation means in this exemplary embodiment. The pseudo-labeling unit 13 is configured to realize a pseudo-labeling means in this exemplary embodiment.

[0018] The inference unit 11 infers the class of the input data using an inference model. The calculation unit 12 calculates the degree of agreement for the inference result output by the inference unit 11. The pseudo label assignment unit 13 assigns a pseudo label to the input data by referring to the calculation result of the calculation unit 12. The functions of each unit of the information processing device 1 will be described in more detail with reference to FIG. 2.

[0019] Fig. 2 is a diagram for explaining the function of each unit of the information processing device 1 in Fig. 1. In this example, it is assumed that time series data SD is input to the information processing device 1. As an example, the time series data is video data consisting of data of frames that are consecutive in time.

[0020] The time series data shown in Fig. 2 includes partial data. As an example, the partial data is frame data included in the video data. In the example of Fig. 2, a partial data ID is assigned to each piece of partial data, and partial data IDs PD01, PD02, PD03, PD04, PD05, and PD06 are assigned. Hereinafter, partial data assigned a partial data ID will be appropriately referred to as partial data PD01, partial data PD02, and so on.

[0021] Further, no class label has been assigned to the partial data PD01 to PD06, and therefore, the class label "not assigned" is shown in Fig. 2.

[0022] The inference unit 11 infers a class for each piece of partial data constituting the time series data. The inference unit 11 infers a class for each piece of partial data by using an inference model that is composed of model parameters and an arithmetic expression.

[0023] As an example, the time-series data is video data. In this case, the inference unit 11 infers the behavior of the subject (for example, walking, sitting, etc.) in each frame constituting the video as a class.

[0024] 2 shows an inference result by the inference unit 11. The inference unit 11 infers the class of each piece of partial data, and outputs a score representing the probability that each piece of partial data is classified into each of class 1, class 2, and class 3. For example, in the example of FIG. 2, "CL1: 0.90, CL2: 0.07, CL3: 0.03" is output as the inference result of the class of partial data PD1.

[0025] This inference result indicates that the probability that partial data PD1 will be classified into class 1 is 0.90, the probability that it will be classified into class 2 is 0.07, and the probability that it will be classified into class 3 is 0.03. Here, CL1, CL2, and CL3 respectively mean class 1, class 2, and class 3, and 0.90 is the score corresponding to class 1, 0.07 is the score corresponding to class 2, and 0.03 is the score corresponding to class 3.

[0026] Similarly, the inference results for the partial data PD02 to PD06 in Fig. 2 are also output. That is, as the inference results by the inference unit 11, the confidence levels for each of the multiple classes are output.

[0027] The calculation unit 12 extracts a partial data group of a temporally continuous section from the partial data PD01 to PD06. In the example of Fig. 2, the partial data PD02 to PD04 are extracted as the partial data group. At this time, a class label has not yet been assigned to the extracted partial data group, so the class label "not assigned" is shown.

[0028] The calculation unit 12 calculates the degree of agreement between the inference results of the extracted partial data. As an example, the degree of agreement is calculated based on whether the classes having the highest scores in the inference results match. In the example of FIG. 2, in the partial data group of partial data PD02 to partial data PD04, the classes having the highest scores are all class 1, so that three classes out of three partial data match. Therefore, the degree of agreement is calculated as 3 / 3=1.

[0029] That is, the calculation unit 12 calculates the degree of coincidence between the inference results by the inference unit 11 for each of a plurality of partial data included in a time-continuous section.

[0030] The pseudo labeling unit 13 judges whether the degree of agreement between the inference results of the extracted partial data satisfies a predetermined condition. At this time, for example, the calculation unit 12 judges whether the calculated degree of agreement exceeds a preset threshold. For example, when the threshold is 0.66 (= 2 / 3), it is judged that the degree of agreement between the inference results of the partial data group in FIG. 2 satisfies the predetermined condition.

[0031] The pseudo labeling unit 13 assigns a pseudo label to a partial data group whose mutual matching of the inference results is determined to satisfy a predetermined condition. Usually, a label assigned to training data represents a class of actual data, and indicates a class that is a correct answer when inferring the class of the data.

[0032] On the other hand, the label assigned by the pseudo label assignment unit 13 is an inference result by the inference unit 11, and does not represent the class of the actual data. Such a label is called a pseudo label. In the example of Fig. 2, partial data groups of partial data PD02 to PD04 are described as partial data groups after pseudo labels are assigned, together with class labels (pseudo labels). In this example, CL1 representing class 1 is assigned as the pseudo label.

[0033] In general, it is considered that temporally consecutive sections in time-series data are highly correlated with each other. For example, in a video of a person, there is a high possibility that the person is performing the same action (e.g., walking, sitting, etc.) in temporally consecutive sections. Therefore, if the inference results of the classes for each partial data of the partial data group have a high degree of agreement with each other, it is considered that the inference result is highly likely to be correct.

[0034] For this reason, the pseudo label assignment unit 13 assigns a pseudo label to a partial data group for which the degree of matching between the inference results is determined to satisfy a predetermined condition.

[0035] Through such processing, the time series data SD after the pseudo labels are added is obtained. In the example of Fig. 2, the class labels are not added to the partial data PD01, the partial data PD05, and the partial data PD06, and the time series data SD in which the class labels (pseudo labels) are added to the partial data PD02 to the partial data PD04 is shown.

[0036] In this case, pseudo labels are assigned to all of the partial data PD02 to PD04 that make up the partial data group, but the pseudo label assignment unit 13 may assign pseudo labels to only a portion of the partial data that make up the partial data group.

[0037] In this way, the pseudo label assignment unit 13 assigns a pseudo label based on the inference result in the section to at least one of the multiple partial data in the section according to the degree of match.

[0038] <Flow of information processing method S1 by information processing device 1> The flow of the information processing method S1 executed by the information processing device 1 configured as above will be described with reference to Fig. 3. Fig. 3 is a flowchart showing the flow of the information processing method. As shown in the figure, the information processing includes steps S11, S12, and S13.

[0039] In step S11, the inference unit 11 infers a class for each piece of partial data constituting the time-series data. At this time, the inference unit 11 infers a class for each piece of partial data, and outputs a score representing the probability that each piece of partial data is classified into class 1, class 2, or class 3.

[0040] In step S12, the calculation unit 12 calculates the degree of agreement between the inference results for each of the plurality of partial data included in the time-continuous sections. At this time, for example, the calculation unit 12 calculates the degree of agreement based on whether or not the classes having the highest scores in the inference results match.

[0041] In step S13, the pseudo labeling unit 13 assigns a pseudo label based on the inference result in the section to at least one of the multiple partial data in the section according to the degree of coincidence. At this time, the pseudo labeling unit 13 assigns a pseudo label to a partial data group in which the degree of coincidence between the inference results is determined to satisfy a predetermined condition.

[0042] In this manner, information processing is performed. By doing so, time-series data SD after pseudo-labeling, such as that shown in FIG.

[0043] <Effects of Information Processing Device 1 and Information Processing Method> According to the information processing device 1 and information processing method S1 of this exemplary embodiment, a class is inferred for each partial data constituting time series data, the degree of consistency between the inference results for each of the multiple partial data included in a temporally consecutive interval is calculated, and a pseudo label based on the inference result for the interval is assigned to at least one of the multiple partial data in the interval depending on the degree of consistency.

[0044] In this way, even if a section to which no class label is assigned continues for a long time in the time series data, the pseudo label can be assigned to the entire time series data. That is, according to the information processing device 1 and the information processing method S1 according to the present exemplary embodiment, the pseudo label can be assigned regardless of the presence or absence of data with class labels.

[0045] Exemplary embodiment 2 A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first exemplary embodiment are given the same reference numerals, and the description thereof will be omitted as appropriate.

[0046] <Configuration of information processing device 1A> The configuration of an information processing device 1A according to this exemplary embodiment will be described with reference to the block diagram of FIG.

[0047] As an example, when video data is input as time-series data, the information processing device 1A is used for behavior recognition processing in which the behavior of a subject (e.g., walking, sitting, etc.) in each frame constituting the video is inferred as a class. As another example, when voice data is input as time-series data, the information processing device 1A is used for behavior recognition processing in which the behavior of a subject in each frame constituting the video is inferred as a class. Vocalization It is used in speech recognition to infer patterns (e.g., words, letters, etc.) as classes.

[0048] Fig. 4 is a block diagram illustrating an example of a functional configuration of the information processing device 1A. As shown in Fig. 4, the information processing device 1A includes a control unit 10A, a storage unit 20A, an input unit 30A, and a communication unit 40A.

[0049] The control unit 10A is a functional block having the same functions as the information processing device 1 described in the exemplary embodiment 1. The control unit 10A includes a data acquisition unit 101, an inference unit 11, a calculation unit 12, a pseudo label assignment unit 13, and a learning unit 14.

[0050] The data acquisition unit 101 acquires time-series data SD. The inference unit 11, the calculation unit 12, and the pseudo-labeling unit 13 each have the functions described with reference to Fig. 1 and Fig. 2. However, in the example of Fig. 4, the calculation unit 12 has an extraction unit 121 and a coincidence calculation unit 122.

[0051] The extraction unit 121 extracts a partial data group of a temporally continuous section from the partial data of the time series data SD. For example, the extraction unit 121 extracts a partial data group of a temporally continuous section from the partial data PD01 to PD06 in FIG.

[0052] The coincidence calculation unit 122 calculates the degree of coincidence between the inference results of the extracted partial data. As described above, the inference result is output as a confidence level for each of a plurality of classes for each partial data. An example of the method of calculating the degree of coincidence is as described in the first exemplary embodiment.

[0053] The learning unit 14 is a functional block that causes the inference unit 11 to learn by updating parameters of the inference model. After the pseudo label assignment unit 13 assigns pseudo labels to the partial data, the inference unit 11 can learn using the time-series data to which the pseudo labels have been assigned as teacher data. In the example of FIG. 4, the learning unit 14 has a first loss function calculation unit 141, a second loss function calculation unit 142, and a parameter update unit 143.

[0054] The first loss function calculation unit 141 calculates a loss function required for the learning of the inference unit 11 using the class label. The class label here indicates the class of actual data, and indicates a class that is the correct answer when inferring the class of the data, and is not a pseudo label. The learning of the inference unit 11 using this class label is performed, for example, before performing information processing for assigning a pseudo label in the information processing device 1A.

[0055] The second loss function calculation unit 142 calculates a loss function necessary for learning by the inference unit 11 using the pseudo labels.

[0056] The parameter update unit 143 updates the model parameters of the inference model using the loss function calculated by the first loss function calculation unit 141 or the loss function calculated by the second loss function calculation unit 142.

[0057] That is, the learning unit 14 can train the inference unit 11 using time-series data to which a class label that is a correct answer has been previously assigned. The learning unit 14 can also train the inference unit 11 using time-series data including a pseudo label assigned by a pseudo label assignment unit.

[0058] The storage unit 40 is configured, for example, by a semiconductor memory device, and stores data. In this example, the storage unit 40 stores the time series data SD, the inference result PR, the pseudo-labeled time series data PLSD, and the inference model PM.

[0059] The time series data SD, the inference result PR, and the pseudo-labeled time series data PLSD correspond to the time series data SD, the inference result PR, and the time series data after pseudo labels are added, respectively, described above with reference to Fig. 2. The pseudo-labeled time series data PLSD can also be used as teacher data when the learning unit 14 trains the inference unit 11.

[0060] The inference model PM is an inference model used by the inference unit 11 when inferring the class of each partial data, and more specifically, is a model parameter of the inference model.

[0061] The input unit 30A accepts various inputs to the information processing device 1A. The specific configuration of the input unit 30A is not limited to this exemplary embodiment, but as an example, the input unit 30A may be configured to include input devices such as a keyboard and a touch pad. The input unit 30A may also be configured to include a data scanner that reads data via electromagnetic waves such as infrared rays and radio waves, and a sensor that senses the environmental state.

[0062] The communication unit 40A is an interface for connecting the information processing device 1A to a network. Although the specific configuration of the network does not limit the present exemplary embodiment, as an example, a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, or a combination of these networks can be used.

[0063] <Flow of information processing method S1A by information processing device 1A> Next, the flow of the information processing method S1A by the information processing device 1A will be described with reference to the flowchart of Fig. 5. It is assumed that prior to this process, learning is performed by the inference unit 11 using class labels representing classes of actual data. Therefore, it is assumed that the inference unit 11 can infer the class of partial data using parameters in the initial state (initialized parameters).

[0064] In step S101, the data acquiring unit 101 acquires the time series data SD. At this time, for example, the time series data SD stored in the storage unit 40 is acquired by the data acquiring unit 101.

[0065] In step S102, the inference unit 11 initializes parameters. At this time, for example, model parameters (inference model PM) of an inference model stored in the storage unit 40 and used when the inference unit 11 infers the class of each piece of partial data are initialized.

[0066] In step S11, the inference unit 11 infers the class of the partial data. This process corresponds to step S11 in the flowchart of Fig. 3. At this time, the inference unit 11 infers the class of the partial data by using the inference model PM.

[0067] The process from step S120 to step S130 is a loop process that is repeatedly executed until it is determined that there is no partial data group to be extracted from the time-series data SD acquired in step S101.

[0068] In step S121, the extraction unit 121 extracts a partial data group from the time series data SD acquired in step S101. For example, as described above with reference to FIG. 2, the extraction unit 121 extracts a partial data group (partial data PD02 to partial data PD04) in a section that is continuous in time. Note that class labels have not yet been assigned to the partial data group extracted at this time. In step S122, the coincidence calculation unit 122 calculates the degree of coincidence between the inference results of the partial data extracted in step S121.

[0069] In step S123, the pseudo labeling unit 13 determines whether the degree of coincidence calculated in step S122 satisfies a predetermined condition. If it is determined in step S123 that the degree of coincidence does not satisfy the condition, the process returns to step S121. If it is determined in step S123 that the degree of coincidence satisfies the condition, the process proceeds to step S13.

[0070] In step S13, the pseudo label assignment unit 13 assigns pseudo labels to the partial data. This process corresponds to step S11 in the flowchart of FIG.

[0071] When the loop process from step S120 to step S130 is completed, pseudo-labeled time-series data is obtained, and the process proceeds to step S14.

[0072] In step S14, the learning unit 14 calculates a loss function and updates the parameters. At this time, the inference model PM stored in the storage unit 40 is updated. Note that the flowchart in FIG. 5 assumes learning of an inference model using pseudo labels. In step S14, the parameter update unit 143 updates the inference model PM using the loss function calculated by the second loss function calculation unit 142.

[0073] Thereafter, the process returns to step S11, and the subsequent processes are repeatedly executed, thus executing the information processing by the information processing device 1A.

[0074] (Example of calculation of match and specific conditions) Next, a specific example of the degree of match calculated in step S122 and the condition determined in step S123 will be described.

[0075] (Example 1) According to the method of calculating the degree of coincidence described in the first exemplary embodiment, the ratio of partial data in a group of partial data that matches the class having the highest score in the inference result is calculated. That is, the inference unit 11 calculates the confidence level for each of a plurality of classes for each partial data, and the pseudo label assignment unit 13 assigns a pseudo label based on the inference result for at least one of the plurality of partial data in the interval when the identity of the class with the highest confidence level for the plurality of partial data included in the interval is equal to or greater than a predetermined ratio. Here, the predetermined ratio can be, for example, 80% or more.

[0076] For example, if the inference results at times i and j in a temporally consecutive section are represented as pi and pj, the matching degree calculation unit 122 can calculate the matching degree at times i and j by the following formula.

[0077] C_ij =1 (if argmax(pi)==argmax(pj)) = 0 (else) The pseudo label assignment unit 13 assigns a pseudo label to the partial data of the partial data group when C_ij>=1 holds for 80% or more of combinations of arbitrary times i and j within the section.

[0078] Also, a pseudo label may be assigned to all of the plurality of partial data. That is, when the identity of the class with the highest confidence level is equal to or greater than a predetermined ratio for the plurality of partial data included in the interval, the pseudo label assignment unit 13 may assign a pseudo label based on the inference result for the interval to all of the plurality of partial data in the interval.

[0079] Alternatively, the inference unit 11 may calculate a confidence level for each of a plurality of classes for each partial data, and the pseudo label assignment unit 13 may assign a pseudo label based on the inference result for at least one of the plurality of partial data in a section when the distribution distance of the confidence levels for the plurality of partial data included in the section is equal to or less than a predetermined value.

[0080] In this case, the distribution of the confidence for each of the multiple classes is further calculated for the multiple partial data included in the interval, and the distribution distance for these distributions is calculated. As an example, the distribution distance is calculated as the Kullback-Leibler divergence (KL divergence).

[0081] When the distribution of the confidence levels for each of a plurality of classes is similar, the partial data are considered to be similar, and therefore the pseudo label assignment unit 13 assigns pseudo labels based on the inference results in the intervals.

[0082] (Example 2) In the first specific example, a pseudo label is assigned when the identity of the class with the highest confidence is equal to or greater than a predetermined ratio, but a pseudo label may be assigned when the class with the highest confidence is the same in the inference results of all partial data. In this case, the coincidence calculation unit 122 calculates the coincidence in the same way as in the first specific example, while the threshold value for condition determination is set high (for example, 1 or more). In this way, a pseudo label with higher certainty can be assigned.

[0083] In other words, when the class with the highest confidence level is the same for multiple partial data included in a section, the pseudo label assignment unit 13 may assign a pseudo label based on the inference result for the section to all of the multiple partial data in the section.

[0084] (Example 3) Even if the inference results of each partial data show that the class with the highest confidence (score) is the same, if the scores are significantly different, for example, the feature quantities of each partial data may also be significantly different. In such cases, it cannot be said that the class identity of each partial data is necessarily high.

[0085] Therefore, for example, the scores of the inference results between the partial data may be compared, and if the scores do not differ significantly, a pseudo label may be assigned.

[0086] For example, the partial data PD02 to PD04 included in the partial data group of FIG. 2 have the same class (CL1) with the highest score, and the scores are 0.90, 0.86, and 0.92. For example, the highest score "0.92" is compared with the lowest score "0.86" among them, and the degree of difference is calculated. The degree of difference can be calculated by the coincidence calculation unit 122, for example, from the difference or ratio between the two scores. Then, the pseudo label assignment unit 13 determines that the scores are not significantly different, for example, when the two scores do not differ by more than twice. In this case, it is determined whether the scores of CL1 differ by more than twice for the pair of partial data PD03 and partial data PD04.

[0087] Note that "two times or more" is an example, and for example, if the scores do not differ by three times or more (or four times or more), it may be determined that the scores do not differ significantly. That is, the pseudo label assignment unit 13 may assign a pseudo label based on the inference result for at least one of the multiple partial data in the interval when the class with the highest confidence level is the same for any pair of multiple partial data included in the interval and the difference in the highest confidence level is within an arbitrary constant multiple.

[0088] For example, if the inference results at times i and j in a temporally consecutive section are represented as pi and pj, the matching degree calculation unit 122 can calculate the matching degree at times i and j by the following formula.

[0089] C_ij = max(|max(p_i) / max(p_j)|, |max(p_j) / max(p_i)|) The pseudo label assignment unit 13 assigns a pseudo label to partial data of the partial data group when, for example, C_ij<=2.0 holds for any time i, j in the interval.

[0090] (Example 4) Alternatively, when the variance (dispersion) of the scores of the inference results between the partial data is low, it may be determined that the scores are not significantly different. For example, the coincidence calculation unit 122 calculates the variance of the scores of CL1 as the coincidence for the inference results of the partial data PD02 to PD04 included in the partial data group of FIG. 2. Then, the pseudo label assignment unit 13 may assign a pseudo label by when the coincidence (i.e., the variance) is less than a threshold value.

[0091] For example, when the time series data SD is video data, the partial data PD02 to PD04 are frames of a time-sequential section, so that the subject person is likely to be performing the same action. However, if there is a variation in the scores of the inference results for the frames of the time-sequential section, the inference result is likely to be incorrect.

[0092] FIG. 6 is a diagram illustrating the variation in the highest confidence (maximum confidence) among the inference results of partial data. The two graphs at the top of the figure show the variation in maximum confidence along the time axis. In the two graphs at the top of the figure, the horizontal axis is the frame time, and the vertical axis is the value of maximum confidence. In the graph on the left, the value of maximum confidence is almost constant on the time axis, and the variation in maximum confidence is small. On the other hand, in the graph on the right, the maximum confidence changes along the time axis, and the variation in maximum confidence is large.

[0093] The pseudo label assignment unit 13 may assign a pseudo label based on the inference result in the interval to at least one of the multiple partial data in the interval, depending on the variation along the time axis of the highest confidence in each of the multiple partial data included in the interval.

[0094] (Example 5) Alternatively, for partial data of a time-contiguous section, it may be determined whether the variance (dispersion) of the inference result scores corresponding to each class is large, and if it is determined that the variance is not large, a pseudo label may be assigned.

[0095] In the example described above with reference to Fig. 2, the inference result includes scores corresponding to the classes CL1, CL2, and CL3. In this case, if the score corresponding to any one of the classes is significantly high and the scores corresponding to the other classes are low, it can be said that the inference result is highly likely to be correct. On the other hand, if the score of any one of the classes is not significantly high, the inference result is less likely to be correct.

[0096] The two graphs at the bottom of Fig. 6 are diagrams illustrating the variation in the certainty of each class in the inference result of one partial data of the partial data group extracted by the extraction unit 121. In the two graphs at the bottom of the figure, the horizontal axis is the class number (e.g., CL1, CL2, CL3, ...) and the vertical axis is the maximum certainty value. Of the two graphs at the bottom of Fig. 6, the graph on the left has a maximum certainty that is significantly higher than the other certainties, and the variation in the certainty of each class is small. On the other hand, the graph on the right has a maximum certainty that is not significantly higher than the other certainties, and the variation in the certainty of each class is large.

[0097] In this case, for example, the consistency calculation unit 122 calculates the variance of the confidence included in the inference result of the partial data as the consistency. Then, when the consistency (i.e., the variance) is less than a threshold, the pseudo label assignment unit 13 may assign a pseudo label.

[0098] In other words, the pseudo label assignment unit 13 may assign a pseudo label based on the inference result for at least one of the multiple partial data in the interval, in accordance with the degree of certainty for at least one of the multiple partial data included in the interval and the variation in the degree of certainty for each class in the partial data.

[0099] (Example 6) In the above-mentioned specific example 1 to specific example 5, when the calculated degree of coincidence satisfies a predetermined condition, a pseudo label is assigned to at least one of the plurality of partial data in the section, or to all of the plurality of partial data in the section. However, a pseudo label may be assigned to only one specific piece of partial data among a group of partial data that is the plurality of partial data in the section. Here, the specific piece of partial data is referred to as a target partial data.

[0100] For example, the calculation of the degree of coincidence described in the first to fifth specific examples may be performed for a partial data group including the target partial data, and if it is determined that the condition is satisfied, a pseudo label may be assigned only to the target partial data. The target partial data may be partial data located at the center of a chronologically continuous section corresponding to the partial data group, or partial data located at the earliest or latest position in time. In other words, the position of the target partial data in a chronologically continuous section corresponding to the partial data group is arbitrary.

[0101] Furthermore, the length of the partial data group including the target partial data may vary depending on the inference result of the target partial data. For example, if the class with the highest confidence in the target partial data is class A, three frames near the target partial data become the partial data group. On the other hand, if the class with the highest confidence in the target partial data is class B, ten frames near the target partial data may become the partial data group.

[0102] In this manner, pseudo labels may be assigned while changing the time-continuous sections corresponding to the partial data groups extracted by the extraction unit 121 in accordance with the target partial data.

[0103] <Effects of information processing device 1A and information processing method S1A> According to the information processing device 1A and the information processing method S1A of this exemplary embodiment, even if a section to which no class label is assigned continues for a long time in the time-series data, a pseudo label can be assigned to the entire time-series data. That is, according to the information processing device 1A and the information processing method S1A of this exemplary embodiment, a pseudo label can be assigned regardless of the presence or absence of class-labeled data.

[0104] In addition, by appropriately specifying the calculation of the degree of coincidence and the conditions, it is possible to realize pseudo labeling according to the characteristics of the time-series data. For example, in time-series data that is data of moving images, it is possible to appropriately assign pseudo labels according to the characteristics of each case, such as when the movement of the subject is large, small, fast, or slow.

[0105] Furthermore, in the training data, data near the time that is not labeled can also be used for learning, making it possible to further improve the accuracy of inference.

[0106] Exemplary embodiment 3 Next, a third exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first or second exemplary embodiment are denoted by the same reference numerals, and the description thereof will be omitted as appropriate.

[0107] Fig. 7 is a block diagram showing a configuration example of an information processing device 1A according to exemplary embodiment 3. The information processing device 1A in Fig. 7 is different from the information processing device 1A in Fig. 4 in that the control unit 10A is not provided with a learning unit 14. The other configurations are the same as those in the configuration example shown in Fig. 4, and therefore detailed description thereof will be omitted.

[0108] The information processing device 1A in Fig. 7 does not have a function of training the inference unit 11, and is for generating pseudo-labeled time-series data PLSD. As described above, in the information processing device 1A in Fig. 4, the learning unit 14 can train the inference unit 11 by using the pseudo-labeled time-series data PLSD as teacher data.

[0109] The information processing device 1A in Fig. 7 provides the generated pseudo-labeled time series data PLSD to another information processing device (for example, the information processing device 1A shown in Fig. 4). As a result, the other information processing device uses the provided pseudo-labeled time series data PLSD as training data to train the inference unit 11. On the other hand, the other information processing device can omit the execution of the process related to pseudo labeling of the time series data SD.

[0110] <Advantages of Exemplary Embodiment 3> In this manner, the information processing device 1A according to this exemplary embodiment can generate pseudo-labeled time-series data that can be used as training data by other information processing devices.

[0111] [Software implementation example] Some or all of the functions of the information processing device 1 and the information processing device 1A may be realized by hardware such as an integrated circuit (IC chip), or may be realized by software.

[0112] In the latter case, the information processing device 1 and the information processing device 1A are realized by, for example, a computer that executes instructions of a program that is software that realizes each function. An example of such a computer (hereinafter, referred to as computer C) is shown in FIG.

[0113] The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for operating the computer C as the information processing device 1 and the information processing device 1A. In the computer C, the processor C1 reads the program P from the memory C2 and executes it to realize the functions of the information processing device 1 and the information processing device 1A.

[0114] The processor C1 may be, for example, a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a microcontroller, or a combination of these. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination of these.

[0115] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may further include a communication interface for transmitting and receiving data to and from other devices. The computer C may further include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0116] Furthermore, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can obtain the program P via such a recording medium M. Furthermore, the program P can be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also obtain the program P via such a transmission medium.

[0117] [Additional Note 1] The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiment are also included in the technical scope of the present invention.

[0118] [Additional Note 2] Some or all of the above-described embodiments can be described as follows. However, the present invention is not limited to the following described aspects.

[0119] (Appendix 1) An inference means for inferring a class for each piece of partial data constituting the time series data; a calculation means for calculating a degree of coincidence between inference results by the inference means for each of a plurality of partial data included in a time-consecutive section; a pseudo label assignment means for assigning a pseudo label based on the inference result in the section to at least one of the plurality of partial data in the section according to the degree of coincidence; An information processing device comprising:

[0120] (Appendix 2) The inference means calculates a confidence level for each of a plurality of classes for each of the partial data; The pseudo labeling means includes: When the identity of the class with the highest confidence level for a plurality of partial data included in the interval is equal to or greater than a predetermined rate, a pseudo label based on the inference result for the interval is assigned to at least one of the plurality of partial data in the interval. 2. An information processing device according to claim 1.

[0121] (Appendix 3) The pseudo labeling means includes: When the identity of the class with the highest confidence level for a plurality of partial data included in the interval is equal to or greater than a predetermined rate, a pseudo label based on the inference result for the interval is assigned to all of the plurality of partial data in the interval. 3. An information processing device according to claim 2.

[0122] (Appendix 4) The pseudo labeling means includes: When the classes with the highest confidence levels are all the same for the plurality of partial data included in the interval, a pseudo label based on the inference result for the interval is assigned to all of the plurality of partial data in the interval. 4. The information processing device according to claim 3.

[0123] (Appendix 5) The pseudo labeling means includes: When the class with the highest confidence level is the same for any pair of a plurality of partial data included in the interval and the difference between the highest confidence levels is within a factor of two, a pseudo label is assigned to at least one of the plurality of partial data in the interval based on the inference result in the interval. 2. An information processing device according to claim 1.

[0124] (Appendix 6) The pseudo labeling means includes: assigning a pseudo label based on the inference result for the section to at least one of the plurality of partial data in the section in accordance with a variation along a time axis of the highest confidence in each of the plurality of partial data included in the section; 2. An information processing device according to claim 1.

[0125] (Appendix 7) The pseudo labeling means includes: A pseudo label is assigned to at least one of the plurality of partial data in the interval, based on the inference result in the interval, in accordance with a certainty of at least one of the plurality of partial data included in the interval, the certainty of each class in the partial data being a variance. 2. An information processing device according to claim 1.

[0126] (Appendix 8) The inference means calculates a confidence level for each of a plurality of classes for each of the partial data; The pseudo labeling means includes: When a distribution distance of certainty levels for a plurality of partial data included in the interval is equal to or less than a predetermined value, a pseudo label based on the inference result for the interval is assigned to at least one of the plurality of partial data in the interval. 2. An information processing device according to claim 1.

[0127] (Appendix 9) The method further includes a learning unit that causes the inference unit to learn using time-series data including the pseudo labels assigned by the pseudo label assignment unit. 9. An information processing device according to any one of appendix 1 to 8.

[0128] (Appendix 10) Inferring classes for each piece of data that makes up the time series data; Calculating the degree of agreement between inference results for each of a plurality of partial data included in a time-consecutive section; and assigning a pseudo label based on the inference result for the section to at least one of the plurality of partial data in the section according to the degree of match. Information processing methods.

[0129] (Appendix 11) Computer, An inference means for inferring a class for each piece of partial data constituting the time series data; a calculation means for calculating a degree of coincidence between inference results by the inference means for each of a plurality of partial data included in a time-consecutive section; and a pseudo label assignment means for assigning a pseudo label based on the inference result in the section to at least any of the plurality of partial data in the section according to the degree of coincidence. program.

[0130] [Additional Note 3] A part or all of the above-described embodiments can be further expressed as follows.

[0131] At least one processor, the processor comprising: A process of inferring classes for each piece of partial data constituting the time series data; A process of calculating a degree of agreement between inference results for each of a plurality of partial data included in a time-consecutive section; A process of assigning a pseudo label based on the inference result for the section to at least one of the plurality of partial data in the section according to the degree of match is executed.

[0132] The information processing device may further include a memory that stores a program for causing the processor to execute the processes of inferring a class, calculating a degree of coincidence, and assigning a pseudo label. The program may be recorded in a computer-readable, non-transitory, tangible recording medium. [Explanation of symbols]

[0133] 1. Information processing device 10A Control section 11 Reasoning part 12 Calculation section 13 Pseudo labeling unit 14 Learning Department 20A storage section 30A input section 40A Communication section 121 Extraction part 122 Matching calculation part 141 First loss function calculation unit 142 Second loss function calculation part 143 Parameter Update Section

Claims

1. An inference means for inferring a class for each piece of partial data constituting the time series data; a calculation means for calculating a degree of coincidence between inference results by the inference means for each of a plurality of partial data included in a time-consecutive section; a pseudo label assignment means for assigning a pseudo label based on the inference result in the section to at least one of the plurality of partial data in the section according to the degree of coincidence; An information processing device comprising:

2. The inference means calculates a confidence level for each of a plurality of classes for each of the partial data; The pseudo labeling means includes: When the identity of the class with the highest confidence level for a plurality of partial data included in the interval is equal to or greater than a predetermined ratio, a pseudo label based on the inference result for the interval is assigned to at least one of the plurality of partial data in the interval. The information processing device according to claim 1 .

3. The pseudo labeling means includes: When the identity of the class with the highest confidence level for a plurality of partial data included in the interval is equal to or greater than a predetermined rate, a pseudo label based on the inference result for the interval is assigned to all of the plurality of partial data in the interval. The information processing device according to claim 2 .

4. The pseudo labeling means includes: When the classes with the highest confidence levels are all the same for the plurality of partial data included in the interval, a pseudo label based on the inference result for the interval is assigned to all of the plurality of partial data in the interval. The information processing device according to claim 3 .

5. The pseudo labeling means includes: When the class with the highest confidence level is the same for any pair of a plurality of partial data included in the interval and the difference in the highest confidence level is within an arbitrary constant multiple, a pseudo label based on the inference result for the interval is assigned to at least one of the plurality of partial data in the interval. The information processing device according to claim 1 .

6. The pseudo labeling means includes: A pseudo label based on the inference result for the section is assigned to at least one of the plurality of partial data in the section in accordance with a variation along a time axis of the highest confidence in each of the plurality of partial data included in the section. The information processing device according to claim 1 .

7. The pseudo labeling means includes: A pseudo label is assigned to at least one of the plurality of partial data in the interval, based on the inference result in the interval, in accordance with the degree of certainty of at least one of the plurality of partial data included in the interval, the degree of certainty being a variance of the degree of certainty of each class in the partial data. The information processing device according to claim 1 .

8. The inference means calculates a confidence level for each of a plurality of classes for each of the partial data; The pseudo labeling means includes: When a distribution distance of certainty levels for a plurality of partial data included in the interval is equal to or less than a predetermined value, a pseudo label based on the inference result for the interval is assigned to at least one of the plurality of partial data in the interval. The information processing device according to claim 1 .

9. A computer comprising: Inferring classes for each piece of data that makes up the time series data; Calculating the degree of agreement between inference results for each of a plurality of partial data included in a time-consecutive section; and assigning a pseudo label based on the inference result for the section to at least one of the plurality of partial data in the section according to the degree of match. Information processing methods.

10. Computer, An inference means for inferring a class for each piece of partial data constituting the time series data; a calculation means for calculating a degree of coincidence between inference results by the inference means for each of a plurality of partial data included in a time-consecutive section; and a pseudo label assignment means for assigning a pseudo label based on the inference result in the section to at least any of the plurality of partial data in the section according to the degree of coincidence. program.

Images