Information processing device, method, and program

Abstract

This creates a time-series correspondence between data representing actions. [Solution] The information processing device 1X includes an acquisition means 21X and a identification means 24X. The acquisition means 21X acquires queries having data for each time step and templates to be compared with the queries. When the identification means 24X identifies the correspondence between the time steps of queries and templates, it extracts the time steps of queries and templates that are not currently associated, and repeats the process of associating the time steps of queries and templates that have the highest similarity between the extracted time steps of queries and templates.

Description

[Technical Field]

[0001] This disclosure relates to the technical field of information processing devices, methods, and programs for processing data representing operations at each time step. [Background technology]

[0002] Techniques exist for analyzing time-series motion data. For example, Patent Document 1 discloses a technique for augmenting motion data using multiple motion data sets. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2024-117861 [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] When generating pseudo-data representing time-series data through data augmentation, it is necessary to establish time-series correspondences between the time-series data used for data augmentation. On the other hand, there was a problem in that if data augmentation was performed based on time-series data where the same operational stages were not properly correlated, it would not be possible to generate realistic operational data.

[0005] In view of the above-mentioned issues, one of the objectives of this disclosure is to provide an information processing device, method, and program that can accurately perform time-series mapping of data representing operations at each time step. [Means for solving the problem]

[0006] One aspect of an information processing device is: A means for obtaining a query, which is data representing the operation for each time step, and a template to be compared with the query, When identifying the correspondence between the time steps of the aforementioned query and the aforementioned template, A correspondence identification means that repeatedly performs the process of extracting the time step of the query and the time step of the template that are not currently associated, and associating the query time step and the template time step that have the highest similarity among the extracted query time steps and template time steps, It is an information processing device.

[0007] One aspect of the method is: Computers The system obtains a query, which is data representing the actions at each time step, and a template to compare with the query. When identifying the correspondence between the time steps of the aforementioned query and the aforementioned template, Extract the time steps of the query and the template that are not currently associated, and associate the time steps of the query and the template that have the highest similarity among the extracted time steps of the query and the template. Repeat this process. It is a method.

[0008] One aspect of the program is: The system obtains a query, which is data representing the actions at each time step, and a template to compare with the query. When identifying the correspondence between the time steps of the aforementioned query and the aforementioned template, The process involves extracting the time steps of the query and the template that are not currently associated, and then associating the query time step with the template time step that has the highest similarity among the extracted query time steps with the template time steps, and repeating this process. It is a program that instructs a computer to perform a process. [Effects of the Invention]

[0009] As an example of one effect of the present disclosure, it is possible to suitably perform the association in the time series of data representing the operation for each time step.

Brief Description of the Drawings

[0010] [Figure 1] Shows a schematic configuration of a pseudo data generation system. [Figure 2] Shows a hardware configuration of a pseudo data generation device. [Figure 3] Is an example of a functional block of a pseudo data generation device. [Figure 4] Shows an example of specifying the correspondence between a template and a query. [Figure 5] Shows an overview of the association between a template having 8 time steps and a query having 6 time steps. [Figure 6] Shows a similarity matrix representing the similarity between a template and a query for each time step. [Figure 7] Is a diagram showing an overview of the first to third processes of (Process X) and (Process Y) by a similarity matrix. [Figure 8] Is a diagram showing an overview of the fourth to sixth processes of (Process X) and (Process Y) by a similarity matrix. [Figure 9] Shows an overview of the association between a template and a query in a comparative example in which time steps of a template and a query having the highest similarity are associated with each other. [Figure 10] Is an example of a flowchart executed by a pseudo data generation device. [Figure 11] Is an example of a functional block of a pseudo data generation device having a configuration for adjusting the time series length of all samples. [Figure 12] Is an example of a flowchart representing a processing flow of a pseudo data generation device having a configuration for adjusting the time series length of all samples. [Figure 13] Is a block diagram of an information processing device. [Figure 14] Is an example of a flowchart showing a processing procedure of an information processing device. [Modes for carrying out the invention]

[0011] The following describes embodiments of the information processing device, method, and program with reference to the drawings.

[0012] <First Embodiment> (1) System Configuration Figure 1 shows a schematic configuration of the pseudo-data generation system 100. The pseudo-data generation system 100 generates pseudo-data of time-series data for machine learning through data augmentation. The pseudo-data generation system 100 mainly comprises a pseudo-data generation device 1, a storage device 2 that stores time-series data group D1 and feature extractor information D2, a display device 3, and an input device 4.

[0013] The pseudo-data generation device 1 performs data augmentation on the time series data set D1, which is the training data for a machine learning model, and generates pseudo-data that corresponds to intermediate data between any two time series data sets included in the time series data set D1. In this case, the pseudo-data generation device 1 extracts features from the time series data set D1 using a feature extractor configured with feature extractor information D2, and identifies the correspondence between time series data sets at the time step level based on the similarity of the extracted features. As a result, the pseudo-data generation device 1 maintains time series consistency between any two time series data sets and generates pseudo-data that serves as intermediate data between them. The pseudo-data generation device 1 may also display information to be presented to the user of the pseudo-data generation system 100 using the display device 3, or accept user input (so-called external input) using the input device 4.

[0014] The storage device 2 is a memory that stores various information necessary for processing the pseudo-data generation device 1, and functionally has a time-series data group D1 and feature extractor information D2.

[0015] The time series data set D1 consists of "N" (N is an integer greater than or equal to 2) time series data points used in machine learning of a machine learning model. Each time series data point is a pair of a sample representing the subject's actions at each time step and a label indicating whether the sequence of actions is correct or incorrect. For example, time series data set D1 is training data for a machine learning model that infers the correctness of actions. In this case, time series data set D1 consists of N records of training data, where one record represents the sample for input to the machine learning model and the label indicating the correct answer that the machine learning model should output. The actions could be, for example, rehabilitation exercises, sports exercises, or any other arbitrary actions. For the sake of explanation, the following explanation will assume that the subject of the actions is a person, but the subject of the actions is not limited to a person; it could be any moving object such as an animal or a robot.

[0016] A sample may be, for example, a coordinate value representing the position of a person's joints (skeleton) at each time step, or a video image (i.e., RGB values ​​for each pixel) of a person in motion captured at each time step. A sample representing the position of a joint (skeleton) at each time step will be a tensor of size "number of channels (dimensions of the coordinate space) × number of joints × number of time steps", and a sample representing a video image will be a tensor of size "number of channels (RGB) × vertical resolution × horizontal resolution × number of time steps". A label is, for example, a binary value representing whether the motion is correct or incorrect. Samples may have different number of time steps. Note that the time series data group D1 may be treated as a single data in tensor format with size "number of samples (N) × number of channels × number of joints × number of time steps" or "number of samples (N) × number of channels × vertical resolution × horizontal resolution × number of time steps".

[0017] Feature extractor information D2 is the information necessary to construct a feature extractor, which is a machine learning model that extracts features from a sample. For example, it includes parameters related to the architecture for constructing the feature extractor and parameters obtained through machine learning. Specifically, if the feature extractor is a neural network, feature extractor information D2 includes various parameters (including hyperparameters) such as the layer structure, the neuron structure of each layer, the number and size of filters in each layer, and the weights of each element of each filter. For example, a feature extractor extracts a feature vector of a predetermined number of dimensions from the input sample at each time step. Therefore, when N samples are input, the feature extractor outputs a feature quantity (feature map) equal to (number of samples (N) × number of time steps × number of dimensions of the feature vector). In this case, the feature quantity (feature map) corresponds to a tensor obtained by convolution.

[0018] The storage device 2 may be an external storage device such as a hard disk connected to or built into the pseudo-data generation device 1, or it may be a portable storage medium such as flash memory. Furthermore, the storage device 2 may be a server device that communicates data with the pseudo-data generation device 1. Also, the storage device 2 may be composed of multiple devices.

[0019] The display device 3 displays information based on the control of the pseudo-data generation device 1. Examples of the display device 3 include displays, projectors, etc. When the display device 3 receives a display signal supplied from the pseudo-data generation device 1, it displays information based on the received display signal.

[0020] Input device 4 is an interface that accepts user input, which is external input based on user operations using the pseudo-data generation system 100. Examples include a touch panel, buttons, a keyboard, and a voice input device. Input device 4 supplies the input signal generated based on the user input to the pseudo-data generation device 1.

[0021] The configuration of the pseudo-data generation system 100 shown in Figure 1 is an example, and various modifications may be made to this configuration. For example, the pseudo-data generation device 1, the storage device 2, the display device 3, and the input device 4 may be integrated into any combination. The pseudo-data generation system 100 may also be equipped with a sound output device such as a speaker. Furthermore, the pseudo-data generation device 1 may be composed of multiple devices. In this case, the multiple devices constituting the pseudo-data generation device 1 exchange information among themselves that is necessary to execute pre-assigned processes.

[0022] (2) Hardware configuration Figure 2 shows the hardware configuration of the pseudo-data generation device 1. The pseudo-data generation device 1 includes a processor 11, memory 12, and interface 13 as hardware components. The processor 11, memory 12, and interface 13 are connected via a data bus 19.

[0023] The processor 11 executes predetermined processes by running programs stored in memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0024] Memory 12 is composed of various volatile and non-volatile memories, such as RAM (Random Access Memory) and ROM (Read Only Memory). Memory 12 also stores programs for the pseudo-data generation device 1 to perform various processes. Furthermore, Memory 12 is used as working memory to temporarily store information obtained from the storage device 2. Memory 12 may also function as the storage device 2. Similarly, storage device 2 may function as the memory 12 of the pseudo-data generation device 1. The programs executed by the pseudo-data generation device 1 may be stored in a storage medium other than Memory 12.

[0025] Interface 13 is an interface for electrically connecting the pseudo-data generation device 1 with other devices. These interfaces may be wireless interfaces such as network adapters for wirelessly transmitting and receiving data with other devices, or they may be hardware interfaces for connecting with other devices via cables, etc.

[0026] The hardware configuration of the pseudo-data generation device 1 is not limited to the configuration shown in Figure 2. For example, the pseudo-data generation device 1 may include at least one of the display device 3 or the input device 4. Furthermore, the pseudo-data generation device 1 may be connected to or have a sound output device such as a speaker built into it.

[0027] (3) Overview of the pseudo-data generation process This section outlines the pseudo-data generation process performed by the pseudo-data generation device 1. In general, the pseudo-data generation device 1 defines time-series data selected from the time-series data group D1 as templates and time-series data other than templates as queries, and identifies the correspondence between templates and queries at the time-step level. In this case, the pseudo-data generation device 1 sequentially selects pairs of templates and queries that have the highest feature similarity between the time steps of the unassociated templates and the time steps of the queries, without repetition. This allows the device 1 to accurately maintain time-series consistency between time-series data and suitably generate pseudo-data that serves as intermediate data between time-series data with consistent time-series relationships.

[0028] Figure 3 shows an example of the functional blocks of the pseudo-data generation device 1. As shown in Figure 3, the processor 11 of the pseudo-data generation device 1 functionally includes a feature extraction unit 21, a template selection unit 22, a similarity calculation unit 23, a correspondence identification unit 24, and a data generation unit 25. In Figure 3, blocks where data is exchanged are connected by solid lines, but the combination of blocks where data is exchanged is not limited to this. The same applies to the diagrams of other functional blocks described later.

[0029] The feature extraction unit 21 extracts feature quantities from each of the N time-series data samples extracted from the time-series data set D1. In this case, the feature extraction unit 21 obtains the feature quantities that the feature extractor would output if N samples were input to the feature extractor configured by referring to the feature extractor information D2. In this case, the feature extraction unit 21 obtains a feature quantity for each of the N samples that has a size of (number of time steps × feature dimension). The feature extraction unit 21 then supplies the feature quantities corresponding to the N samples, along with the corresponding labels, to the template selection unit 22.

[0030] The template selection unit 22 selects a template from N samples. For example, the template selection unit 22 randomly selects a sample as a template that has a label indicating that it represents the correct operation. However, the template selection unit 22 may select a template from N samples by any method other than the one described above. For example, if the template selection unit 22 obtains an input specifying a template from an input device 4 or the like, it may select the sample specified by that input as the template. The template selection unit 22 supplies the feature quantities of the selected template and the feature quantities of the samples other than the template (i.e., queries) to the similarity calculation unit 23.

[0031] The similarity calculation unit 23 calculates the similarity between the template features and the query features selected from samples other than the template, on a time-step basis. In this case, the similarity calculation unit 23 calculates a similarity matrix that represents the similarity of all combinations of features between the template and the query on a time-step basis. For example, between a template with time step number "T1" and a query with time step number "T2", the similarity calculation unit 23 calculates a similarity matrix that represents T1 × T2 similarities, corresponding to the total number of time step combinations. The similarity calculation unit 23 may use cosine similarity, L1 distance, L2 distance, or any other index representing the degree of similarity between vectors as the similarity. The similarity calculation unit 23 then calculates the above-mentioned similarity matrix for each of the N-1 queries and supplies the calculated similarity matrix to the correspondence identification unit 24.

[0032] The correspondence identification unit 24 identifies the correspondence between the template and each query on a timestep basis, based on the similarity matrix calculated by the similarity calculation unit 23. In this case, the correspondence identification unit 24 associates the pairs of timesteps of the template that are not yet associated with the timesteps of the queries to be processed with the highest similarity of features, without any overlap. Specifically, when the correspondence identification unit 24 identifies the correspondence between the template and the queries to be processed on a timestep basis, it repeats the following (process X) and (process Y) until the association of all timesteps of the template is completed.

[0033] (Process X) Extract the unsupported time steps of the query to be processed and the unsupported time steps of the template, respectively. (Process Y) The pairs with the highest feature similarity between the unmatched time steps of the extracted queries and the unmatched time steps of the template are matched.

[0034] In (processing Y), if there are multiple pairs with the highest similarity of features, the correspondence identification unit 24 associates one pair, for example, by randomly selecting it.

[0035] Furthermore, if the query's time step is shorter than the template's time step, and there are no longer any query time steps that are not associated in (process X), the correspondence identification unit 24 will, for example, perform the association using the following first or second method.

[0036] In the first method, the correspondence identification unit 24 associates each time step of the unassociated template with the time step of the query that is most similar to it. In this case, the query will have time steps that are associated with multiple time steps of the template.

[0037] In the second method, the correspondence identification unit 24 assigns each time step of the unassigned template to a time step that is consistent with the assignment of adjacent time steps to each time step of the unassigned template. For example, if the second time step of the template is not assigned, the time step of the query that is assigned to the first or third time step of the adjacent template is also assigned to the second time step of the template.

[0038] Then, the correspondence identification unit 24 reconstructs the query based on the identified correspondence relationship so that it matches the sequence of time steps in the template. In other words, the correspondence identification unit 24 rearranges the time steps of the query that correspond to these time steps according to the sequence of time steps in the template.

[0039] For example, consider a case where a query with four time steps, "Q1, Q2, Q3, Q4," is associated with a template with five time steps, "T1, T2, T3, T4, T5," as "(T1,Q1), (T2,Q1), (T3,Q3), (T4,Q2), (T5,Q4)." In this case, the correspondence identification unit 24 rearranges the queries in the order "Q1," "Q1," "Q3," "Q2," and "Q4." As a result, the reconstructed queries have the same number of time steps as the template (5), and are consistent with the template in terms of time series.

[0040] The correspondence identification unit 24 then supplies each reconstructed query and template, along with its corresponding label, to the data generation unit 25 based on the correspondence with the template. Hereafter, the N-1 reconstructed queries and templates will not be distinguished and will also be referred to as "samples". In this case, the correspondence identification unit 24 supplies N samples along with their corresponding labels to the data generation unit 25.

[0041] The data generation unit 25 generates pseudo data based on the pairs of samples and labels supplied from the correspondence identification unit 24. In this case, the pseudo data is a pair of a pseudo sample and a pseudo label corresponding to the pseudo sample. For example, if the N samples after reconstruction are X = {x1, x2,..., x N}, and the corresponding labels are N = {y1, y2,..., y N}, when expressed like this, the data generation unit 25 randomly selects any two pairs of samples and labels (x i , y i ), (x j , y j ), linearly interpolates the selected samples x i , x j to generate a pseudo sample, and linearly interpolates the labels y i , y j to generate a pseudo label. In this case, if λ is a hyperparameter that becomes 0 to 1 or a value randomly sampled, the pseudo sample and the pseudo label are calculated as follows respectively. λx i +(1 - λ)x j λy i +(1 - λ)y j

[0042] Note that the data generation unit 25 is not limited to the above-mentioned linear interpolation, and may generate a pseudo sample and a pseudo label using any algorithm for integrating two pieces of data.

[0043] Then, the data generation unit 25 generates a predetermined number of pseudo data by repeating the selection of any two pairs of samples and labels and the generation of pseudo samples and pseudo labels any number of times. The pseudo data generated in this way becomes intermediate data generated based on samples in which the same operation stages are appropriately associated even when one sample includes an incorrect unit operation.

[0044] Here, the feature extraction unit 21, template selection unit 22, similarity calculation unit 23, correspondence identification unit 24, and data generation unit 25 can be realized, for example, by the processor 11 executing a program. Alternatively, the necessary programs may be recorded on any non-volatile storage medium and installed as needed to realize each component. At least a portion of these components may be realized not only by software programs, but also by any combination of hardware, firmware, and software. Furthermore, at least a portion of these components may be realized using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the program composed of the above components may be realized using this integrated circuit. Also, at least a portion of each component may be composed of an ASSP (Application Specific Standard Produce), ASIC (Application Specific Integrated Circuit), or quantum processor (quantum computer control chip). Thus, each component may be realized by various hardware. The same applies to other embodiments described later. Furthermore, each of these components may be realized through the collaboration of multiple computers, for example, using cloud computing technology.

[0045] Here, we will use Figure 4 to provide a supplementary explanation of the effects of executing the process based on the functional block diagram shown in Figure 3.

[0046] Figure 4 shows an example of identifying the correspondence between templates and queries. Here, each time-series data represents the joint positions (corresponding to the circles in the figure) of a person performing unit actions A, B, C, and D in order, at each time step. The time-series data of a person who correctly performed all unit actions A through D is selected as the template, and the time-series data of a person who correctly performed unit actions A, B, and D, but incorrectly performed unit action C is selected as the query.

[0047] Here, if we assume that we are associating the time step pairs that have the highest similarity between the template and the query, then since the query's unit action C is an incorrect action, the similarity of unit action C between the template and the query is low despite them being the same operation stage. As a result, the similarity between the template's unit action C and the query's unit action C (0.6) is lower than the similarity between the template's unit action C and the query's unit action D (0.7). Therefore, when we associate the time steps that have the highest similarity, the template's unit action C and the query's unit action D are associated, resulting in the association of different operation stages. In this case, the query is reconstructed so that it follows the order of "correct unit action A", "correct unit action B", "correct unit action D", and "correct unit action D", and "incorrect unit action C" is no longer included. As a result, the pseudo-data generation device 1 cannot generate an intermediate operation between the correct and incorrect operation for unit action C, and there is a problem that the time step of the incorrect operation is not used in the generation of pseudo-data.

[0048] Taking the above into consideration, the correspondence identification unit 24 according to this embodiment sequentially selects, without duplication, the pair with the highest similarity of features between the time step of the unassociated template and the time step of the query. In this case, those with uncertain correspondence relationships remain, and candidates for correspondence are selected from among them, so correspondence is made sequentially starting with "pairs of the same operation stage that constitute correct unit operations" where the correspondence relationship is certain. Subsequently, "pairs of the same operation stage that include an incorrect unit operation" where the correspondence relationship is uncertain, and other unassociated unit operations remain, and "pairs of the same operation stage that include an incorrect unit operation" are associated. In this way, the pseudo-data generation device 1 can accurately limit the candidates for incorrect unit operations and accurately associate time steps of the same operation stage. Specific examples will be described later.

[0049] (4) Specific example Figure 5 shows an overview of the correspondence between a template with 8 time steps and a query with 6 time steps. Figure 6 shows a similarity matrix representing the similarity between the template and the query for each time step. Here, the sample is assumed to be data for each time step, including unit actions 1 to 4, and each time step in the template and query is represented by a circle. Furthermore, the circle representing a time step corresponding to any of unit actions 1 to 4 is explicitly labeled with a number (1 to 4) indicating that it corresponds to one of unit actions 1 to 4. Also, the data for the 4th time step of the query represents an incorrect unit action 3.

[0050] In the example shown in Figure 5, the correspondence identification unit 24 repeatedly executes the above-described (process X) and (process Y) based on the similarity matrix shown in Figure 6. Here, the correspondence identification unit 24 associates the third time step of the template with the second time step of the query using the first (process X) and (process Y), and associates the eighth time step of the template with the sixth time step of the query using the second (process X) and (process Y). Furthermore, the correspondence identification unit 24 associates the first time step of the template with the first time step of the query using the third (process X) and (process Y), and associates the fourth time step of the template with the third time step of the query using the fourth (process X) and (process Y). Furthermore, the correspondence identification unit 24 associates the sixth time step of the template with the fourth time step of the query through the fifth (process X) and (process Y), and associates the seventh time step of the template with the fifth time step of the query through the sixth (process X) and (process Y).

[0051] Figure 7 shows a similarity matrix illustrating the outlines of the first to third processes of (Process X) and (Process Y), and Figure 8 shows a similarity matrix illustrating the outlines of the fourth to sixth processes of (Process X) and (Process Y). In Figures 7 and 8, elements corresponding to the associated time steps are filled in.

[0052] In the first processing of (Process X) and (Process Y), the similarity between the third time step of the template and the second time step of the query is maximized among all elements of the similarity matrix, and therefore the third time step of the template and the second time step of the query are associated. As a result, the third time step of the template and the second time step of the query are excluded from the candidates for association in subsequent processes.

[0053] In the second processing of (Process X) and (Process Y), the eighth time step of the template is associated with the sixth time step of the query, and the eighth time step of the template and the sixth time step of the query are excluded from the candidates for association in the third and subsequent processing. In the third processing of (Process X) and (Process Y), the first time step of the template is associated with the first time step of the query, and the first time step of the template and the first time step of the query are excluded from the candidates for association in the fourth and subsequent processing.

[0054] In the fourth iteration of (Process X) and (Process Y), the fourth time step of the template is associated with the third time step of the query, and the fourth time step of the template and the third time step of the query are excluded from the candidates for association in the fifth and subsequent iterations. In the fifth iteration of (Process X) and (Process Y), the sixth time step of the template is associated with the fourth time step of the query, and the sixth time step of the template and the fourth time step of the query are excluded from the candidates for association in the sixth and subsequent iterations. The sixth iteration of (Process X) and (Process Y) associates the seventh time step of the template with the fifth time step of the query.

[0055] Subsequently, the correspondence identification unit 24, since there are no unmatched time steps in the query during the 7th (process X), associates the unmatched 2nd and 5th time steps of the template with (process X) and (process Y) independently. For example, the correspondence identification unit 24 associates the 1st and 3rd time steps of the query, which correspond to the 1st and 4th time steps that are in the vicinity of the 2nd and 5th time steps of the template, with the 2nd and 5th time steps of the template.

[0056] As described above, the correspondence identification unit 24 can appropriately associate time steps of the same operation stage, including the fourth time step of a query that would result in an incorrect unit operation.

[0057] Figure 9 shows an overview of the template-query mapping in a comparative example where the time steps of the template and query with the highest similarity are matched. In the comparative example, when the similarity matrix shown in Figure 6 is obtained, each time step of the template is matched with the time step of the query with the highest similarity based on this similarity matrix. As a result, the fourth time step of the query, which represents an incorrect unit operation, is not matched. In this case, intermediate operations between the incorrect operation and the correct operation are not generated, and unit operations at different operation stages are linearly interpolated, resulting in the generation of pseudo-data representing unrealistic intermediate operations.

[0058] (5) Processing flow Figure 10 shows an example of a flowchart executed by the pseudo-data generation device 1.

[0059] The pseudo-data generator 1 acquires the time-series data set D1 (step S11). Then, the pseudo-data generator 1 extracts the features of N samples from the time-series data set D1 (step S12). In this case, the pseudo-data generator 1 uses a feature extractor configured by referring to the feature extractor information D2 to extract the feature vectors for each time step of N samples from the time-series data set D1.

[0060] Next, the pseudo-data generator 1 selects a template (step S13). In this case, the pseudo-data generator 1 selects one template from samples associated with labels indicating correct operation. Then, the pseudo-data generator 1 selects a query from samples that are not templates and calculates a similarity matrix representing the similarity between the selected query and the template in terms of time steps (step S14).

[0061] Next, the pseudo-data generator 1 extracts the unmatched time steps in the query and the unmatched time steps in the template (step S15). This process corresponds to (process X). Then, the pseudo-data generator 1 associates the time steps of the query and the template with the highest similarity from the extracted unmatched time steps (step S16). This process corresponds to (process Y). Furthermore, if there are no unmatched time steps in the query, the pseudo-data generator 1 performs an association for each unmatched time step in the template that is consistent with the association of adjacent time steps.

[0062] The pseudo-data generator 1 then determines whether all time steps of the template have been matched (step S17). If there are unmatched time steps in the template (step S17; No), the pseudo-data generator 1 returns to step S15 and executes the processes of steps S15 and S16 again.

[0063] On the other hand, if it is determined that all time steps of the template have been mapped (step S17; Yes), the pseudo-data generator 1 determines whether the mapping of all samples has been completed (step S18). That is, the pseudo-data generator 1 determines whether it has selected N-1 samples other than the sample that will be used as the template as queries. If the mapping of all samples has not been completed (step S18; No), the pseudo-data generator 1 returns to step S14 and selects the samples that have not been selected as either templates or queries as queries.

[0064] Then, once the matching of all samples is complete (step S18; Yes), the pseudo-data generator 1 generates pseudo-data from the time-series data set reconstructed based on the matching (step S19). In this case, the pseudo-data generator 1 rearranges the data for each of the N-1 samples selected as queries so that the template and time step are consistent. The pseudo-data generator 1 then generates pseudo-data that includes pseudo-samples and pseudo-labels that are intermediate data between any two samples and their corresponding labels.

[0065] (6) Application examples According to the above embodiment, it is possible to perform data augmentation on training data for machine learning models applicable to fields such as healthcare, and to train machine learning models that can output highly accurate inference results. For example, it is possible to augment the training data that can be used for machine learning a motion correctness judgment model that outputs an inference result on whether a rehabilitation exercise is correct or incorrect when video data of a rehabilitation exercise is input. Then, by using the inference results output by the motion correctness judgment model that has been machine-learned with high accuracy using the augmented training data, it is possible to suitably support decision-making by medical professionals such as doctors and nurses regarding guidance on rehabilitation to patients. Thus, this embodiment is suitably applied, for example, to data augmentation of training data for machine learning models that output inference results that can be used for decision-making by medical professionals in the field of healthcare.

[0066] <Second Embodiment> In the second embodiment, the pseudo-data generation device 1 processes each sample in the time-series data group D1 so that the time-series length (i.e., the number of time steps) of all samples becomes uniform, and then performs the processing based on the first embodiment. As a result, the pseudo-data generation device 1 in the second embodiment makes the number of time steps of the template and the query the same, and performs the correspondence between the template and the query appropriately.

[0067] Figure 11 shows an example of a functional block of a pseudo-data generation device 1 having a configuration for adjusting the time series length of all samples. In this case, the processor 11 of the pseudo-data generation device 1 has a feature extraction unit 21, a template selection unit 22, a similarity calculation unit 23, a correspondence identification unit 24, and a data generation unit 25 based on the first embodiment, and a time series length adjustment unit 20 provided in front of the feature extraction unit 21.

[0068] The time series length adjustment unit 20 acquires the time series data group D1 and then obtains the time step count (also called the "maximum time step count") of the sample that has the maximum time step count (time series length) from among the N samples of the time series data group D1. The time series length adjustment unit 20 then interpolates each sample in the time axis direction (e.g., linear interpolation) so that all N samples become data with the maximum time step count. For example, if the sample of the time series data group D1 is in the tensor format of (number of channels × number of joints × number of time steps), the unit converts the sample into tensor format data of (number of channels × number of joints × maximum time step count). As a result, the N samples output by the time series length adjustment unit 20 are represented by tensor format data of (number of samples × number of channels × number of joints × maximum time step count). The time series length adjustment unit 20 supplies the N samples with adjusted time series lengths and their corresponding labels to the feature extraction unit 21. The time series length adjustment unit 20 may adjust the number of sample time steps to a predetermined number, rather than setting the number of N sample time steps to the maximum number of time steps.

[0069] The feature extraction unit 21 extracts feature quantities from N samples after adjusting the time series length. The template selection unit 22, similarity calculation unit 23, correspondence identification unit 24, and data generation unit 25 perform the same processing as in the first embodiment based on the feature quantities generated by the feature extraction unit 21 to generate pseudo-data.

[0070] Figure 12 is an example flowchart showing the processing flow of a pseudo-data generation device 1 that has a configuration to adjust the time series length of all samples.

[0071] First, the pseudo-data generator 1 acquires the time-series data group D1 and unifies the time-series lengths of the N samples included in the time-series data group D1 (step S21). In this case, the pseudo-data generator 1 transforms the N samples to achieve the maximum number of time steps. Then, the pseudo-data generator 1 extracts the features of the transformed N samples (step S22).

[0072] Next, the pseudo-data generator 1 selects a template (step S23). Then, the pseudo-data generator 1 selects a query from a sample that is not a template and calculates a similarity matrix representing the similarity between the selected query and the template in terms of time steps (step S24). Next, the pseudo-data generator 1 extracts the unmatched time steps of the query and the unmatched time steps of the template (step S25). Then, the pseudo-data generator 1 associates the time step of the query with the highest similarity from the extracted unmatched time steps with the time step of the template (step S26).

[0073] Then, the pseudo-data generator 1 determines whether all time steps of the query have been matched (step S27). If there are unmatched time steps in the template (step S27; No), the pseudo-data generator 1 returns to step S25 and executes the processes of steps S25 and S26 again.

[0074] On the other hand, if it is determined that all time steps of the template have been matched (step S27; Yes), the pseudo-data generator 1 determines whether the matching of all samples has been completed (step S28). If the matching of all samples has not been completed (step S28; No), the pseudo-data generator 1 returns to step S14 and selects as a query any sample that has not been selected as either a template or a query.

[0075] Then, once the correspondence of all samples is complete (Step S28; Yes), the pseudo-data generation device 1 generates pseudo-data from the time-series data set reconstructed based on the correspondence (Step S29).

[0076] As described above, the pseudo-data generation device 1 in the second embodiment can reliably associate time steps of identical operation stages even when incorrect unit operations are included.

[0077] <Second Embodiment> Figure 13 is a block diagram of the information processing device 1X. The information processing device 1X includes an acquisition means 21X and a identification means 24X. The information processing device 1X can be the pseudo-data generation device 1 in the first or second embodiment.

[0078] The acquisition means 21X acquires a query containing data for each time step and a template to be compared with the query. The acquisition means 21X can be the feature extraction unit 21 in the first embodiment or the time series length adjustment unit 20 in the second embodiment.

[0079] When the identification means 24X identifies the correspondence between the time steps of a query and a template, it extracts the time steps of queries and templates that have not been matched, and then matches the time steps of the query and template that have the highest similarity among the extracted query time steps and template time steps, repeating this process. The identification means 24X can be the correspondence identification unit 24 in the first or second embodiment.

[0080] Figure 14 is an example of a flowchart showing the processing procedure of the information processing device 1X. First, the acquisition means 21X acquires queries that have data for each time step and templates to be compared with the queries (step S31). Next, the identification means 24X, when identifying the correspondence between the time steps of queries and templates, repeats the following process (step S32): (i) extract the time steps of queries and templates that are not yet associated, and (ii) associate the time steps of queries and templates that have the highest similarity between the extracted time steps of queries and templates.

[0081] According to the second embodiment, the information processing device 1X can accurately associate queries and templates on a time-step basis so that they correspond to the same operational stage.

[0082] In each of the embodiments described above, the program can be stored using various types of non-transitory computer-readable medium and supplied to a computer, such as a processor. Non-transitory computer-readable mediums include various types of tangible storage mediums. Examples of non-transitory computer-readable mediums include magnetic storage mediums (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage mediums (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program may also be supplied to the computer by various types of transient computer-readable mediums. Examples of transient computer-readable mediums include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable mediums can supply the program to the computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

[0083] Furthermore, some or all of the above embodiments may also be described as follows, but are not limited to the following. Also, some or all of the configurations described in Appendices 2 to 9, which are dependent on Appendice 1 above, may be similarly dependent on Appendices 10 and 11. Moreover, not limited to the devices, methods, and storage media described in the appendices, some or all of the configurations described in the appendices may also be dependent on methods, various hardware, software, various recording means for recording software (including storage media), or systems, without departing from the above embodiments.

[0084] [Note 1] A means for obtaining a query, which is data representing the operation for each time step, and a template to be compared with the query, When identifying the correspondence between the time steps of the aforementioned query and the aforementioned template, A correspondence identification means that repeatedly performs the process of extracting the time step of the query and the time step of the template that are not currently associated, and associating the query time step and the template time step that have the highest similarity among the extracted query time steps and template time steps, An information processing device having [Note 2] The information processing apparatus according to claim 1, further comprising: the query reconfigured to match the time step of the template based on the correspondence relationship described above; and generation means for generating pseudo-data that simulates the operation based on the template. [Note 3] A time series length adjustment means that adjusts the time lengths of multiple samples, which are data representing the operation at each time step, so that the number of time steps of the multiple samples are equal, The information processing apparatus according to claim 1, further comprising: selection means for selecting the query and the template from the plurality of samples after the time length has been adjusted. [Note 4] Each of the aforementioned samples is associated with a label indicating whether the operation was correct or incorrect. The information processing apparatus according to claim 3, wherein the selection means selects a sample corresponding to the label indicating that the operation is correct as the template. [Note 5] The system further includes a feature extraction means for extracting feature quantities for each time step from the query and the template, respectively. The information processing apparatus according to claim 1, wherein the correspondence identification means identifies the correspondence relationship based on the similarity calculated based on the feature quantity. [Note 6] The system further includes a similarity calculation means for calculating a similarity matrix representing the similarity of all combinations of the time step of the query and the time step of the template, The information processing apparatus according to claim 1, wherein the correspondence identification means identifies the correspondence relationship based on the similarity matrix. [Note 7] The aforementioned query and template are training data used in machine learning for a model that outputs information used to support decision-making. The information processing apparatus according to claim 2, wherein the pseudo-data is the training data generated by data augmentation. [Note 8] The information processing apparatus according to claim 2, wherein the generation means generates the pseudo-data based on any pair selected from the reconstructed one or more queries and the template. [Note 9] A method by which a computer performs an action. The system obtains a query, which is data representing the actions at each time step, and a template to compare with the query. When identifying the correspondence between the time steps of the aforementioned query and the aforementioned template, Extract the time steps of the query and the template that are not currently associated, and associate the time steps of the query and the template that have the highest similarity among the extracted time steps of the query and the template. Repeat this process. method. [Note 10] The system obtains a query, which is data representing the actions at each time step, and a template to compare with the query. When identifying the correspondence between the time steps of the aforementioned query and the aforementioned template, The process involves extracting the time steps of the query and the template that are not currently associated, and then associating the query time step with the template time step that has the highest similarity among the extracted query time steps with the template time steps, and repeating this process. A program that instructs a computer to perform a process. [Note 11] A storage medium containing the program described in Appendix 10.

[0085] Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above embodiments. Various modifications to the structure and details of the present invention can be made as understood by those skilled in the art within the scope of the present invention. That is, the present invention includes the full disclosure, including the claims, and of course, various modifications and alterations that those skilled in the art could make in accordance with the technical idea. Furthermore, the above-mentioned patent and non-patent disclosures are incorporated herein by reference. [Explanation of Symbols]

[0086] 1. Pseudo-data generation device 1X Information Processing Device 2 Storage device 3 Display device 4 Input devices 11 processors 12 memory 13 Interfaces 100 Pseudo-Data Generation Systems

Claims

1. A means for obtaining a query, which is data representing the operation for each time step, and a template to be compared with the query, When identifying the correspondence between the time steps of the aforementioned query and the aforementioned template, A correspondence identification means that repeatedly performs the process of extracting the time step of the query and the time step of the template that are not currently associated, and associating the query time step and the template time step that have the highest similarity among the extracted query time steps and template time steps, An information processing device having

2. The information processing apparatus according to claim 1, further comprising a generation means for generating pseudo-data that simulates the operation based on the query reconfigured to match the time step of the template based on the correspondence relationship and the template.

3. A time series length adjustment means that adjusts the time lengths of multiple samples, which are data representing the operation at each time step, so that the number of time steps of the multiple samples are equal, The information processing apparatus according to claim 1, further comprising: selection means for selecting the query and the template from the plurality of samples after the time length has been adjusted.

4. Each of the aforementioned samples is associated with a label indicating whether the operation was correct or incorrect. The information processing apparatus according to claim 3, wherein the selection means selects a sample corresponding to the label indicating that the operation is correct as the template.

5. The system further includes a feature extraction means for extracting feature quantities for each time step from the query and the template, respectively. The information processing apparatus according to claim 1, wherein the correspondence identification means identifies the correspondence relationship based on the similarity calculated based on the feature quantity.

6. The system further includes a similarity calculation means for calculating a similarity matrix representing the similarity of all combinations of the time step of the query and the time step of the template, The information processing apparatus according to claim 1, wherein the correspondence identification means identifies the correspondence relationship based on the similarity matrix.

7. The aforementioned query and template are training data used in machine learning for a model that outputs information used to support decision-making. The information processing apparatus according to claim 2, wherein the pseudo-data is the training data generated by data augmentation.

8. The information processing apparatus according to claim 2, wherein the generation means generates the pseudo-data based on any pair selected from the reconstructed one or more queries and the template.

9. A method by which a computer performs an action. The system obtains a query, which is data representing the actions at each time step, and a template to compare with the query. When identifying the correspondence between the time steps of the aforementioned query and the aforementioned template, Extract the time steps of the query and the template that are not currently associated, and associate the time steps of the query and the template that have the highest similarity among the extracted time steps of the query and the template. Repeat this process. method.

10. The system obtains a query, which is data representing the actions at each time step, and a template to compare with the query. When identifying the correspondence between the time steps of the aforementioned query and the aforementioned template, The process involves extracting the time steps of the query and the template that are not currently associated, and then associating the query time step with the template time step that has the highest similarity among the extracted query time steps with the template time steps, and repeating this process. A program that instructs a computer to perform a process.

Images