Information processing method, information processing device, and information processing program
The method generates novel data by combining feature quantities not in a connection relationship using encoders and generators, addressing data preparation challenges and enhancing data continuity.
Patent Information
- Application Number
- JP2022519894
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-05-04
- Filing Date
- 2020-08-19
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2040-08-19
AI Technical Summary
Existing methods face challenges in preparing data with the same amount as the desired output, leading to a significant data preparation burden.
An information processing method and device that utilizes a learned model to generate new data from multiple feature quantities not in a connection relationship, using a combination of encoders and generators to create novel data while reducing unnatural discontinuities.
Reduces data preparation burden and generates novel data with improved continuity by combining feature quantities that retain original characteristics, overcoming the limitations of direct combination methods.
Smart Images

Figure 0007582307000001 
Figure 0007582307000002 
Figure 0007582307000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an information processing method, an information processing device, and an information processing program. [Background technology]
[0002] For example, Patent Document 1 discloses a technique for generating a different song by changing the feature amounts of each piece of partial data of one song. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] International Publication No. 2020 / 080268 Summary of the Invention [Problem to be solved by the invention]
[0004] It may be difficult to prepare source data having a volume comparable to that of the data to be generated.
[0005] One aspect of the present disclosure proposes an information processing method, an information processing device, and an information processing program that can reduce the burden of data preparation. [Means for solving the problem]
[0006] An information processing method according to one aspect of the present disclosure uses a plurality of features that are not in a combined relationship and a trained model to generate new data obtained from a plurality of modified features, and when the plurality of features are input, the trained model outputs the plurality of modified features.
[0007] An information processing device according to one aspect of the present disclosure includes a generation unit that uses a plurality of features that are not in a combined relationship and a trained model to generate new data obtained from a plurality of modified features, and the trained model outputs the plurality of modified features when the plurality of features are input.
[0008] An information processing program according to one aspect of the present disclosure is an information processing program for causing a computer to function, and causes the computer to execute the following: generating new data obtained from a plurality of modified features using a plurality of features that are not in a combined relationship and a trained model; when the plurality of features are input, the trained model outputs the modified plurality of features. [Brief description of the drawings]
[0009] [Figure 1] 1 is a diagram illustrating an example of a schematic configuration of an information processing device according to an embodiment. [Diagram 2] FIG. 13 is a diagram showing an example of an input screen. [Diagram 3] FIG. 13 is a diagram showing an example of an output screen. [Figure 4] FIG. 2 is a diagram illustrating an example of functional blocks of an information processing device. [Diagram 5] FIG. 11 is a diagram showing an example of a plurality of partial data. [Figure 6] FIG. 1 is a diagram illustrating an example of a schematic configuration of a trained model. [Figure 7] FIG. 13 is a diagram illustrating an example of pre-learning of an encoder. [Figure 8] FIG. 13 is a diagram illustrating an example of pre-training of a classifier and a generator. [Figure 9] FIG. 13 is a diagram showing an example of new data. [Figure 10] 4 is a flowchart showing an example of a process (information processing method) executed in the information processing device. [Figure 11] 1 is a flowchart illustrating an example of generating a trained model. [Figure 12] FIG. 1 is a diagram illustrating an example of a schematic configuration of a trained model. [Figure 13] FIG. 13 is a diagram showing an example of an input screen. [Figure 14] FIG. 11 is a diagram showing an example of a plurality of partial data. [Figure 15] FIG. 13 is a diagram showing an example of new data. [Figure 16] FIG. 13 is a diagram showing an example of an output screen. [Figure 17] FIG. 1 is a diagram illustrating an example of a schematic configuration of a trained model. [Figure 18] FIG. 13 is a diagram showing an example of an input screen. [Figure 19] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.
[0011] The present disclosure will be described in the following order. 1. Embodiment 2. Modifications 3. Effects
[0012] 1. Embodiment The following mainly describes an information processing device that can be used in the information processing method according to the embodiment. Examples of information to be processed include music data, language data, DNA sequence data, image data, etc. Examples of music data include music sequences such as symbolic music (symbol series), audio, etc. Examples of language include documents, poems, programming languages, etc.
[0013] The information processing device generates new data from a plurality of feature quantities that are not in a combined relationship. A plurality of feature quantities that are not in a combined relationship refers to, for example, a relationship in which unnaturalness such as discontinuity occurs when those feature quantities are combined as they are (for example, arranged in a row and connected in a row). An example of a plurality of feature quantities that are not in a combined relationship is a feature quantity that is not in a combined relationship between data (hereinafter referred to as "partial data") from which each feature quantity is extracted. For example, a plurality of partial data that exist consecutively in the same data can be said to have a combined relationship. A plurality of partial data that exist in the same data such that there is no continuity or that unnaturalness occurs when the data is connected as it is can be said to have no combined relationship. A plurality of partial data that exist in different data can be said to have no combined relationship. The data length of the new data to be generated is longer than the data length of the partial data. In other words, the partial data has a data length shorter than the data length of the new data. Another example of a feature quantity is a feature quantity from which no partial data exists as an extraction source, and is a feature quantity sampled from a standard normal distribution N(0,I) as will be described later with reference to FIG. 16.
[0014] In many cases, preparing multiple features that are not linked together is easier than preparing features for data that has the same amount of data as new data, for example. This reduces the burden of data preparation.
[0015] The new data is data obtained from the multiple feature amounts that have been changed. The multiple feature amounts that have been changed can be novel feature amounts that cannot be obtained by simply combining the multiple feature amounts before the change while retaining the characteristics of each of the multiple feature amounts before the change. Unnaturalness such as discontinuity that may occur when the multiple feature amounts before the change are combined as they are can also be reduced.
[0016] For example, if partial data from which the feature values were extracted exists for each of the multiple feature quantities, the new data will be data in which multiple partial data that have been modified to enhance fusion properties have been combined (associative fusion of series). The new data generated by this type of associative fusion will retain the characteristics of the partial data before the modification, and will be novel data that cannot be obtained by simply combining the partial data before the modification. Unnaturalness such as discontinuity that can occur when each partial data before the modification is combined as is can also be reduced. For example, by using fragments of ideas as partial data, it may be possible to generate new ideas from the fragments of ideas.
[0017] Below, examples of feature amounts from which partial data exists as an extraction source will be described with reference to Figures 1 to 16. Then, examples of feature amounts from which partial data does not exist will be described with reference to Figures 17 and 18. A music sequence will be used as an example of partial data.
[0018] FIG. 1 is a diagram showing an example of the appearance of an information processing device according to an embodiment. The information processing device 1 is realized, for example, by executing a predetermined program (software) in a general-purpose computer. In the example shown in FIG. 1, the information processing device 1 is a laptop used by a user U. The display screen of the information processing device 1 is referred to as a display screen 1a and illustrated. The information processing device 1 can be realized by various devices other than a laptop, such as a PC or a smartphone.
[0019] FIG. 2 is a diagram showing an example of an input screen. In this example, the user U inputs two music sequences. In the item "Sequence 1," the user U refers to a folder or the like and selects a file. The music sequence shown in the selected file is visualized and displayed. Similarly, in the item "Sequence 2," the user U refers to a folder or the like and selects a file. The music sequence shown in the selected file is visualized and displayed.
[0020] Furthermore, in the item "number of iterations", the user U sets the number of iterations. The number of iterations is related to the degree of fusion between the two music sequences, and will be described in detail later with reference to FIG. 6. In this example, the number of iterations can be set automatically or specified. Here, it is assumed that the number of iterations is set to 3.
[0021] FIG. 3 is a diagram showing an example of an output screen. According to the above-mentioned iteration number 3, three new sequences corresponding to iteration numbers 1 to 3 are visualized and displayed. The new sequence can be a novel sequence that cannot be obtained by simply combining sequence 1 and sequence 2 as they are. Unnaturalness such as discontinuity that can occur when sequence 1 and sequence 2 are combined as they are can also be reduced.
[0022] 1 to 3 described above are merely examples of the external appearance and input / output screen configuration of the information processing device 1, and any configuration that can accommodate various input / output modes, including those described below, may be adopted.
[0023] 4 is a diagram showing an example of functional blocks of an information processing device 1. The information processing device 1 includes an input unit 10, a storage unit 20, a generation unit 30, and an output unit 40.
[0024] A plurality of partial data are input to the input unit 10. As mentioned at the beginning, the plurality of partial data are a plurality of partial data that are not in a combined relationship. An example of the data will be described with reference to FIG.
[0025] 5 is a diagram showing an example of a plurality of partial data. In this example, the plurality of partial data are two musical sequences represented by time and pitch value (MIDI pitch). Time (bars) indicates a bar number. The data length of each partial data (four bars in this example) is shorter than the data of the new data to be generated (eight bars in the example of FIG. 9 described later).
[0026] Returning to Figure 4, the input unit 10 also receives data indicating the number of iterations, which was previously described with reference to Figure 3, data indicating the presence or absence of a style (see Figure 13, etc.), which will be described later with reference to Figure 13, and setting data for feature S, which will be described with reference to Figure 16.
[0027] The storage unit 20 stores various information used in the information processing device 1. Among these, a trained model 21 and an information processing program 22 are exemplified. The trained model 21 will be described later. The information processing program 22 is a program for realizing processing executed in the information processing device 1.
[0028] The generating unit 30 generates new data using a plurality of feature amounts and the trained model 21 according to the input result of the input unit 10. Each of the plurality of feature amounts is a feature amount of each of a plurality of partial data, and is a feature amount that is not in a combined relationship. The trained model 21 will be described with reference to Figs. 6 to 8.
[0029] FIG. 6 is a diagram illustrating an example of a schematic configuration of a trained model. The trained model 21 includes an encoder qθ and a generator G Ψ and a decoder pθ.
[0030] The encoder qθ is the partial data X L and partial data X R The feature quantity Z L and feature quantity Z R The encoder qθ extracts the partial data X L and partial data X R Let Z be the feature L and feature quantity Z R It can also be said to be a transformation function that transforms the feature quantity Z L and feature quantity Z R indicates a position (point) in a multidimensional space. Such a multidimensional space is also called a latent space or a latent feature space, and will be referred to as a latent space below. L and feature quantity Z Rmay be a vector. The encoder qθ is generated by pre-training with the decoder pθ. This will be described with reference to FIG.
[0031] Fig. 7 is a diagram showing an example of pre-learning of an encoder and a decoder. It should be noted that the encoder qθ and the decoder pθ shown in Fig. 7 are (provisional) encoders and decoders before learning is completed, and are different from the learned encoder and decoder shown in Fig. 6 described above.
[0032] 7, the encoder qθ and the decoder pθ are both neural networks, and are generated by learning the feature value Z of the partial data X, i.e., the position in the latent space, using a variational auto encoder (VAE). The partial data X and the feature value Z here are the same as those of the partial data X described above with reference to FIG. L and partial data X R And feature quantity Z L and feature quantity Z R The encoder qθ and the decoder pθ of the VAE are implemented using, for example, a recurrent neural network (RNN). The encoder qθ extracts a feature quantity Z of the partial data X. The decoder pθ reconstructs the feature quantity Z as the partial data X′.
[0033] Figure 7 shows the loss function L rec and loss L pri The loss L rec is the reconstruction error used to make the partial data X' closer to the partial data X. pri is the prior error used to make the distribution of the features extracted by the encoder qθ, i.e., the distribution of positions in a multidimensional space, closer to the standard normal distribution N(0,I).
[0034] Returning to Figure 6, generator G Ψ is the feature value Z L and feature quantity Z R When the input is L and feature quantity Z'R Generate (output) the feature Z' L and feature quantity Z' R is the modified feature Z L and feature quantity Z R Note that changing the feature is equivalent to moving the position of the feature in the latent space. Ψ is generated by pre-training with a classifier, as will be described with reference to FIG.
[0035] FIG. 8 is a diagram illustrating an example of pre-training of a classifier and a generator. Ψ is a (temporary) generator before the learning is completed, and is the trained generator G shown in Figure 6 above. Ψ Please note that this is different from the above.
[0036] Referring to FIG. 8, the generator G Ψ is the feature value Z L and feature quantity Z R The input is the feature Z' L and feature quantity Z' R The discriminator D Ψ is the generator G Ψ The feature Z' generated by L and feature quantity Z' R Identify the joint features (joint vectors) of the generator G Ψ and the discriminator D Ψ is implemented using Generative Adversarial Networks (GANs). Example architectures include Multi-Layer Perceptron, RNN, and Transformer.
[0037] Figure 8 shows the loss function L p dis , loss L N dis , loss L p gen and loss L sim The loss L p dis is the discriminator D Ψ is the loss function for positive examples of the loss L Ndis is the discriminator D Ψ is the loss function for negative examples of the loss L p gen is the generator G Ψ is a loss function for the loss L sim is the feature Z' L and feature quantity Z' R Let Z be the feature quantity. L and feature quantity Z R These loss functions are used to approximate the feature Z' L Feature quantity Z L The correlation with feature Z' remains. L and feature value Z R The similarity with feature Z' is also guaranteed. R Feature quantity Z R The correlation with feature Z' remains. R and feature value Z R This guarantees the similarity between the partial data X and the new data. L and partial data X R The characteristics remain.
[0038] Discriminator D Ψ We will further discuss the training of the classifier D Ψ In the learning, gradient descent is used to distinguish positive and negative examples. Cross entropy may be used as the error function. Positive and negative examples will be explained in turn.
[0039] An example of a positive example is a combined feature (combined vector) of the features of the first half (e.g., the first 4 bars) and the second half (e.g., the second 4 bars) of data that has the same data length as the entire data length of two partial data (e.g., 8 bars). Such combined features are features obtained by combining the features of partial data that are in a combined relationship, i.e., features that are in a combined relationship, and are appropriate for generation purposes.
[0040] Four examples are given as negative examples. The first example is a combined feature of the features of two partial data samples sampled randomly. The second example is a combined feature of two features sampled from a standard normal distribution. These combined features are considered negative examples because they are simply a direct combination of features from two partial data sets that were not in a combined relationship, i.e., features that are not in a combined relationship, which goes against the purpose of generation.
[0041] In the third example, the features of the two partial data samples taken in the first example are generated by a provisional generator G Ψ The fourth example is a combination feature of features generated by inputting the features of the two partial data sampled in the second example above into a provisional generator G Ψ These combined features are used as negative examples for the adversarial learning in GANs.
[0042] Generator G Ψ We further describe the training of the generator G Ψ In the learning, the combination feature in the first example and the combination feature in the second example are input to the generator G Ψ Input to the generator G Ψ The output of Ψ and the discriminator D Ψ Calculate the cross-entropy error function that treats the output of feature Z as a positive example. In addition, calculate the similarity error function between feature Z and feature Z', and the feature Z R and feature Z´ R An example of the similarity error function is the squared error of both features. However, other similarity error functions may be used. L and feature Z´ L Similarity error function with feature quantity Z R and feature Z´ R The sum of the three error functions is the final error function, and the generator G Ψ The parameters of are updated using the gradient method.
[0043] Returning to Figure 6, the decoder pθ calculates the feature Z' L and feature quantity Z' R Let X' be the partial data. L and partial data X' R The decoder pθ reconstructs the feature Z′ L and feature quantity Z' R Partial data X' L and partial data X' R It can also be said to be a transformation function that transforms
[0044] The generating unit 30 generates the partial data X' L and partial data X' R New data is generated by combining
[0045] In FIG. 6, “optionally iterate” means that the generation unit 30 Ψ The output of the generator G Ψ We show that it is OK to return the input of the generator G Ψ The features can be further modified by repeating the generation by Ψ Let the number of times that the feature vector Z is generated be the iterate number. As the iterate number increases, L and feature quantity Z R Feature value Z´ for L and feature quantity Z' R The degree of fusion of the new data becomes larger, and the partial data X´ L and partial data X' R Therefore, the number of different new data is generated equal to the number of iterations.
[0046] 2, the output section 40 outputs the new data generated by the generation section 30. An example of the data will be described with reference to FIG.
[0047] FIG. 9 is a diagram showing an example of new data. In this example, three new data corresponding to three different iterate numbers are illustrated. Each new data has a data length of eight bars, that is, the same data length as the entire data length of the multiple partial data input (FIG. 5). The music sequence of the first four bars is a music sequence in which the original music sequence (left side of FIG. 5) has been changed. The music sequence of the last four bars is a music sequence in which the original music sequence (right side of FIG. 5) has been changed. The iterate numbers of the three new data displayed in a vertical line increase from top to bottom, and for example, the degree of change from the original music sequence also increases, enhancing the degree of fusion.
[0048] The output section 40 may display the new data in association with the iterate number, as previously described with reference to Fig. 3. This allows the iterate number to be confirmed, and also allows new data to be generated using a different iterate number.
[0049] FIG. 10 is a flowchart showing an example of a process (information processing method) executed in the information processing device.
[0050] In step S1, a plurality of partial data are input. For example, as described above with reference to FIG. 2, two music sequences are input and accepted by the input unit 10. The two input music sequences are divided into the partial data X L and partial data X R It should be noted that other data such as the number of iterations may also be input to the input unit 10.
[0051] In step S2, a feature amount is extracted. For example, the generation unit 30 extracts the partial data X L and partial data X R The feature quantity Z L and feature quantity Z R Extract.
[0052] In step S3, the modified feature is generated. For example, the generator 30 generates the modified feature from the generator G of the trained model 21. Ψ Using the feature quantity Z extracted in step S2 L and feature quantity Z R The feature Z' changed from L and feature quantity Z' R Generate.
[0053] In step S4, the changed feature quantity is reconstructed as partial data. For example, the generation unit 30 reconstructs the feature quantity Z′ generated in the previous step S3 using the decoder pθ of the trained model 21. L and feature quantity Z' R Let X' be the partial data. L and partial data X' R Reconstruct it as:
[0054] In step S5, new data is generated. For example, the generating unit 30 generates the partial data X′ obtained in the previous step S4. L and partial data X' R New data is generated by combining
[0055] In step S6, the new data is output. For example, the output unit 40 outputs the new data as described above with reference to Fig. 3 and Fig. 9. An output mode such as that described later with reference to Fig. 16 etc. is also possible.
[0056] After the process of step S6 is completed, the process of the flowchart ends. For example, new data can be generated in this manner.
[0057] An example of a procedure for generating the trained model 21 will be described with reference to FIG.
[0058] FIG. 11 is a flowchart showing an example of generating a trained model.
[0059] In step S11, learning of the VAE is performed using the partial data. For example, as described above with reference to FIG. 7, learning is performed using the partial data X to generate an encoder qθ and a decoder pθ.
[0060] In step S12, the value of a variable i is set to 1. The variable i is used to repeat the processes of steps S13 and S14 described below a determined number of iterations. The number of iterations may be set appropriately within the range of expected iteration numbers.
[0061] In step S13, one batch of samples is acquired. The samples are, for example, mini-batch samples, and are first processed by a classifier D Ψ and generator G Ψ 8 is an example of training data used for learning. For example, when one batch corresponds to 100 pieces of data, 100 groups of positive examples and negative examples as described above with reference to FIG.
[0062] In step S14, the classifier and generator are trained. That is, a classifier D is trained using the VAE trained in the previous step S11 and one batch of samples acquired in the previous step S12. Ψ and generator G Ψ The features sampled from the standard normal distribution N(0,I) may also be used for learning. This allows the generator G to be used for features that are difficult to obtain from data created by humans. Ψ Such features are generated by the generator G Ψ It is highly likely that such data will be input into the database, and in this sense it is particularly useful for learning purposes.
[0063] In step S15, it is determined whether the variable i is equal to or greater than the determined number of iterations. If the variable i is equal to or greater than the determined number of iterations (step S15: Yes), the process of the flowchart ends. If not (step S15: No), the process proceeds to step S16.
[0064] In step S16, the value of the variable i is incremented by 1, and the process returns to step S13.
[0065] For example, a trained model 21 can be generated in the manner described above.
[0066] An embodiment of the present disclosure has been described above. The present disclosure is not limited to the above embodiment. Some modified examples will be described.
[0067] 2. Modifications In the above embodiment, the feature quantity Z L and feature quantity Z R An example was explained in which only feature quantity Z L and feature quantity Z R In addition, the feature value Z L and feature quantity Z R An additional feature that gives the directionality of the change may also be used. This will be described with reference to FIGS.
[0068] FIG. 12 is a diagram illustrating an example of a schematic configuration of a trained model. A trained model 21A illustrated in FIG. 12 has a feature quantity Z L and feature quantity Z R In addition, the feature S is also generated by the generator G Ψ The difference is that the feature S is input to a multidimensional distribution u(0, 1) ds is a feature (vector) sampled from feature Z L and feature quantity Z R Similarly, it indicates a position (point) in the latent space. Note that the multidimensional distribution u(0, 1) ds ds indicates the number of dimensions.
[0069] Generator G Ψ As a result of the input of feature S, feature Z' L and feature quantity Z' R A change appears in the partial data X' L and partial data X' R, and changes will also appear in the new data that is the combined data of these. For example, when the position of feature S is moved in a certain direction in the latent space of feature S, a certain tendency of change that appears in the new data will become apparent or latent. When feature S is moved in another direction in the latent space, a different tendency of change that appears in the new data will become apparent or latent. Such feature S can be said to be a feature (a vector in style space) that gives variation to the style of the new data to be generated.
[0070] The training of the trained model 21A is compared with the generation of the trained model 21, and the feature quantity Z L and feature quantity Z R The feature S is also generated by the generator G Ψ That is, during learning, the multidimensional distribution u(0, 1) is input to the combined feature in the first example described above. ds After combining the sampled features from Ψ In addition, the multidimensional distribution u(0, 1) is input to the joint feature in the second example described above. ds After combining the sampled features from Ψ In the flowchart shown in FIG. 11 described above, step S14 is changed to conform to the learning of the trained model 21A described above. The trained model 21A thus generated is L , feature value Z R When feature S is input, feature Z' L and feature quantity Z' R Output.
[0071] The procedure for generating new data by the generation unit 30 is also changed so as to be compatible with the use of the trained model 21A. In the flowchart shown in FIG. 10 described above, in step S3, the generation unit 30 generates the generator G Ψ , feature value Z L and feature quantity Z R In addition to the feature S, we also use the feature Z' L and feature quantity Z' R Generate.
[0072] An example of input and output when the trained model 21A is used will be described with reference to FIGS.
[0073] FIG. 13 is a diagram showing an example of an input screen. Compared to FIG. 2 described above, an item "style" has been added. The user U selects whether or not there is a style. As described above, data indicating whether or not there is a style is also input to the input unit 10 (FIG. 4). When "with style" is selected, for example, a predetermined number of the above-mentioned feature amounts S are used to generate new data. In this example, it is assumed that nine feature amounts S are used.
[0074] 14 is a diagram showing an example of a plurality of partial data. As in the previous examples, two music sequences are shown as examples. The data length of each music sequence is four bars.
[0075] FIG. 15 is a diagram showing an example of new data. Nine new data corresponding to different feature amounts S are illustrated. Each new data is a musical sequence having a data length of eight bars. This data length of eight bars is the total data length of the data lengths of the two input musical sequences. The musical sequence of the first four bars is a musical sequence obtained by modifying the original musical sequence (left side of FIG. 14). The musical sequence of the last four bars is a musical sequence obtained by modifying the original musical sequence (right side of FIG. 14).
[0076] When the trained model 21A is used, the output unit 40 (FIG. 4) may display new data in association with the feature S. This allows the user U to know the relationship between the change appearing in the new data and the feature S. The output unit 40 may display new data in association with the feature S in a manner that allows the user U to specify the feature S. This allows the user U to generate new data using a feature S other than the feature S of the displayed new data. An example of the output screen will be described with reference to FIG. 16.
[0077] FIG. 16 is a diagram showing an example of an output screen. Nine new data are displayed in association with the feature S, more specifically, the position of the feature S on a two-dimensional plane in the latent space. Labels AAA and BBB are displayed on both ends of an arrow extending in the left-right direction. Label AAA indicates the tendency of change that appears in the new data when the feature S is moved leftward on the two-dimensional plane. Examples of the contents of the label are the atmosphere of the music (bright, lonely, etc.), chord progression, etc. The contents of such labels are obtained, for example, by analyzing each new data. The analysis may be performed by the output unit 40 or the generation unit 30.
[0078] Regarding other labels, the label BBB indicates the tendency of change when the feature S is moved to the right on the two-dimensional plane. Similarly, the labels CCC and DDD are displayed at both ends of the arrows extending in the up and down directions. The label CCC indicates the tendency of change when the feature S is moved downward on the two-dimensional plane. The label DDD indicates the tendency of change when the feature S is moved upward on the two-dimensional plane.
[0079] If new data corresponding to the feature S on another two-dimensional plane in the latent space is also generated, the display may be switched to one in which the new data is associated with the feature S.
[0080] In the example shown in Fig. 16, grid lines GR that divide a two-dimensional plane are displayed. A user U can set a feature amount S corresponding to a position by selecting a portion divided by the grid lines GR with a pointer P. The setting data of this feature amount S is also input to the input unit 10 (Fig. 4). As a result, new data can be generated and output using the set feature amount S.
[0081] New data may be associated with both the features S and the number of iterates. In this case, the association of both factors may be displayed at once. For example, one factor may be associated with the screen plane direction and the other factor may be associated with the screen depth direction.
[0082] In the above embodiment, the partial data X L and partial data X R The features of the generator G Ψ However, the partial data X L and / or partial data X R Instead of , features sampled from a standard normal distribution may be used, as will be explained with reference to FIG.
[0083] 17 is a diagram illustrating an example of a schematic configuration of a trained model. Compared with the trained model 21A (FIG. 12), the trained model 21B illustrated in FIG. 17 has partial data X L Feature and partial data X R The difference is that it is possible to use a feature sampled from a standard normal distribution N(0,I) instead of at least one of the features.
[0084] The feature sampled from the standard normal distribution N(0, I) is called feature Z. L and / or feature quantity Z R By using it as the partial data X L and / or partial data X R Even if no input is made, new data can be generated. As described above in step S14 of FIG. 11, the trained model 21B is trained by using the feature sampled from the standard normal distribution N(0, I) and the generator G Ψ This is done by entering:
[0085] The procedure for generating new data by the generating unit 30 is modified to suit the use of the trained model 21B. In the flowchart shown in FIG. 10 described above, in step S2, the partial data X L and / partial data X R Instead of extracting features from N(0,I), features can be sampled from a standard normal distribution N(0,I).
[0086] An example of an input screen when the trained model 21B is used will be described with reference to FIG.
[0087] FIG. 18 is a diagram showing an example of an input screen. Compared to FIG. 2 described above, in the items "Sequence 1" and "Sequence 2," in addition to selecting a file, it is also possible to select "no file." These selection results are also input to the input unit 10 (FIG. 4). When "no file" is selected in the item "Sequence 1," the feature quantity Z L The feature sampled from the standard normal distribution N(0,I) is used as the feature Z R The feature values sampled from the standard normal distribution N(0,I) are used as the vectors. This eliminates the need to prepare partial data.
[0088] Finally, the hardware configuration of the information processing device 1 will be described with reference to FIG.
[0089] 19 is a diagram showing an example of a hardware configuration of an information processing device. In this example, the information processing device 1 is realized by a computer 1000. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.
[0090] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processes corresponding to the various programs.
[0091] The ROM 1300 stores boot programs such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000.
[0092] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records an information processing program according to the present disclosure, which is an example of program data 1450.
[0093] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.
[0094] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading a program or the like recorded on a predetermined recording medium. The medium may be, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.
[0095] For example, when the computer 1000 functions as the information processing device 1, the CPU 1100 of the computer 1000 executes an information processing program loaded onto the RAM 1200 to realize functions of the generating unit 30, etc. Also, the HDD 1400 stores a program according to the present disclosure (the information processing program 22 of the storage unit 20) and data in the storage unit 20. Note that the CPU 1100 reads and executes the program data 1450 from the HDD 1400, but as another example, these programs may be obtained from other devices via an external network 1550.
[0096] In the above embodiment, the multiple feature quantities are two feature quantities Z L and feature quantity Z R However, three or more feature amounts may be used. The number of partial data items may also be three or more.
[0097] In the above embodiment, in the form (FIG. 17) in which the feature quantity sampled from the standard normal distribution N(0,I) is selectively used, an example in which the feature quantity S is also used has been described. However, in the form (FIG. 6, etc.) in which the feature quantity S is not used, the feature quantity sampled from the standard normal distribution N(0,I) may be selectively used.
[0098] In the above embodiment, an example of generating a musical sequence has been described, but in addition to this, any other data may be generated, such as music such as audio, documents, poems, languages such as programming languages, DNA sequences, images, etc., as mentioned at the beginning.
[0099] Some of the functions of the information processing device 1 may be realized outside the information processing device 1 (for example, an external server). In this case, the information processing device 1 may include some or all of the functions of the storage unit 20 and the generation unit 30 in the external server. The information processing device 1 communicates with the external server to similarly realize the processes of the information processing device 1 described above.
[0100] 3. Effects The above-described embodiment can be specified, for example, as follows. As described with reference to FIG. 4, FIG. 6, FIG. 10, etc., the information processing method is a method for processing a plurality of feature quantities (feature quantities Z L and feature quantity Z R ) and the trained model 21, the modified multiple features (feature Z' L and feature quantity Z' R ) is used to generate new data (steps S2 to S5). Ψ ) takes multiple input features and outputs multiple modified features.
[0101] According to the above information processing method, new data is generated using a plurality of features that are not in a combined relationship and a trained model. Preparation of each of the plurality of features that are not in a combined relationship is often easier than preparation of features of data having a data volume similar to that of the new data. Therefore, the burden of data preparation can be reduced. In addition, the changed plurality of features can be novel features that cannot be obtained by simply combining the respective features before the change while retaining the characteristics of each of the plurality of features before the change. Unnaturalness such as discontinuity that may occur when the respective features before the change are combined as they are can also be reduced.
[0102] As described with reference to FIGS. 5, 6, and 9, the plurality of feature quantities are stored in the partial data (partial data X L and / or partial data X R ) feature (feature Z L and / or feature quantity Z R The plurality of feature quantities may include partial data (partial data X L and partial data X R ) feature (feature Z L and feature quantity Z R), in which case the new data may have the same data length as the total data length of the partial data corresponding to each of the multiple feature amounts. This reduces the burden of preparing the partial data. The new data generated retains the characteristics of the partial data and can be novel data that cannot be obtained by simply combining the partial data before the change. Unnaturalness such as discontinuity that can occur when each partial data is combined as is can also be reduced.
[0103] As described with reference to FIG. 6 etc., the generator G Ψ ) output result and the trained model 21 (generator G Ψ ) to generate new data obtained from the further modified features. This allows the degree of fusion in the new data, for example, the partial data X' L and partial data X' R This can further enhance the fusion of the two.
[0104] As described with reference to FIG. 3 and FIG. 9, the generated new data and the trained model 21 (generator G Ψ ) and the number of times (iteration number) that multiple feature quantities were changed by the iteration. This allows the user to check the iteration number and, for example, generate new data using a different iteration number.
[0105] As described with reference to FIG. 12 etc., multiple feature quantities (feature quantities Z L and feature quantity Z R The trained model 21A used in this case may generate new data by using an additional feature (feature S) that gives a direction of change of the plurality of features (feature Z' L and feature quantity Z' R ), which allows for variation in the style of the new data generated.
[0106] As described with reference to Fig. 16 etc., the generated new data may be displayed in association with the directionality of change given by the additional feature (feature S). Also, the additional feature other than the additional feature corresponding to the generated new data may be displayed in a specifiable manner. This allows further generation of new data using the additional feature.
[0107] As described with reference to FIG. 17 etc., multiple feature quantities (feature quantities Z L and feature quantity Z R ) may include features sampled from a standard normal distribution N(0,I). The features sampled from the standard normal distribution N(0,I) may be divided into partial data (partial data X L and partial data X R ) instead of the features extracted from the partial data X L and partial data X R Even if no data is entered, new data can be generated. This further reduces the burden of data preparation.
[0108] The information processing device 1 described with reference to FIG. 4 and the like is also one aspect of the present disclosure. That is, the information processing device 1 processes a plurality of feature amounts (feature amounts Z L and feature quantity Z R ) and the trained model 21, the modified multiple features (feature Z' L and feature quantity Z' R ) and a generating unit 30 that generates new data obtained from the feature quantities (feature quantities Z L and feature quantity Z R ) and the trained model 21, the modified multiple features (feature Z' L and feature quantity Z' R) to generate new data obtained from the information processing device 1 and the information processing program 22. As with the information processing method described above, the information processing device 1 and the information processing program 22 also have the effect of reducing the burden of data preparation.
[0109] Note that the effects described in this disclosure are merely examples and are not limited to the disclosed contents. Other effects may also be obtained.
[0110] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present disclosure. In addition, components of different embodiments and modifications may be appropriately combined.
[0111] The present technology can also be configured as follows. (1) generating new data obtained from the changed plurality of features using a plurality of features not in a combined relationship and a trained model; When the plurality of features are input, the trained model outputs the plurality of features that have been changed. Information processing methods. (2) the plurality of feature amounts include feature amounts extracted from partial data having a data length shorter than a data length of the new data; The information processing method described in (1). (3) each of the plurality of feature amounts is a feature amount extracted from partial data having a data length shorter than a data length of the new data; the new data has a data length equal to a total data length of the partial data corresponding to each of the plurality of feature amounts; 2. An information processing method according to claim 1 or 2. (4) Using the output result of the trained model and the trained model, generate the new data obtained from the plurality of further modified features. The information processing method according to any one of (1) to (3). (5) The generated new data is displayed in association with the number of times the plurality of feature quantities have been changed by the trained model. (4) An information processing method according to the above. (6) generating the new data using an additional feature that provides a direction of change of the plurality of feature values; When the plurality of features and the additional features are input, the trained model outputs the plurality of features that have been changed. The information processing method according to any one of (1) to (5). (7) displaying the generated new data in association with the directionality of the change given by the additional feature amount; (6) An information processing method according to the above. (8) displaying the generated new data, and displaying the additional feature amount other than the additional feature amount corresponding to the displayed new data in a specifiable manner; The information processing method according to (6) or (7). (9) the plurality of features include features sampled from a standard normal distribution, The information processing method according to any one of (1) to (8). (10) The feature sampled from the standard normal distribution is used instead of the feature extracted from partial data having a data length shorter than the data length of the new data. (9) An information processing method according to the above. (11) A generation unit that generates new data obtained from the changed plurality of feature quantities by using a plurality of feature quantities that are not in a combined relationship with each other and a trained model, When the plurality of features are input, the trained model outputs the plurality of features that have been changed. Information processing device. (12) An information processing program for causing a computer to function, generating new data obtained from the changed plurality of features using a plurality of features not in a combined relationship and a trained model; causing the computer to execute When the plurality of features are input, the trained model outputs the plurality of features that have been changed. Information processing program. [Explanation of symbols]
[0112] 1. Information processing device 1a Display screen 10 Input section 20 Memory section 21 Pre-trained models 22 Information Processing Programs 30 Generation part 40 Output section
Claims
1. An information processing method for generating new data obtained from a plurality of unconnected feature quantities and a trained model, the method comprising: When the plurality of features are input, the trained model outputs the plurality of features that have been changed; each of the plurality of feature amounts is a feature amount extracted from partial data having a data length shorter than a data length of the new data; the new data has a data length equal to a total data length of the partial data corresponding to each of the plurality of feature amounts; the new data and the partial data are both music data having a data length in units of a measure, the number of bars in the data length of the new data is the total number of bars in the data length of the partial data corresponding to each of the plurality of feature amounts; Information processing methods.
2. Using the output result of the trained model and the trained model, generate the new data obtained from the plurality of further modified features. The information processing method according to claim 1 .
3. The generated new data is displayed in association with the number of times the plurality of feature quantities have been changed by the trained model. The information processing method according to claim 2 .
4. generating the new data using an additional feature that provides a direction of change of the plurality of feature values; When the plurality of features and the additional features are input, the trained model outputs the plurality of features that have been changed. The information processing method according to claim 1 .
5. displaying the generated new data in association with the directionality of the change given by the additional feature amount; The information processing method according to claim 4.
6. displaying the additional feature amount other than the additional feature amount corresponding to the generated new data in a specifiable manner; The information processing method according to claim 4.
7. the plurality of features include features sampled from a standard normal distribution, The information processing method according to claim 1 .
8. The feature sampled from the standard normal distribution is used instead of the feature extracted from partial data having a data length shorter than the data length of the new data. The information processing method according to claim 7.
9. An information processing device including a generation unit that generates new data obtained from a plurality of changed feature quantities by using a plurality of feature quantities that are not in a combined relationship and a trained model, When the plurality of features are input, the trained model outputs the plurality of features that have been changed; each of the plurality of feature amounts is a feature amount extracted from partial data having a data length shorter than a data length of the new data; the new data has a data length equal to a total data length of the partial data corresponding to each of the plurality of feature amounts; the new data and the partial data are both music data having a data length in units of a measure, the number of bars in the data length of the new data is the total number of bars in the data length of the partial data corresponding to each of the plurality of feature amounts; Information processing device.
10. An information processing program for causing a computer to function, generating new data obtained from the changed plurality of features using a plurality of features not in a combined relationship and a trained model; causing the computer to execute When the plurality of features are input, the trained model outputs the plurality of features that have been changed; each of the plurality of feature amounts is a feature amount extracted from partial data having a data length shorter than a data length of the new data; the new data has a data length equal to a total data length of the partial data corresponding to each of the plurality of feature amounts; the new data and the partial data are both music data having a data length in units of a measure, the number of bars in the data length of the new data is the total number of bars in the data length of the partial data corresponding to each of the plurality of feature amounts; Information processing program.
Citation Information
Patent Citations
Device and method for correcting chord progression, computer-readable recording medium with recorded program applied to the same device, method and device for automatic music composition, and computer-readable recording medium applied to the same device
JP2002032077A
Information processing device, information processing method, and information processing program
WO2020080268A1
