Information processing device, information processing method, and information processing program

Abstract

This information processing device (1) comprises: a control means (30); a data input means (10) for inputting series data; a machine learning model (21) for generating new series data on the basis of the series data inputted by the data input means (10); and a series data selection means (10) for selecting target series data in which alterations are made to the series data and / or context series data in which alterations are not made, when generating new series data by the machine learning model (21). The control means (30) generates: (i) new target series data that interpolates at least two series data having already been generated by the machine learning model; or (ii) new series data that is different for the series data having already been generated by the machine learning model.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Various proposals have been made regarding trained models for automatically generating content (hereinafter also referred to as "series") that provide a series of information such as music. For example, Patent Document 1 discloses a method for selectively learning features specified by a user so that a series is generated in the manner desired by the user. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2020 / 080239 Summary of the Invention [Problem to be solved by the invention]

[0004] There may be cases where it is desired to generate a sequence in which only a portion is newly generated and the remainder is maintained. Patent Document 1 does not specifically consider this point.

[0005] One aspect of the present disclosure provides an information processing device, an information processing method, and an information processing program that are capable of generating a sequence in which only a portion is newly generated and the remainder is maintained. [Means for solving the problem]

[0006] An information processing device according to one aspect of the present disclosure includes a control means, a data input means for inputting sequential data, a machine learning model for generating new sequential data based on the sequential data input by the data input means, and a sequential data selection means for selecting target sequential data to be modified and / or context sequential data to be unchanged when generating new sequential data using the machine learning model, wherein the control means (i) generates new target sequential data that interpolates at least two sequential data sets already generated by the machine learning model, or (ii) generates new sequential data that is different from the sequential data already generated by the machine learning model.

[0007] An information processing device according to one aspect of the present disclosure includes a generation unit that generates a sequence including a determined context sequence and a new target sequence using input information, which is information about a sequence that provides a series of information and is composed of a target sequence in part and a context sequence in the remainder, and a trained model, and when data corresponding to the input information is input, the trained model outputs data corresponding to the new target sequence.

[0008] An information processing device according to another aspect of the present disclosure includes a generation unit that generates a sequence including a determined context sequence and a new target sequence using input information, which is information regarding a sequence that provides a series of information and is partially composed of a target sequence and partially composed of a context sequence, and a trained model, and a user interface that accepts the input information and presents the generation results of the generation unit, and the trained model outputs data corresponding to the new target sequence when data corresponding to the input information is input.

[0009] An information processing method according to one aspect of the present disclosure includes generating a sequence including a determined context sequence and a new target sequence using input information, which is information about a sequence that provides a series of information and is partially composed of a target sequence and partially composed of a context sequence, and a trained model, wherein the trained model outputs data corresponding to the new target sequence when data corresponding to the input information is input.

[0010] An information processing program according to one aspect of the present disclosure causes a computer to generate a sequence including a determined context sequence and a new target sequence using input information, which is information about a sequence that provides a series of information and is partially composed of a target sequence and partially composed of a context sequence, and a trained model, and when data corresponding to the input information is input, the trained model outputs data corresponding to the new target sequence. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a diagram illustrating an example of the appearance of an information processing apparatus according to an embodiment. [Figure 2] FIG. 10 is a diagram showing an example of an operation screen presented by a user interface. [Figure 3] FIG. 10 is a diagram showing an example of an operation screen presented by a user interface. [Figure 4] FIG. 10 is a diagram showing an example of an operation screen presented by a user interface. [Figure 5] FIG. 1 is a diagram illustrating an example of a schematic configuration of an information processing device. [Figure 6] FIG. 10 is a diagram illustrating an example of a token. [Figure 7] FIG. 1 is a diagram illustrating an example of a schematic configuration of a trained model. [Figure 8] FIG. 1 is a diagram illustrating an example of a schematic configuration of a trained model. [Figure 9] 10 is a flowchart illustrating an example of learning. [Figure 10] 1 is a flowchart showing an example of a process (information processing method) executed in an information processing device. [Figure 11] 1 is a flowchart showing an example of a process (information processing method) executed in an information processing device. [Figure 12] 1 is a flowchart showing an example of a process (information processing method) executed in an information processing device. [Figure 13] FIG. 10 is a diagram illustrating an example of searching for a sequence. [Figure 14] FIG. 1 illustrates an example of a hardware configuration of an information processing device. [Figure 15] FIG. 1 is a diagram illustrating an example of a schematic configuration of an RNN. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same elements are designated by the same reference numerals, and redundant description will be omitted.

[0013] The present disclosure will be described in the following order: 1. Embodiment 2. Hardware configuration example 3. Example of RNN configuration 4. Variations 5.Effects

[0014] 1. Embodiment The information to be processed by the information processing device according to the embodiment is a sequence (sequence data) that provides a series of information. Examples of sequences include music (music sequences, audio, etc.) and language (documents, poems, etc.). The following description will mainly focus on the case where the sequence is a music sequence.

[0015] FIG. 1 is a diagram showing an example of the appearance of an information processing device according to an embodiment. The information processing device 1 is realized, for example, by executing a predetermined program (software) on a general-purpose computer. In the example shown in FIG. 1, the information processing device 1 is a laptop used by a user U. The information processing device 1 is provided with a user interface (user interface 10 in FIG. 5 described later) for exchanging information with the user U. For example, if the information processing device is a laptop as shown in the figure, the user interface may include a display, keyboard, touchpad, speaker, etc. The display may be a touch panel display. The information processing device 1 is not limited to a laptop, and may be realized by various devices such as a tablet terminal or a smartphone.

[0016] 2 to 4 are diagrams showing examples of operation screens presented by the user interface. Referring to FIG. 2, in the item "series selection," a folder or the like is referenced and a file is selected. The series shown in the selected file is input and visualized. In this example, a musical sequence showing the correspondence between time and pitch value is displayed as a series. The entire series is referred to as series x and illustrated. Hereinafter, an operation by user U (user operation) will be simply referred to as an "operation."

[0017] By performing an operation related to the item "Specify Range," the series x is divided into multiple series. For example, a part of the visualized series x is selected, and the selected part is divided into the other part. The part of the divided series x is called the target series (shown with hatching), and the remaining part is called the context series x. C The target sequence is the part that is desired to be changed. The context sequence x C is the part that is required not to change (maintain, do not make changes). Context sequence x C is not changed, so the determined context sequence x C By the operation related to this item "Specify range", the context sequence x in sequence x can be CThe position information (corresponding to the position information R in FIG. 8 described later) is input.

[0018] A sequence is generated by the operation related to the item "Search." As will be described later in detail, when "Normal generation" is specified, the context sequence x input by the operation related to the above-mentioned "Sequence selection" is generated. C and the position information input by the operation related to "range specification," a series is generated.

[0019] Referring to FIG. 3, in the item "Generated Sequences," the generated sequence A is visualized and displayed in a manner that allows for operations such as playback. Sequence A is compared with the original sequence x (FIG. 2) and has the same context sequence x. C While they share the same feature of including a new target series x T For convenience of explanation, the target sequence x of sequence A is T , the target series x TA This is referred to and illustrated.

[0020] By the operation related to the item "Search", a further series is generated based on (starting from) series A. As will be described later in detail, when "Variation Generation" is specified, the target series x of the generated series is generated. T A sequence is generated that includes a target sequence different from the target sequence x of sequence A. In the "feature specification" operation, the features of the sequence are specified. In this example, an arbitrary position (feature) in the latent space FS that defines the feature of the sequence is specified, and a sequence having that feature (the feature corresponding to the specified position) is generated. This sequence is also generated based on the target sequence x of sequence A. TA For example, through these operations, multiple generated sequences are obtained, each containing a different new target sequence.

[0021] Referring to FIG. 4, in the item "Generated Series," the further generated series B and series C are visualized and displayed together with series A in a manner that allows operations such as designation and playback. T , the target series x TB The target sequence x of sequence C isT , the target series x TC Hereinafter, Series A, Series B, and Series C may be simply referred to as "Series A, etc."

[0022] In the "Search" item, further series are generated based on series A, etc. Details will be described later. When "Interpolation Generation" is specified, a series having characteristics between the specified series (in this example, series A and series B) is generated. "Variation Generation" and "Feature Specification" are as described above with reference to Figure 3. By repeating the operations related to the "Search" item, successive series are generated.

[0023] Note that various modes of operation may be presented by the user interface in addition to the operation screens exemplified in the above-mentioned FIGS.

[0024] 5 is a diagram showing an example of a schematic configuration of an information processing device 1. The information processing device 1 includes a storage unit 20 and a generation unit 30 in addition to the user interface 10 described above with reference to FIG.

[0025] The user interface 10 functions as an input unit (receiving unit) that accepts information through user operation. It can also be said that the user interface 10 functions as a data input means for inputting sequential data. As previously described with reference to FIG. 2, the user interface 10 can also function as sequential data selection means for selecting a target sequence (target sequential data) and / or a context sequence (context sequential data). The information accepted by the user interface 10 is referred to as "input information." Some examples of input information are described below.

[0026] The input information includes information about the sequence. The information about the sequence is a determined context sequence x CExamples of such input information are the information about sequence x described above with reference to Fig. 2 and the information about generated sequences (sequence A, etc.) described above with reference to Figs. 3 and 4. The generated sequences are sequences generated by generation unit 30, which will be described later.

[0027] The input information may include information specifying at least one of the multiple generated sequences. An example of such input information is information specifying sequence A, etc., as described above with reference to Figure 4. The input information may also be information specifying two sequences, for example, sequence A and sequence B.

[0028] The input information may include information specifying features of the sequence. An example of such input information is the information specifying positions (features of the sequence) in the latent space FS described above with reference to Figures 3 and 4.

[0029] The user interface 10 functions as an output unit (presentation unit) that presents information to the user. The user interface 10 outputs the results of generation by the generation unit 30, which will be described later. For example, the sequence A, etc., is presented (screen display, sound output, etc.) in the manner previously described with reference to FIGS. 3 and 4. Furthermore, the characteristics of the sequence A, etc., are presented as positions in the latent space FS. The user interface 10 can also be said to function as a display means that displays positions in the latent space FS in a manner that allows them to be specified.

[0030] The storage unit 20 stores various information used by the information processing device 1. As examples of information stored in the storage unit 20, a trained model 21 and an information processing program 22 are illustrated.

[0031] When data corresponding to the above input information is input, the trained model 21 generates a new target sequence x TThe trained model 21 is a trained model that is generated (trained) using training data so as to output data corresponding to the context sequence x. The trained model 21 can also be said to be a machine learning model that generates new sequence data based on input sequence data. The generation unit 30 generates corresponding data from input information and inputs it to the trained model 21. The generation unit 30 also generates a corresponding sequence from data output by the trained model 21. The input / output data of the trained model 21 includes, for example, a sequence of tokens (token sequence). In this case, the data input to the trained model 21 is a context sequence x C The data output by the trained model 21 is a new target sequence x T The tokens will be described with reference to FIG.

[0032] Fig. 6 is a diagram showing an example of a token. A musical sequence is shown at the top of the diagram as an example of a series. The horizontal axis indicates time, and the vertical axis indicates pitch value (MIDI pitch). One unit of time corresponds to one measure period. In other words, in this example, the series of information provided by the series is musical information indicating the pitch value of a sound at each time.

[0033] The bottom of the figure shows a token string corresponding to a musical sequence. In this example, a token indicates either the pitch value of a sound or the duration of the sound. In the token string, a first token and a second token are arranged in chronological order. A first token indicates the onset and cessation of each sound included in the sequence. A second token indicates the duration during which the state indicated by the corresponding first token is maintained. Each part represented by angle brackets <> corresponds to one token.

[0034] For example, the token<ON, W, 60> is a token (first token) indicating that a sound with a pitch value of 60 from a sound source W (e.g., indicating the type of musical instrument) begins to be generated at time 0.<SHIFT, 1> is a token (corresponding second token) that indicates that the state (sound source W, pitch value 60) indicated by the corresponding first token is maintained for one unit of time. In other words, SHIFT means that only the time moves (only time passes) while the state indicated by the immediately preceding token remains the same. Other tokens related to ON and SHIFT are explained in the same way. Tokens<OFF, W, 60> is a token (first token) indicating that the generation of a sound with a pitch value of 60 from the sound source W has ended. Other tokens related to OFF are explained in the same way. In this example, when multiple sounds exist at the same time, the tokens are arranged in order from the lowest to the highest sound. By determining the order in this way, it becomes easier to train the trained model 21.

[0035] The above is an example of tokens for a sequence when the sequence is music. If the sequence is language, the tokens are words, etc.

[0036] FIG. 7 is a diagram illustrating an example of a schematic configuration of a trained model. In this example, the trained model 21 is a variational autoencoder (VAE) model, and includes an encoder ENC and a decoder DEC. Examples of architectures include a Transformer and a recurrent neural network (RNN). An example of an RNN is a long short-term memory (LSTM). The encoder ENC outputs (extracts) features of an input token sequence. The decoder DEC generates (reconstructs) a token sequence to be output from the features output by the encoder ENC, for example, by using the token sequence with the highest probability.

[0037] 8 is a diagram showing an example of training. In the training stage, three models are used: an encoder model 211, a prior model 212, and a decoder model 213. In this example, the architectures of the encoder model 211 and the prior model 212 are Transformer and RNN. The architecture of the decoder model 213 is Transformer. After training, the trained model 21 may include the prior model 212 and the decoder model 213 as the encoder ENC and the decoder DEC in FIG. 7 above.

[0038] The encoder model 211 provides a feature z. The feature z may be a vector indicating a position (point) in the latent space FS. It can be said that the position in the latent space FS indicates the feature of the sequence. The latent space FS is a multidimensional space, and is also called a latent feature space. In the embodiment, the latent space FS is a space for storing a predetermined context sequence x C The latent space FS in FIGS. 3 and 4 described above is a representation of some dimensions (e.g., a two-dimensional representation) among multiple dimensions. The sequence x and position information R are input to the encoder model 211. The position information R may be a variable j and a variable k as described below.

[0039] The sequence x input to the encoder model 211 is composed of tokens s1,...s k-1 ,s k ,…,s j ,s j+1 ,…,s L The subscripts indicate the order of each token in the sequence. Among the subscripts, the variables j and k give the position information R. The first to k-1th tokens s1 to s k-1 , and the jth to Lth tokens s j ~s L is the context sequence x C In other words, the kth to j-1th tokens s k ~s j-1is generated later, the new target sequence x T is identified as the position of

[0040] In the encoder model 211, among the tokens whose positions are identified as described above, the context sequence x C Only tokens from the input context sequence x are input to the RNN. C In this way, the encoder model 211 outputs the feature amount z when the sequence x and the position information R are input, and is therefore represented as "q(z|x, R)" in the diagram.

[0041] The prior model 212 also provides a feature z, similar to the encoder model 211. The prior model 212 is provided with a context sequence x C and position information R are input.

[0042] Context sequence x C is the token s1,…s k-1 and tokens j+1 ,…,s L The remaining token is given as a predetermined token M. If there are multiple remaining tokens, they may all be given as the same token M. The context sequence x of the sequence x c The part other than the new target sequence x T The token M is a part of the context sequence x C The feature z may be determined to be different from any of the feature z corresponding to tokens that may be input as the token of the above.

[0043] The location information R is as explained above. In this example, the first to k-1th tokens s1 to s k-1 , and the jth to Lth tokens s j ~s L is the context sequence x C is identified as the position of

[0044] In the prior model 212, only token M among the tokens whose positions have been identified as described above is input to the RNN. The RNN outputs the feature value z of the input token M. In this way, the prior model 212 calculates the feature value z of the context sequence x C When the location information R is input, the feature value z is output, so "p(z|x C ,R)" and illustrated.

[0045] The decoder model 213 is a decoder that uses the feature z and the context sequence x C Based on the tokens of the new target series x T Specifically, the decoder model 213 generates tokens from the context sequence x C and target series x T Target series x of T Reconstruct only tokens in the reconstructed target sequence x T token and the original defined context sequence x C and the tokens are combined by, for example, the generator 30 to generate a context sequence x C and a new target series x T In this way, the decoder model 213 generates a sequence including the feature z and the context sequence x C When the position information R is input, the target sequence x T Since only p(x T |z,x C ,R)" and illustrated.

[0046] In the example shown in FIG. 8, the decoder model 213 j+1 ,…,s L ,B,s1,…,s k-1 Token s k ,…,s j Generates the referenced token s j+1 ,…,s L ,B,s1,…,s k-1 So, tokens j+1 ,…,s L and tokens s1,…,s k-1However, the tokens are placed in the opposite position from the original position, with token Y (the token indicating the start of the sequence) in between. By generating tokens in this order, token s k-1 followed by tokens k This makes it easier to model. j+1 When placed at the back, the token s k However, there may be a problem that the token s cannot be referenced when generating it. j+1 If token Y does not exist, for example, when k=1, there is no token after token Y, and token s L Based on tokens k However, the existence of token Y eliminates this problem.

[0047] The encoder model 211, the prior model 212, and the decoder model 213 are trained to minimize a loss function. In this example, the loss function is rec and loss function L pri The loss function L rec and loss function L pri The parameters of the encoder model 211, the prior model 212, and the decoder model 213 are trained so as to minimize the sum (addition value, etc.) of the loss function L rec is the error (reconstruction error) when the decoder model 213 reconstructs the target sequence using the feature z output by the prior model 212. The loss function L pri is the difference (prior error) between the distributions of the encoder model 211 and the prior model 212. An example of the prior error is the Kullback-Leibler (KL) distance.

[0048] 9 is a flowchart showing an example of learning. It is assumed that learning data containing a large number of different sequences x has been prepared.

[0049] In step S1, a mini-batch of sequences is obtained from training data. For example, an arbitrary predetermined number (e.g., 64) of sequences x are obtained (sampled) from the training data.

[0050] In step S2, position information is set. For example, the position information R described above with reference to Fig. 8, more specifically, the values ​​of j and k, are set (sampled) to arbitrary values.

[0051] In step S3, the parameters are updated using a loss function. For example, as described above with reference to FIG. 8, the loss function L rec and loss function L pri The parameters of the encoder model 211, the prior model 212, and the decoder model 213 are updated (learned) so as to minimize the sum of

[0052] The learning process in steps S1 to S3 is repeated a predetermined number of times. That is, as shown in step S4, if the number of learning times is less than the predetermined number (step S4: YES), the process returns to step S1. If the number of learning times reaches the predetermined number (step S4: NO), the process of the flowchart ends.

[0053] For example, the trained model 21 is generated in the above manner. Note that different position information may be set for the same mini-batch to perform parameter updating. In this case, the processes of steps S2 and S3 may be repeated the number of times equal to the number of patterns of the set position information R.

[0054] Returning to FIG. 5, the information processing program 22 is a program (software) for realizing the processes executed in the information processing device 1.

[0055] The generation unit 30 generates a predetermined context sequence x using input information input to the user interface 10 and the trained model 21. C and a new target series x TThe generated sequence is the generated sequence (sequence A, etc.) previously described with reference to FIGS. 3 and 4. The generation unit 30 can also be considered a control means for generating a sequence. The function of the control means may be realized by a processor or the like (for example, CPU 1100 in FIG. 14, which will be described later). Some examples of the method for generating a sequence by the generation unit 30 will be described below.

[0056] 10 to 12 are flowcharts showing examples of processing (information processing methods) executed in an information processing device.

[0057] An example of the first generation method is shown in Fig. 10. In the first generation method, the generation unit 30 generates a context sequence x C A sequence is generated (for example, randomly) using the position information R and the trained model 21. The first generation method is called "normal generation" and is illustrated.

[0058] In step S11, the feature quantity is acquired (sampled) using the input context sequence and position information and the prior model. For example, the user interface 10 acquires (samples) the context sequence x by the operation related to the items "sequence selection" and "range specification" as described above with reference to FIG. C and position information R as input information. As previously described with reference to FIG. 8, the generation unit 30 generates a context sequence x C A sequence including a predetermined token M and position information R are input to the prior model 212. The prior model 212 outputs (extracts) a feature amount z corresponding to the token M.

[0059] In step S12, a target sequence is generated using the context sequence, the features, and a decoder. For example, the generation unit 30 generates a target sequence by using the trained model 21 to generate a target sequence from the context sequence x C The obtained feature quantity z is input to the decoder model 213 as described above with reference to FIG. T Generate (reconstruct)

[0060] In step S13, a sequence including a context sequence and a target sequence is generated. For example, the generation unit 30 generates the context sequence x C and the generated new target sequence x T and generate a sequence containing them.

[0061] FIG. 11 shows an example of a second generation technique. In the second generation technique, the generation unit 30 generates a sequence that includes a target sequence different from a target sequence of a previously generated sequence as a new target sequence. For example, the generation unit 30 generates a sequence that includes a target sequence that has characteristics between two specified sequences (that is, that complements the two sequence data) as a new target sequence. The second generation technique is referred to as "interpolation generation" and is illustrated.

[0062] In step S21, a feature different from the feature of the specified plurality of series is identified. For example, the user interface 10 receives, as input information, information specifying series A and series B, and information specifying "interpolation generation," as previously described with reference to FIG. 4. The generation unit 30 uses the trained model 21 to generate a feature z A and the feature value z of sequence B B The feature value of the position between the positions of AB Since the trained model 21 has learned the latent space FS, such a feature z AB It is possible to identify the following.

[0063] Feature z AB is the feature z A and feature z B For example, Z AB =(1-α)Z A +αZ B α may be calculated as follows: AB Feature z in A and feature z BIn this example, (1-α) represents the ratio (blending ratio) of the feature z A α indicates the ratio of feature z B For example, when α=0.25, the feature z A and feature z B The feature obtained by combining (blending) these features at a ratio of 0.75:0.25 is the feature z AB For example, the user interface 10 may provide a display that allows the user to specify α.

[0064] In step S22, a target sequence is generated using the identified feature, the context sequence, and a decoder. For example, the generation unit 30 generates a target sequence using the trained model 21 based on the feature z identified in the previous step S21. AB is input to the decoder model 213. The decoder model 213 calculates the feature z AB The target sequence x corresponding to TAB The target sequence x obtained in this way is TAB and the context sequence x C are combined to generate a new sequence AB.

[0065] FIG. 12 shows an example of the third generation method. In the third generation method, the generation unit 30 also generates a sequence that includes a target sequence that is different from (different from) a target sequence of a generated sequence as a new target sequence. In the third generation method, it is sufficient to specify one sequence. The third generation method is referred to as "variation generation" and is illustrated.

[0066] In step S31, feature quantities near the feature quantities of the specified series are identified. For example, the user interface 10 receives, as input information, information specifying series A in the example of FIG. 3 or FIG. 4 described above and information specifying "variation generation." The generation unit 30 uses the trained model 21 to identify the feature quantities z A The feature value z at a position slightly moved from A The movement is determined by, for example, the feature z AThis is done by adding noise to the latent space FS. The noise may be sampled from a normal distribution in each dimension of the latent space FS. The mean and variance of the normal distribution may be set arbitrarily (e.g., mean 0, variance 0.01).

[0067] In step S32, a target sequence is generated using the identified feature, the context sequence, and a decoder. For example, the generation unit 30 generates a target sequence using the trained model 21 based on the feature z identified in the previous step S31. A ' is input to the decoder model 213. The decoder model 213 calculates the feature z A ´ corresponding target sequence xT A The target sequence xT obtained in this way is A ´ and context sequence x C and are combined to generate a new sequence A'. Note that multiple different features may be identified in the previous step S32, in which case new target sequences are generated in the same number as the number of features (number of variations), and thus new sequences are generated. For example, the user interface 10 may provide a display or the like that allows the user to specify the number of variations.

[0068] Note that the source series and generated series for the above-described variation generation may overlap with the source series and generated series for the above-described interpolation generation. For example, as described above, series B is generated by interpolation generation from series A and series C. Series A and series C can be generated by variation generation from series B.

[0069] In addition to the above-described normal generation, interpolation generation, and variation generation, various other generation methods may be used. As a fourth generation method, the generation unit 30 may generate a sequence having specified features. For example, as previously described with reference to FIGS. 3 and 4, the user interface 10 receives, as input information, information specifying a position in the latent space FS (sequence features). The generation unit 30 inputs the feature amount at the specified position to the decoder model 213. The decoder model 213 generates a target sequence corresponding to the feature amount. The context sequence x C and the target sequence are combined to generate new sequences D, E, F, etc.

[0070] By combining the various generation methods described above, it is possible to search for a desired sequence. This will be explained with reference to FIG.

[0071] FIG. 13 is a diagram showing a schematic example of a sequence search. The search proceeds from the left side of the diagram to the right side. As a premise, it is assumed that a sequence A, etc. has been obtained using the various generation methods described above. The position of the sequence A, etc. in the latent space FS is shown in the diagram.

[0072] Based on sequence A, etc. (as a starting point), a search for further sequences is performed. For example, as shown in the upper part of the figure, interpolation generation may be performed. In this example, sequence AB (shown by white circles) having characteristics between sequence A and sequence B, and sequence BC (shown by white circles) having characteristics between sequence B and sequence C are generated. From the generated sequence AB and sequence BC, etc., further sequence may be generated by interpolation generation, variation generation, feature specification, etc.

[0073] Alternatively, as shown in the middle part of the figure, variation generation may be performed. In this example, series A', series A'', and series A''' (all shown as white circles) having features obtained by adding noise to the features of series A are generated. Further series may be generated from the generated series A', series A'', and series A''', etc. by interpolation generation, variation generation, feature specification, etc.

[0074] Alternatively, feature specification may be performed as shown in the lower part of the figure. In this example, series D, series E, and series F (all shown as white circles) having specified features are generated. Further series may be generated from the generated series D, series E, series F, etc. by interpolation generation, variation generation, feature specification, etc.

[0075] For example, in this manner, user U can repeat the generation of sequences until a desired sequence is obtained.

[0076] As described above, information processing device 1 can generate sequences by combining various generation techniques, providing sequence generation with excellent operability. User U can narrow down the search to obtain a desired target sequence. For example, user U can generate sequences A to G, each containing different target sequences, and then blend preferred sequences B and F to generate a sequence using interpolation generation. User U can also improve a preferred target sequence by making minor corrections. For example, user U can generate sequences similar to sequence A but slightly different (e.g., sequences B to E) using variation generation. Among the generated sequences, user U can blend sequences close to the image (e.g., sequences C and E) using interpolation generation to generate a further sequence.

[0077] 2. Hardware configuration example 14 is a diagram showing an example of the hardware configuration of an information processing device. In this example, the information processing device 1 is realized by a computer 1000. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, an HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0078] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0079] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0080] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records an information processing program according to the present disclosure, which is an example of program data 1450.

[0081] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0082] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disk), magneto-optical recording media such as an MO (Magneto-Optical disk), tape media, magnetic recording media, and semiconductor memories.

[0083] For example, when the computer 1000 functions as the information processing device 1, the CPU 1100 of the computer 1000 executes an information processing program loaded onto the RAM 1200 to realize functions of the generation unit 30, etc. Also, the HDD 1400 stores a program according to the present disclosure (the information processing program 22 in the storage unit 20) and data in the storage unit 20. Note that the CPU 1100 reads and executes program data 1450 from the HDD 1400, but as another example, the CPU 1100 may obtain these programs from another device via an external network 1550.

[0084] 3. Example of RNN configuration FIG. 15 is a diagram showing an example of the schematic configuration of an RNN. The illustrated RNN includes an input layer, an intermediate layer, and an output layer. Several neurons included in the layers are schematically shown as white circles. Tokens, etc., as previously described with reference to FIG. 5, etc., are input to the input layer. In this example, the intermediate layer includes an LSTM block, which learns long-term dependencies and is suitable for handling sequences (e.g., time-series data) such as music and documents. The output layer is a fully connected layer, which outputs tokens, as previously described with reference to FIG. 5, etc., along with probabilities.

[0085] 4. Variations The disclosed technology is not limited to the above-described embodiment, and some modifications will be described below.

[0086] In the above embodiment, in the range specification (Fig. 1), the sequence x is divided into one target sequence and two context sequences (context sequence x C1 and the context sequence x C2 ) has been described (Figure 1). However, the scope may be specified so that the sequence x is divided into one target sequence and one context sequence.

[0087] Some of the functions of the information processing device 1 may be realized outside the information processing device 1 (for example, an external server). In this case, the information processing device 1 may include some or all of the functions of the storage unit 20 and the generation unit 30 in the external server. The information processing device 1 communicates with the external server, thereby similarly realizing the processing of the information processing device 1 described above.

[0088] The trained model 21 may also include an encoder model 211 as an encoder ENC. In this case, the trained model 21 can be used to extract features from a sequence x that also includes a target sequence, as described with reference to FIG.

[0089] 5.Effects The information processing device 1 described above can be specified, for example, as follows: As described with reference to FIGS. 1 to 5, the information processing device 1 includes a control means (generation unit 30), a data input means (user interface 10) for inputting sequence data (sequence x, etc.), a machine learning model (trained model 21) for generating new sequence data (sequence A, for example) based on the sequence data (sequence x, for example) input by the data input means (user interface 10), and target sequence data (target sequence x, for example) for modifying the sequence data (sequence A, for example) when generating the new sequence data (sequence A) using the machine learning model (trained model 21). TA ), and / or the unaltered context sequence data (context sequence x C and a sequence data selection means (user interface 10) for selecting a sequence data item from the sequence data items (e.g., sequence A and sequence B) that have already been generated by the machine learning model (trained model 21), and the control means (generation unit 30) is characterized in that it (i) generates new target sequence data that interpolates at least two sequence data items (e.g., sequence A and sequence B) that have already been generated by the machine learning model (trained model 21), or (ii) generates new sequence data that is different from the sequence data (e.g., sequence A) that have already been generated by the machine learning model (trained model 21).

[0090] The information processing device 1 may further include a display means (user interface 10) that displays, in a manner that allows a user to specify a position in a space (latent space FS) that defines the feature of sequence data (e.g., sequence A) learned by a machine learning model (trained model 21), and the control means (generation unit 30) may generate, as new sequence data, sequence data having a feature corresponding to the specified position in the space (latent space FS).

[0091] The information processing device 1 is also specified as follows: As described with reference to FIGS. 1 to 5, the information processing device 1 uses input information, which is information about a sequence that provides a series of information and is partly composed of a target sequence and the rest of which is composed of a context sequence, and a trained model 21 to generate a determined context sequence x C and a new target series x TWhen data corresponding to the input information is input, the trained model 21 generates a new target sequence x T The information processing device 1 may include a user interface 10 that receives input information and presents the results of generation by the generation unit 30.

[0092] According to the information processing device 1, a predetermined context sequence x C and a new target series x T A sequence containing the context sequence x is generated. C constitutes part of the series, and the target series x T constitutes the remainder of the sequence. It is therefore possible to generate a sequence in which only a portion is newly generated and the remainder is preserved.

[0093] As described with reference to FIGS. 2 and 8, the input information (for example, received by the user interface 10) is a predetermined context sequence x C and a determined context sequence x in the sequence C For example, using such input information and the trained model 21, a determined context sequence x C and a new target series x T A sequence containing:

[0094] As described with reference to FIGS. 3 and 4, the input information (e.g., received by the user interface 10) includes information about the sequence (e.g., sequence A) generated by the generator 30, and the generator 30 determines the target sequence (e.g., target sequence x) of the sequence generated by the generator 30. TA A sequence may be generated that includes a sequence different from the original sequence (e.g., the original sequence) as a new target sequence. This allows further sequence generation based on the generated sequence.

[0095] As described with reference to FIGS. 3 and 4 , the input information (e.g., received by the user interface 10) includes information specifying at least one of the multiple sequences (e.g., sequence A) generated by the generation unit 30, and the generation unit 30 generates a target sequence (e.g., target sequence x TA A sequence may be generated that includes a target sequence different from the specified sequence as a new target sequence. This allows further sequences to be generated based on the specified sequence.

[0096] As described with reference to FIG. 4 and the like, the input information (e.g., received by the user interface 10) includes information specifying two sequences (e.g., sequence A and sequence B) among a plurality of sequences (e.g., sequence A) generated by the generation unit 30, and the generation unit 30 generates a target sequence (e.g., target sequence x TA and target series x TB ) as a new target sequence, thereby generating a sequence having characteristics between the two specified sequences.

[0097] 3 and 4, the input information (e.g., received by the user interface 10) may include information specifying the characteristics of a sequence (e.g., its position in the latent space FS), and the generation unit 30 may generate a sequence having the specified characteristics. In this way, a sequence having the specified characteristics can be generated.

[0098] As described with reference to FIGS. 6 to 8, the data input to the trained model 21 is a predetermined context sequence x C tokens (e.g. tokens s1,…s k-1 and tokens j+1 ,…,s L ), and the data output by the trained model 21 is a new target sequence x T token (e.g. token s k ,…,s j) The input data may further include a predetermined token M. The series of information provided by the sequence is musical information indicating the pitch value of a sound at each time, and the token may indicate at least one of the pitch value of the sound and the duration of the sound. For example, the trained model 21 can be used with such tokens as input and output data.

[0099] The information processing method described with reference to Figures 10 to 12 and the like is also an aspect of the present disclosure. The information processing method uses input information, which is information about a sequence that provides a series of information and is partly composed of a target sequence and the rest of which is composed of a context sequence, and a trained model 21 to generate a predetermined context sequence x C and a new target series x T (step S13, step S22 and / or step S32), and when data corresponding to the input information is input, the trained model 21 generates a new target sequence x T As explained above, this information processing method also makes it possible to generate a sequence in which only a portion is newly generated and the remainder is maintained.

[0100] The information processing program 22 described with reference to Fig. 5 etc. is also an embodiment of the present disclosure. The information processing program 22 uses input information, which is information about a sequence that provides a series of information, part of which is composed of a target sequence and the rest of which is composed of a context sequence, and a trained model 21 to calculate a predetermined context sequence x C and a new target series x T (step S13, step S22 and / or step S32), and the trained model 21 generates a new target sequence x T As explained above, this information processing program 22 also makes it possible to generate a sequence in which only a portion is newly generated and the remainder is maintained.

[0101] The effects described in this disclosure are merely examples and are not limited to the disclosed contents. Other effects may also be obtained.

[0102] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, components of different embodiments and modifications may be combined as appropriate.

[0103] The present technology can also be configured as follows. (1) a control means; a data input means for inputting sequence data; a machine learning model that generates new sequence data based on the sequence data input by the data input means; a sequential data selection means for selecting target sequential data to be modified and / or context sequential data to be unchanged from the sequential data when generating the new sequential data using the machine learning model; Equipped with The control means (i) generating new target sequence data that interpolates at least two sequence data already generated by the machine learning model; or (ii) generating new sequence data that is different from sequence data already generated by the machine learning model; 1. An information processing device comprising: (2) a display means for displaying a position in space that defines the feature quantity of the sequence data learned by the machine learning model in a specifiable manner; the control means generates, as the new sequence data, sequence data having a feature corresponding to the specified position in the space; The information processing device according to (1) above. (3) a generation unit that generates a sequence including a predetermined context sequence and a new target sequence using input information, the input information being information about a sequence that provides a series of information and that is composed of a target sequence and a context sequence, and a trained model; When data corresponding to the input information is input, the trained model outputs data corresponding to the new target sequence. Information processing device. (4) The input information is the determined context sequence; and position information of the determined context sequence in a sequence; Including, (3) An information processing device according to the present invention. (5) the input information includes information about a sequence generated by the generation unit, the generation unit generates a sequence including a target sequence different from a target sequence of the sequence generated by the generation unit as the new target sequence. An information processing device according to (3) or (4). (6) the input information includes information specifying at least one sequence among the plurality of sequences generated by the generation unit, the generation unit generates a sequence including a target sequence different from a target sequence of a specified sequence as the new target sequence. The information processing device according to any one of (3) to (5). (7) the input information includes information specifying two sequences among the plurality of sequences generated by the generation unit, the generation unit generates a sequence including a target sequence having characteristics between the target sequences of the two specified sequences as the new target sequence. The information processing device according to any one of (3) to (6). (8) the input information includes information specifying characteristics of the sequence; The generator generates a sequence having specified characteristics. The information processing device according to any one of (3) to (7). (9) the data input to the trained model includes tokens of the determined context sequence; The data output by the trained model includes tokens of the new target sequence. The information processing device according to any one of (3) to (8). (10) The data input to the trained model includes tokens of the determined context sequence and predetermined tokens, The data output by the trained model includes tokens of the new target sequence. The information processing device according to any one of (3) to (9). (11) The series of information provided by the sequence is music information indicating the pitch value of a sound at each time point, the token indicates at least one of the pitch value of the sound and the duration of the sound; The information processing device according to (9) or (10). (12) a generation unit that generates a sequence including a predetermined context sequence and a new target sequence using input information, which is information about a sequence that provides a series of information and is partially composed of a target sequence and the remainder of which is composed of a context sequence, and a trained model; a user interface that receives the input information and presents the generated results of the generation unit; Equipped with When data corresponding to the input information is input, the trained model outputs data corresponding to the new target sequence. Information processing device. (13) The user interface includes: the determined context sequence; and position information of the determined context sequence in a sequence; as the input information. (12) An information processing device according to (12). (14) the user interface accepts, as the input information, information about the series generated by the generation unit; the generation unit generates a sequence including a target sequence different from a target sequence of the sequence generated by the generation unit as the new target sequence. The information processing device according to (12) or (13). (15) the user interface receives, as the input information, information specifying at least one of the plurality of sequences generated by the generation unit; the generation unit generates a sequence including a target sequence different from a target sequence of a specified sequence as the new target sequence. The information processing device according to any one of (12) to (14). (16) the user interface receives, as the input information, information specifying two of the plurality of sequences generated by the generation unit; the generation unit generates a sequence including a target sequence having characteristics between the target sequences of the two specified sequences as the new target sequence. The information processing device according to any one of (12) to (15). (17) the user interface accepts, as the input information, information specifying characteristics of the series; The generator generates a sequence having specified characteristics. The information processing device according to any one of (12) to (16). (18) the data input to the trained model includes tokens of the determined context sequence; The data output by the trained model includes tokens of the new target sequence. The information processing device according to any one of (12) to (17). (19) The data input to the trained model includes tokens of the determined context sequence and predetermined tokens, The data output by the trained model includes tokens of the new target sequence. The information processing device according to any one of (12) to (18). (20) The series of information provided by the sequence is music information indicating the pitch value of a sound at each time point, the token indicates at least one of the pitch value of the sound and the duration of the sound; The information processing device according to (18) or (19). (twenty one) generating a sequence including a determined context sequence and a new target sequence using input information, the input information being information about a sequence that provides a set of information, the input information being information about a target sequence and a context sequence, and a trained model; When data corresponding to the input information is input, the trained model outputs data corresponding to the new target sequence. Information processing methods. (twenty two) generating a sequence including a predetermined context sequence and a new target sequence using input information, the input information being information about a sequence that provides a set of information and that is partly composed of a target sequence and partly composed of a context sequence, and a trained model; on the computer, When data corresponding to the input information is input, the trained model outputs data corresponding to the new target sequence. Information processing program. [Explanation of symbols]

[0104] 1. Information processing equipment 10 User interface (input means, selection means, display means) 20 Memory section 21 Trained Models (Machine Learning Models) 22 Information Processing Program 30 Generation unit (control unit) 211 Encoder Model 212 Pliers Model 213 Decoder Model ENC Encoder DEC decoder U User

Claims

1. a control means; a data input means for inputting sequence data; a machine learning model that generates new sequence data based on the sequence data input by the data input means; a sequential data selection means for selecting target sequential data to be modified and / or context sequential data to be unchanged from the sequential data when generating the new sequential data using the machine learning model; Equipped with The control by the control means is (i) generating new target sequence data that interpolates at least two sequence data already generated by the machine learning model; and, (ii) generating new sequence data that is different from sequence data already generated by the machine learning model; 1. An information processing device comprising at least the control of (i) above.

2. a display means for displaying a position in space that defines the feature quantity of the sequence data learned by the machine learning model in a specifiable manner; the control by the control means includes control to generate, as the new sequence data, sequence data having a feature corresponding to a specified position in the space; 2. The information processing device according to claim 1, wherein:

3. a generation unit that generates a sequence including a predetermined context sequence and a new target sequence using input information, the input information being information about a sequence that provides a series of information and that is composed of a target sequence and a context sequence, and a trained model; When data corresponding to the input information is input, the trained model outputs data corresponding to the new target sequence; the input information includes information specifying two sequences among the plurality of sequences generated by the generation unit, the generating unit generates a sequence including a target sequence having characteristics between target sequences of two specified sequences as the new target sequence; Information processing device.

4. The input information is the determined context sequence; and position information of the determined context sequence in a sequence; Including, The information processing device according to claim 3 .

5. the input information includes information about a sequence generated by the generation unit, The generation by the generation unit includes generating a sequence including a target sequence different from a target sequence of a sequence generated by the generation unit as the new target sequence. The information processing device according to claim 3 .

6. the input information includes information specifying at least one sequence among the plurality of sequences generated by the generation unit, the generating unit generates a sequence including a target sequence different from a target sequence of a specified sequence as the new target sequence; The information processing device according to claim 3 .

7. the input information includes information specifying characteristics of the sequence; The generation by the generation unit includes generating a sequence having specified characteristics. The information processing device according to claim 3 .

8. the data input to the trained model includes tokens of the determined context sequence; The data output by the trained model includes tokens of the new target sequence. The information processing device according to claim 3 .

9. The data input to the trained model includes tokens of the determined context sequence and predetermined tokens, The data output by the trained model includes tokens of the new target sequence. The information processing device according to claim 3 .

10. The series of information provided by the sequence is music information indicating the pitch value of a sound at each time point, the token indicates at least one of the pitch value of the sound and the duration of the sound; The information processing device according to claim 8 .

11. a generation unit that generates a sequence including a predetermined context sequence and a new target sequence using input information, which is information about a sequence that provides a series of information and is partially composed of a target sequence and the remainder of which is composed of a context sequence, and a trained model; a user interface that receives the input information and presents the generated results of the generation unit; Equipped with When data corresponding to the input information is input, the trained model outputs data corresponding to the new target sequence; the receiving by the user interface includes receiving, as the input information, information specifying two of the plurality of series generated by the generation unit; the generating unit generates a sequence including a target sequence having characteristics between target sequences of two specified sequences as the new target sequence; Information processing device.

12. The reception by the user interface is the determined context sequence; and position information of the determined context sequence in a sequence; as the input information. The information processing device according to claim 11.

13. the receiving by the user interface includes receiving, as the input information, information about the sequence generated by the generation unit; The generation by the generation unit includes generating a sequence including a target sequence different from a target sequence of a sequence generated by the generation unit as the new target sequence. The information processing device according to claim 11.

14. the receiving by the user interface includes receiving, as the input information, information specifying at least one of the plurality of sequences generated by the generation unit; the generating unit generates a sequence including a target sequence different from a target sequence of a specified sequence as the new target sequence; The information processing device according to claim 11.

15. the user interface accepts, as the input information, information specifying characteristics of the series; The generator generates a sequence having specified characteristics. The information processing device according to claim 11.

16. the data input to the trained model includes tokens of the determined context sequence; The data output by the trained model includes tokens of the new target sequence. The information processing device according to claim 11.

17. The data input to the trained model includes tokens of the determined context sequence and predetermined tokens, The data output by the trained model includes tokens of the new target sequence. The information processing device according to claim 11.

18. The series of information provided by the sequence is music information indicating the pitch value of a sound at each time point, the token indicates at least one of the pitch value of the sound and the duration of the sound; The information processing device according to claim 16.

19. generating a sequence including a determined context sequence and a new target sequence using input information, the input information being information about a sequence that provides a set of information, the input information being information about a target sequence and a context sequence, and a trained model; When data corresponding to the input information is input, the trained model outputs data corresponding to the new target sequence. An information processing method, comprising: the input information includes information specifying two of the generated multiple sequences; the generating step includes generating a sequence including a target sequence having characteristics between target sequences of two specified sequences as the new target sequence; Information processing methods.

20. generating a sequence including a predetermined context sequence and a new target sequence using input information, the input information being information about a sequence that provides a set of information and that is partly composed of a target sequence and partly composed of a context sequence, and a trained model; An information processing program that causes a computer to execute the following: When data corresponding to the input information is input, the trained model outputs data corresponding to the new target sequence; the input information includes information specifying two of the generated multiple sequences; the generating step includes generating a sequence including a target sequence having characteristics between target sequences of two specified sequences as the new target sequence; Information processing program.

Images