Information processing device, information processing method, and information processing program

Abstract

To provide an information processing apparatus, an information processing method, and an information processing program that can selectively learn a feature quantity designated by a user.SOLUTION: An information processing apparatus includes: an extraction unit that extracts, on receipt of a designation on first features out of features included in first content, the first features and second features which are other than the first features; and content generation unit that generates second content as new content having the same format as the first content, when receiving another content which is to be processed, on the basis of the first features and third features that are features of the other content and correspond to the second features.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program, and more particularly, to a process for generating a trained model generated through machine learning. [Background technology]

[0002] Information processing using machine learning is being used in various technical fields. For example, neural networks that mimic the mechanisms of the nervous system are used to learn the characteristics of content (images, music, etc.) to automatically generate other content.

[0003] For example, a technology has been proposed that makes it possible to automatically compose an appropriate piece of music that matches lyrics by learning the characteristics of existing songs, without the user having to input parameters other than the lyrics. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2011-175006 A Summary of the Invention [Problem to be solved by the invention]

[0005] According to conventional technology, by learning linguistic features calculated from lyric data representing the lyrics of each song and attribute data representing the attributes of the song, when new lyric data is given, it is possible to automatically generate a song that matches the new lyric data.

[0006] However, conventional techniques only generate melodies and chords that match the lyrics, and do not necessarily generate songs in the manner desired by the user. For example, conventional techniques have difficulty in generating songs that retain the characteristics specified by the user, such as changing the melody while retaining only the rhythm of an existing song. In other words, conventional learning processes have difficulty in selectively learning some of the characteristics of content (for example, the rhythm, melody, and constituent notes of a song).

[0007] Therefore, the present disclosure proposes an information processing device, an information processing method, and an information processing program capable of selectively learning a feature amount designated by a user. [Means for solving the problem]

[0008] In order to solve the above problems, an information processing device of one embodiment of the present disclosure includes an extraction unit that, when a designation regarding a first feature among features contained in a first content is received from a user, extracts the first feature and a second feature that is a feature other than the first feature, and a content generation unit that, when another content to be processed is acquired, generates a second content that is a new content having the same format as the first content based on a third feature that corresponds to the second feature and is a feature of the other content, and the first feature. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of information processing according to an embodiment of the present disclosure. [Diagram 2] FIG. 1 is a diagram illustrating feature amounts according to an embodiment of the present disclosure. [Diagram 3] FIG. 11 is a diagram illustrating a generation process according to an embodiment of the present disclosure. [Figure 4] FIG. 1 is a diagram illustrating a configuration example of an information processing device according to an embodiment of the present disclosure. [Diagram 5] FIG. 2 is a diagram illustrating an example of a music data storage unit according to an embodiment of the present disclosure. [Figure 6]1 is a flowchart (1) showing a procedure of information processing according to an embodiment of the present disclosure. [Figure 7] 11 is a flowchart (2) showing the procedure of information processing according to an embodiment of the present disclosure. [Figure 8] FIG. 11 is a diagram illustrating an example of a learning process according to a first modified example of the present disclosure. [Figure 9] FIG. 11 is a diagram illustrating an example of a learning process according to a second modified example of the present disclosure. [Figure 10] FIG. 13 is a diagram illustrating an example of a generation process according to a third modified example of the present disclosure. [Figure 11] FIG. 2 is a diagram for conceptually explaining a generation process according to the present disclosure. [Figure 12] FIG. 11 is a diagram (1) showing an example of a learning process according to a fourth modified example of the present disclosure. [Figure 13] FIG. 13 is a diagram (2) showing an example of a learning process according to a fourth modified example of the present disclosure. [Figure 14] FIG. 13 is a diagram (3) showing an example of a learning process according to a fourth modified example of the present disclosure. [Figure 15] FIG. 4 is a diagram showing an example of a learning process according to a fourth modified example of the present disclosure. [Figure 16] FIG. 13 is a diagram illustrating an example of a learning process according to a fifth modified example of the present disclosure. [Figure 17] FIG. 11 is a diagram (1) showing an example of a generation process according to a fifth modified example of the present disclosure. [Figure 18] FIG. 13 is a diagram (2) showing an example of a generation process according to a fifth modified example of the present disclosure. [Figure 19] FIG. 13 is a diagram (1) showing an example of information processing according to a sixth modified example of the present disclosure. [Figure 20] FIG. 2 is a diagram showing an example of information processing according to a sixth modified example of the present disclosure. [Figure 21] FIG. 13 is a diagram (3) showing an example of information processing according to the sixth modified example of the present disclosure. [Figure 22] FIG. 2 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.

[0011] The present disclosure will be described in the following order. 1. Embodiment 1-1. Overview of information processing according to an embodiment of the present disclosure 1-2. Configuration of the information processing device according to the embodiment 1-3. Information Processing Procedure According to the Embodiment 2. Other embodiments 3. Hardware Configuration

[0012] (1. EMBODIMENT) [1-1. Overview of information processing according to an embodiment of the present disclosure] 1 is a diagram illustrating an example of information processing according to an embodiment of the present disclosure. The information processing according to the embodiment of the present disclosure is realized by an information processing device 100 illustrated in FIG.

[0013] The information processing device 100 is a device that executes information processing according to the present disclosure, and is, for example, an information processing terminal or a server device.

[0014] In the embodiment, the information processing device 100 generates a trained model for extracting features of content, and generates new content based on the trained model. In the embodiment, the content is composed of digital data in a predetermined format, such as music (songs), images, videos, etc. In the example of FIG. 1, the information processing device 100 uses songs as an example of content for processing.

[0015] The trained model according to the embodiment includes an encoder that extracts features from data constituting the content, and a decoder that reconstructs the content from the extracted features. For example, the information processing device 100 trains the encoder by unsupervised learning, such as VAE (Variational Auto Encoder) or GAN (Generative Adversarial Networks). Specifically, the information processing device 100 inputs the content to the encoder, reconstructs the content from the extracted features, and compares the content before and after the content to adjust the parameters of the encoder and the decoder. The information processing device 100 repeats the above process and optimizes the parameters of the encoder and the decoder to generate a trained model. This allows the information processing device 100 to generate a trained model that can obtain appropriate features from data constituting the content. The features are expressed, for example, as a vector with a lower dimensionality than the data of the input content.

[0016] In addition, the trained model is not limited to the above example, and may be any type of model as long as it can extract features and reconstruct content from the extracted features.

[0017] In the example of Figure 1, the information processing device 100 inputs a song (in other words, digital data indicating the sounds that make up the song) into an encoder, and extracts elements that indicate the characteristics of the song, such as the melody line, constituent sounds, rhythm (the temporal structure of the music, for example, how many notes and rests are included, and the order in which the sounds are played), tempo, and beat, as characteristics of the song.

[0018] Here, when generating a trained model that extracts features of existing songs as described above, one challenge may be to properly extract features specified by the user. For example, if the features of an arbitrary song are extracted as is, it is possible to reproduce the features of the song, but it may be difficult to use the features, such as generating new content using the features. Specifically, the features extracted using the above-mentioned VAE, etc. are a list of vector values, and it is difficult for humans to recognize what features each value indicates. For example, even if an attempt is made to automatically generate a new song that incorporates only the rhythmic features of existing songs, it is difficult for humans to determine which values ​​in the feature vector (in other words, the latent space indicating the features) should be used to generate the new song.

[0019] Therefore, the information processing device 100 according to the present disclosure generates a trained model capable of extracting not only the feature of the entire song but also the feature desired by the user through information processing described below. Specifically, the information processing device 100 extracts predetermined data (hereinafter, referred to as "first data" for distinction) from elements (in other words, digital data constituting the first content) constituting the content (hereinafter, referred to as "first content" for distinction) for which the feature is to be calculated. Then, the information processing device 100 generates a trained model having a first encoder that calculates a first feature that is a feature of the first content, and a second encoder that calculates a second feature that is a feature of the extracted first data.

[0020] In this way, when extracting the feature of the first content, the information processing device 100 generates a trained model including a first encoder that extracts the feature of the entire first content and a second encoder that handles only specific data of the first content. The first data is data extracted from the first content by a predetermined operation to facilitate extraction of the feature specified by the user, as will be described later in detail. In this way, the information processing device 100 uses a plurality of encoders to generate a trained model capable of extracting feature amounts separately from the first content and some elements of the first content. This allows the information processing device 100 to obtain feature amounts that are expressed by separating only the features desired by the user. In addition, the information processing device 100 can automatically generate new content (hereinafter, referred to as "second content" for distinction) by changing or not changing only the feature (e.g., the rhythm of a song) specified by the user from the existing content by using the feature amounts obtained by separation. For the sake of explanation, the encoder that extracts the features of the first content is referred to as the first encoder, and the encoder that extracts the features of the first data is referred to as the second encoder. However, this is for convenience, and for example, the encoder that extracts the features of the first content may be referred to as the second encoder.

[0021] Hereinafter, an overview of the information processing of the present disclosure will be described along with the flow with reference to Fig. 1. Fig. 1 shows the flow of processing in which an information processing device 100 learns a trained model (a first encoder 50, a second encoder 55, and a decoder 60) using training data.

[0022] First, the information processing device 100 acquires a song 30 as an example of learning data (step S1). The song 30 is composed of, for example, a symbol string (digital data) indicating pitch, duration, and rests. As an example, pitch is a frequency that indicates the height of a sound expressed in a predetermined step (for example, 128 steps, etc.). Duration represents how long a reproduced sound will last. A rest represents the timing at which the reproduction of a sound pauses. The data representing the song 30 may also include information such as the beat and tempo of the song 30, symbols indicating bar divisions, chords at a certain timing, and constituent notes that make up the chords.

[0023] For example, the information processing device 100 converts the symbol string so that it can be handled by a model. As an example, the information processing device 100 uses a vector representation of the symbol string (for example, an embedding vector indicating sound information in which pitch, duration, etc. are assigned to each dimension) for processing. Note that data indicating such sounds may be expressed in, for example, MIDI (Musical Instrument Digital Interface) (registered trademark) format, digital data in a known format that can be played back by a general-purpose sequencer, or waveform data in WAV format or the like.

[0024] Through the above process, the information processing device 100 obtains data indicating the song 30 (step S2). In the example of Fig. 1, the data (vector) indicating the song 30 is expressed as "x1".

[0025] Next, the information processing device 100 executes an extraction operation 40 on the data x1 (step S3). The extraction operation 40 is a pre-processing for isolating only specific features (e.g., features desired by the user) from the song 30, and refers to an operation of performing a predetermined calculation on the data x1 to extract predetermined data (corresponding to the above-mentioned "first data").

[0026] As an example, the information processing device 100 extracts the first data by performing order-invariant arithmetic processing on the data x1 (step S4). The order-invariant arithmetic refers to, for example, an arithmetic using a commutative operator. Specifically, the order-invariant arithmetic refers to an arithmetic that does not depend on order, such as addition, multiplication, and max operation (including, for example, an arithmetic that can be said to be almost order-invariant, such as an arithmetic whose calculation result changes slightly by including a predetermined coefficient). In the example of FIG. 1, the information extracted from the data x1 by the extraction operation 40 is expressed as feature information "f1".

[0027] For example, in the example of Fig. 1, it is assumed that a user who causes the information processing device 100 to perform processing desires to extract "rhythm" from among the characteristics of the piece of music 30. In a piece of music, rhythm is a characteristic that has a relationship with time (sequence). For this reason, for example, when attempting to extract only the rhythm from the piece of music 30, a conceivable method would be to leave information indicating the sequence of notes (note lengths and relationships between rests) and hide (mask) the other information.

[0028] For example, the information processing device 100 performs an order-invariant operation on the data included in the data x1, causing the loss of "information indicating the order of sounds." As an example, the information processing device 100 may take the average of embedding vectors in the data x1 as the order-invariant operation.

[0029] This method will be described with a concrete example. Let us assume that the sequence of eight beats of notes that make up song 30 is "C4, _, _, _, E4, _, _, R." In this example, "C4" and "E4" indicate the pitch of the notes. The symbol "_" indicates that the previous note is being extended. And "R" indicates a rest.

[0030] Here, the information processing device 100 has a data table showing embedding vectors corresponding to each note and rest. The embedding vector is, for example, a d-dimensional vector (d is an arbitrary integer) in which "1" is input to the dimension corresponding to the corresponding note (such as "C4" or "E4") and "0" is input to the other dimensions.

[0031] In other words, when the information processing device 100 refers to a data sequence such as "C4, _, _, _, E4, _, _, R", it will refer to the embedding vector of "C4" four times, then refer to the embedding vector of "E4" three times, and refer to the embedding vector of "R" once.

[0032] Here, the information processing device 100 averages (or adds) the embedding vectors corresponding to "C4, _, _, _, E4, _, _, R". In this way, information that depends on the order (i.e., rhythm) is lost in the averaged vector, and information other than information indicating the order of notes (e.g., constituent notes, etc.) is extracted. In this way, the information processing device 100 can extract "information other than information indicating the order of notes" in local parts of a song.

[0033] The characteristic information f1 obtained in this manner maintains the constituent tones contained in the data x1 (i.e., the pitches of the constituent tones that make up the song 30), but loses information indicating the rhythm (such as the timing of when sounds are produced, the timing of rests, etc.).

[0034] Next, the information processing device 100 inputs the original data x1 to the first encoder 50, and inputs the feature information f1 to the second encoder 55, which is an encoder different from the first encoder 50 (step S5). Note that the first encoder 50 and the second encoder 55 are, for example, encoders in the above-mentioned VAE, and extract the feature amount of the input data as a low-dimensional vector.

[0035] The information processing device 100 obtains the feature amounts output from each of the first encoder 50 and the second encoder 55 (step S6). In the example of Fig. 1, the feature amount of the data x1 is expressed as "z1", and the feature amount of the feature information f1 is expressed as "z2".

[0036] Note that the feature amount z1 and the feature amount z2 are described separately for the sake of explanation, but are actually included in one feature amount vector. This point will be explained with reference to Fig. 2. Fig. 2 is a diagram for explaining feature amounts according to an embodiment of the present disclosure.

[0037] As shown in FIG. 2, in the trained model generated by the information processing device 100, it is assumed that the feature vector is a 64-dimensional vector. In this case, the feature z1 output from the first encoder 50 constitutes a part of the feature vector (for example, the first half of the feature vector). In addition, the feature z2 output from the second encoder 55 also constitutes a part of the feature vector (for example, the second half of the feature vector). Note that it is possible to arbitrarily set the feature vector to be 64-dimensional, how many dimensions are assigned to the output of the first encoder 50, and how many dimensions are assigned to the output of the second encoder 55, and the like.

[0038] That is, the information processing device 100 can determine which dimension of a feature vector is a value output from which encoder by configuring a model using a plurality of encoders. That is, the information processing device 100 can output a feature vector that allows human interpretation of which dimension indicates what feature.

[0039] Continuing the explanation by returning to Fig. 1, the information processing device 100 inputs the feature amount z1 and the feature amount z2 output from the first encoder 50 and the second encoder 55, respectively, to the decoder 60 (step S7).

[0040] The decoder 60 is a decoder that is trained to reconstruct content based on features extracted by the encoders. In the example of Fig. 1, the decoder 60 outputs data x2 based on features z1 and z2 output from the first encoder 50 and the second encoder 55, respectively (step S8).

[0041] The data x2 has the same format as the data x1 of the first content. That is, the data x2 can be said to be data (symbol string) for playing the song 35 having the same format as the song 30. In other words, the information processing device 100 reconstructs the content (the song 35 in the example of FIG. 1) based on the data x2 output from the decoder 60 (step S9).

[0042] Then, the information processing device 100 performs a learning process of the model based on the reconstructed content. Specifically, the information processing device 100 compares data before and after the feature extraction process, i.e., data x1 and data x2, and optimizes parameters of the first encoder 50, the second encoder 55, and the decoder 60 so that the data x2 approaches the data x1 (step S10). More specifically, the information processing device 100 optimizes parameters so that the reconstruction error between the data x1 and the data x2 is reduced and the posterior distribution of the feature approaches the prior distribution. Furthermore, the information processing device 100 obtains multiple learning data other than the data x1 and repeats the learning process shown in FIG. 1 to optimize the trained model. As a result, the information processing device 100 can generate a trained model according to the present disclosure.

[0043] According to the trained model generated through the process of FIG. 1, when a new song is input to the model, a feature vector in which the feature of the song is separated can be obtained. In the example of FIG. 1, the second encoder 55 of the trained model is trained based on data from which rhythm information has been lost (for example, feature information f1) among the training data, and therefore can be said to be an encoder trained to extract features other than rhythm. In other words, the feature output by the second encoder 55 can be said to capture order-invariant features such as constituent tones of a song (a certain dimension can be interpreted by humans as indicating "constituent tones"). On the other hand, the first encoder 50 of the trained model can be said to be an encoder trained to mainly extract order-variable features (i.e., rhythm) from the feature vector. In other words, the feature output by the first encoder 50 can be said to capture only the rhythmic features of a song (a certain dimension can be interpreted by humans as indicating "rhythm").

[0044] In this way, the information processing device 100 inputs information previously separated and extracted to a plurality of encoders and performs learning based on the information, thereby separating a feature vector that is normally expressed as a simple list of numerical values ​​into specific features. For example, if the content is a song, the information processing device 100 can output a feature vector in which the rhythm and melody (constituent notes of a chord, etc.) are separated. That is, the information processing device 100 can generate a trained model that can separate features specified by the user (selectively learn features specified by the user).

[0045] Furthermore, the information processing device 100 can also generate new content (second content) having a feature designated by a user by using the generated trained model. This point will be described with reference to Fig. 3. Fig. 3 is a diagram for explaining a generation process according to an embodiment of the present disclosure.

[0046] 3, the information processing device 100 acquires the feature z1 shown in FIG 1 (step S11). As described above, the feature z1 is a feature that includes a relatively large number of rhythm-related features in the song 30. In other words, the feature z1 is a feature from which an object is separated in the song 30.

[0047] Meanwhile, the information processing device 100 acquires a feature z3 that is a feature having the same dimension as the feature z2 shown in Fig. 1 (step S11). The feature z3 indicates a feature in the arbitrary data. The feature of the arbitrary data is, for example, a feature extracted from a song other than the song 30 using the second encoder 55. Alternatively, the feature of the arbitrary data may be a feature sampled from a prior distribution of features such as the feature z2.

[0048] Then, the information processing device 100 inputs the feature amount z1 and the feature amount z2 to the decoder 60 (step S12). The information processing device 100 acquires the data x3 output from the decoder 60 (step S13). Next, the information processing device 100 reconstructs the music 45, which is the second content, from the data x3 (step S14).

[0049] As described above, the feature z1 is a feature that indicates the rhythmic characteristics of the piece 30. Therefore, the piece 45 reconstructed by the data x3 inherits the characteristics of the rhythm of the piece 30 (for example, the order of the notes, the division of the notes, the number of bars, the tempo, etc.), and the constituent tones have the characteristics indicated by the feature z3. Specifically, the piece 45 maintains the rhythm of the piece 30, but the scale played in that rhythm is different from that of the piece 30 (for example, a scale that reflects the characteristics of the constituent tones in the feature z3). After reconstructing the piece 45, the information processing device 100 may repeatedly resample the feature z3 from the prior distribution while keeping the feature z1 fixed (fixing the rhythm). In this way, the information processing device 100 can automatically generate a plurality of pieces that are played in different scales while maintaining the same rhythmic characteristics. Here, when generating the second content, the above-mentioned feature amount z1 is not necessarily the same as the feature amount z1 shown in FIG. 1, and a value sampled from the prior distribution of the feature amount z1 may be used.

[0050] As a result, the information processing device 100 can separate and extract a certain feature (rhythm in this example) and generate content using the extracted feature, so that the user can freely generate songs with the features desired. For example, the information processing device 100 can fix the rhythm and generate variations of songs with different constituent sounds. Furthermore, when the feature of the constituent sounds is fixed instead of the rhythm (in the above example, the feature z2 is fixed and a different feature is input in the place corresponding to the feature z1), the information processing device 100 can generate various arpeggio-like variations in which various rhythms (phrases) are played while the type of chord is fixed. That is, the information processing device 100 can automatically generate songs with various variations by fixing any musical concept, as long as the feature can be specified by the user (in other words, a rule for generating feature information can be written).

[0051] In the above step S4, an example has been shown in which the information processing device 100 extracts the rhythm-related feature information f1 by using the order invariance in the data x1. Here, the information processing device 100 may extract the feature information f1 by using a method that loses information indicating pitch.

[0052] A specific example similar to that of step S4 above will be given for explanation. Let us assume that the sequence of eight beats of notes that make up song 30 is "C4, _, _, _, E4, _, _, R." In this example, "C4" and "E4" indicate pitches. Also, the symbol "_" indicates that the previous note is being extended. Also, "R" indicates a rest.

[0053] Here, the information processing device 100 causes information indicating pitch to be lost. For example, the information processing device 100 replaces information such as "C4" and "E4" with common information such as "M". As a result, the information constituting the song 30 becomes "M, _, _, _, M, _, _, R". Although only the pitch is lost, the rhythm is maintained.

[0054] In this case, since the extracted information (hereinafter referred to as "feature information f2") is different, the learning of the encoder from step S5 onwards is also different. That is, since the feature information f2 has information that maintains the rhythm of the song 30, the second encoder 55 that learns such information is trained to output a feature z2 that indicates the "rhythmic feature" of the song. On the other hand, the first encoder 50 is trained to output a feature z1 other than the "rhythmic feature" of the song. That is, encoders that extract information different from the example shown in FIG. 1 are trained. In this way, the learning of the first encoder 50 and the second encoder 55 varies in various ways depending on the type of the prior extraction operation 40.

[0055] In the example of FIG. 1, an example is shown in which the characteristics of the entire song 30 are learned, but the information processing device 100 may divide the song 30 into predetermined sections and obtain content data (partial data) for each section.

[0056] In this case, the information processing device 100 may divide the song 30 into partial data using various known methods. For example, the information processing device 100 detects preset bar divisions and divides the song 30 into partial data (bars). Alternatively, the information processing device 100 may divide the song 30 into partial data based on the relationship between beats and notes. For example, if the song 30 has a 4 / 4 time signature, the information processing device 100 detects a time point when four notes of a length corresponding to a quarter note are played as one division, and divides the song 30 into bars.

[0057] Furthermore, the information processing device 100 may use divisions other than bars as partial data. For example, the information processing device 100 may detect divisions of the melody line of the tune 30 (e.g., a point where a rest with a length exceeding a predetermined threshold appears, etc.) and divide the tune into partial data. In this case, the partial data does not necessarily have to coincide with a bar. Furthermore, the information processing device 100 may detect timings of chord changes in the tune 30, etc., and divide the tune into partial data based on the detected timings.

[0058] [1-2. Configuration of the information processing device according to the embodiment] Next, a configuration of the information processing device 100, which is an example of an information processing device that executes information processing according to an embodiment, will be described. Fig. 4 is a diagram showing an example of the configuration of the information processing device 100 according to an embodiment of the present disclosure.

[0059] 4, the information processing device 100 includes a communication unit 110, a storage unit 120, and a control unit 130. The information processing device 100 may also include an input unit (e.g., a keyboard, a mouse, etc.) that accepts various operations from an administrator who manages the information processing device 100, and a display unit (e.g., a liquid crystal display, etc.) that displays various information.

[0060] The communication unit 110 is realized by, for example, a network interface card (NIC) etc. The communication unit 110 is connected to a network N (such as the Internet) by wire or wirelessly, and transmits and receives information to and from other devices etc. via the network N.

[0061] The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 120 has a model storage unit 121 and a song data storage unit 122.

[0062] The model storage unit 121 stores the trained model. Specifically, the model storage unit 121 includes a first encoder 50 and a second encoder 55 that extract features of the content, and a decoder 60 that reconstructs the content. The model storage unit 121 may store training data such as the content used for training.

[0063] The song data storage unit 122 stores data related to content (songs) to be input to the model. Fig. 5 shows an example of the song data storage unit 122 according to an embodiment. Fig. 5 is a diagram showing an example of the song data storage unit 122 according to an embodiment of the present disclosure. In the example shown in Fig. 5, the song data storage unit 122 has items such as "song ID", "partial data ID", "pitch information", "length rest information", "chord information", and "rhythm information".

[0064] "Music ID" is identification information for identifying a song. "Partial data ID" is identification information for identifying partial data. Partial data corresponds to, for example, one or more bars that make up a song.

[0065] "Pitch information" indicates information about the pitch (scale) of the notes contained in the partial data. "Length and rest information" indicates the length of the notes contained in the partial data (playback time and number of beats played) and the length and timing of rests. "Chord information" indicates the type of chord contained in the partial data, the constituent notes of the chord, chord changes within a measure, etc. "Rhythm information" indicates the time signature and tempo of the measure, the positions of strong and weak beats, etc.

[0066] In Fig. 5, items such as pitch information are conceptually described as "C01", but in reality, each item stores specific data indicating a note as described above. Also, for the sake of explanation, Fig. 5 shows an example in which "pitch information" and "long rest information" are stored as different items, but these pieces of information may be stored together in one item as information indicating the notes contained in a measure. In other words, the format of the data indicating a piece of music is not limited to that shown in Fig. 5, and may be any format that can be handled by the model.

[0067] 5, the song identified by the song ID "A01" has partial data identified by partial data IDs "B01" and "B02." The partial data with the partial data ID "B01" includes sound data with pitch information "C01," long rest information "D01," chord information "E01," and rhythm information "F01."

[0068] Returning to Fig. 4, the description will be continued. The control unit 130 is realized, for example, by a central processing unit (CPU), a micro processing unit (MPU), or the like executing a program (for example, an information processing program according to the present disclosure) stored inside the information processing device 100 using a random access memory (RAM) or the like as a working area. The control unit 130 is a controller, and may be realized, for example, by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0069] 4, the control unit 130 has an extraction unit 131, a model generation unit 132, an acquisition unit 133, and a content generation unit 134, and realizes or executes the functions and actions of information processing described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 4, and may be other configurations as long as they perform the information processing described below.

[0070] The extraction unit 131 extracts the first data from the elements constituting the first content. For example, the extraction unit 131 extracts the first data from the elements constituting the first content based on a predetermined rule in accordance with a user's designation. For example, the extraction unit 131 executes an extraction operation on digital data constituting the first content, which is a song, and extracts the first data.

[0071] For example, the extraction unit 131 receives an operation designated by the user for the first content and a designation of a range in which the operation is to be performed, and executes the operation in the accepted range. Specifically, the extraction unit 131 receives from the user an operation to extract only the rhythm from the song 30 shown in Fig. 1, and a designation of a range (e.g., a bar, etc.) from which the rhythm is to be extracted. Then, the extraction unit 131 executes the operation related to the extraction according to the user's designation.

[0072] As an example of an operation, the extraction unit 131 extracts the first data by performing order-invariant arithmetic processing on elements included in the range. For example, the extraction unit 131 performs order-invariant arithmetic such as addition, multiplication, max operation, etc. on the data constituting the first content. By such processing, the extraction unit 131 can extract the first data from the first content in which only order-dependent information (rhythm in the example of the song) is masked.

[0073] The extraction unit 131 may extract the first data by losing information that is not time-dependent or order-dependent among the elements included in the range. In a song, information that is not time-dependent or order-dependent may include, for example, the constituent notes of a chord in a measure, the key of the song, the total number of notes, and the time signature of the song. On the other hand, an example of information that is time-dependent or order-dependent is rhythm information of the song.

[0074] For example, the extraction unit 131 can extract first data in which only rhythm information is maintained by losing pitch information, which is information that is not time-dependent or order-dependent, from information in a specific range of the first content.

[0075] The model generation unit 132 generates a trained model having a first encoder 50 that calculates a first feature amount that is a feature amount of the first content, and a second encoder 55 that calculates a second feature amount that is a feature amount of the first data extracted by the extraction unit 131. Note that the number of encoders included in the trained model is not limited to two, and may be greater than two.

[0076] Furthermore, the model generation unit 132 generates a trained model having a decoder 60 that decodes data corresponding to the first content based on the first feature amount and the second feature amount. That is, when decoding features extracted from a plurality of encoders, the model generation unit 132 generates a trained model having a configuration for decoding using one common decoder 60.

[0077] For example, when learning content data such as a song, the model generation unit 132 inputs the content data to the first encoder 50 and inputs feature information extracted from the content data to the second encoder 55. Next, the model generation unit 132 inputs the feature amounts output from the first encoder 50 and the second encoder 55 to the decoder 60 to reconstruct the content (song). Then, the model generation unit 132 adjusts the parameters of the first encoder 50, the second encoder 55, and the decoder 60 so that the difference between the data of the previous and next content is reduced. The model generation unit 132 repeats this process to generate a trained model in which the first encoder 50, the second encoder 55, and the decoder 60 are optimized. As described above, the model generation unit 132 may generate a model using various known methods such as VAE and GAN.

[0078] The acquisition unit 133 acquires various information. For example, the acquisition unit 133 acquires data to be input to the trained model generated by the model generation unit 132.

[0079] For example, the acquisition unit 133 acquires a feature quantity output from the first encoder 50 and a feature quantity output from the second encoder 55. The acquisition unit 133 may also change the output feature quantity and acquire the acquired information as information to be input to the trained model. For example, the acquisition unit 133 may perform sampling from a prior distribution of known feature quantities to acquire new feature quantities.

[0080] The acquisition unit 133 may also acquire any data from an information processing terminal used by a user. For example, the acquisition unit 133 acquires data that constitutes a song. The acquisition unit 133 may then input the acquired data to a trained model (in this case, the same data is input to the first encoder 50 and the second encoder 55), and acquire the feature quantities output from each encoder.

[0081] The content generation unit 134 generates a second content, which is a new content having the same format as the first content, by inputting a value that combines either the feature amounts output from the trained first encoder 50 or the second encoder 55, or the feature amounts generated based on these feature amounts, to the decoder. Note that the feature amounts generated based on the feature amounts output from the trained first encoder 50 or the second encoder 55 refer to, for example, feature amounts obtained by sampling from the prior distribution of the feature amounts output from either encoder.

[0082] For example, the content generation unit 134 may generate the second content based on a feature outputted by inputting arbitrary data to the trained first encoder 50 or the trained second encoder 55. Alternatively, as in the example shown in Fig. 3, the content generation unit 134 may generate the second content from a value (e.g., a 64-dimensional vector) obtained by combining an existing generated feature (feature z1 in the example of Fig. 3) and a feature (feature z3 in the example of Fig. 3) obtained by inputting arbitrary data to the trained first encoder 50 or the trained second encoder 55.

[0083] As described above, the feature amount that holds the feature desired by the user changes depending on the operation specified by the user. That is, which of the feature amount output from the first encoder 50 and the feature amount output from the second encoder 55 holds the feature desired by the user depends on the type of operation or extraction operation specified by the user. In any case, if the content generation unit 134 obtains a value corresponding to the feature amount input to the decoder 60 (for example, if the feature amount is a 64-dimensional vector, the value constituting the 64-dimensional vector), it can generate the second content corresponding to the feature amount.

[0084] Furthermore, the content generating unit 134 may generate the second content by inputting a combined value of a feature output from the trained first encoder 50 and a feature output from the trained second encoder 55 to the decoder 60, rather than using an existing feature. For example, the content generating unit 134 may generate the second content based on a combined value of the respective feature amounts output by inputting any data (such as a song) acquired by the acquiring unit 133 to the trained first encoder 50 and second encoder 55.

[0085] Furthermore, the content generating unit 134 may fix a value corresponding to one of the combined values ​​corresponding to one of the feature quantities and vary the values ​​corresponding to the other feature quantities in sequence to generate a plurality of second contents in sequence. For example, if the first half of the feature quantity indicates a rhythmic feature, the content generating unit 134 can fix the first half of the feature quantity and vary the second half to generate a plurality of songs in sequence with the rhythm maintained.

[0086] [1-3. Information Processing Procedure According to the Embodiment] Next, the procedure of information processing according to the embodiment will be described with reference to Fig. 6 and Fig. 7. First, the flow of learning processing according to the embodiment of the present disclosure will be described with reference to Fig. 6. Fig. 6 is a flowchart (1) showing the procedure of information processing according to the embodiment of the present disclosure.

[0087] 6, the information processing device 100 determines whether or not learning data (content) has been acquired (step S101). If the learning data has not been acquired (step S101; No), the information processing device 100 waits until the learning data is acquired.

[0088] On the other hand, when learning data is acquired (step S101; Yes), the information processing device 100 extracts target information (feature information) (step S102).

[0089] Then, the information processing device 100 inputs the original learning data and the extracted information to separate encoders (step S103). For example, the information processing device 100 inputs the original learning data to the first encoder 50 and inputs the extracted information to the second encoder 55.

[0090] Then, the information processing device 100 calculates the feature amounts of the original learning data and the extracted information (step S104). Subsequently, the information processing device 100 inputs the calculated feature amounts to the decoder 60 (step S105).

[0091] As a result, the information processing device 100 causes the decoder 60 to output data in the same format as the original training data (step S106). Then, the information processing device 100 optimizes the model based on the output data and the original training data (step S107). For example, the information processing device 100 compares the output data with the original training data and optimizes the parameters of the model so as to reduce the reconstruction error. Note that the information processing device 100 may repeat the processes from step S101 to step S107 a sufficient number of times (for example, until the parameters settle within a predetermined threshold) to optimize the model.

[0092] The information processing device 100 stores the trained model (the first encoder 50, the second encoder 55, and the decoder 60) in the storage unit 120 (step S108). With this, the information processing device 100 ends the training process.

[0093] Next, a flow of a generation process according to an embodiment of the present disclosure will be described with reference to Fig. 7. Fig. 7 is a flowchart (2) showing a procedure of information processing according to an embodiment of the present disclosure.

[0094] As shown in Fig. 7, the information processing device 100 acquires a first feature (step S201). For example, the first feature is an output from a trained first encoder 50. The information processing device 100 also acquires a second feature (step S202). For example, the second feature is an output from a trained second encoder 55.

[0095] Then, the information processing device 100 inputs a value obtained by combining the acquired first feature amount and second feature amount to the decoder 60 (step S203).

[0096] Next, the information processing device 100 generates a second content based on the data output from the decoder 60 (step S204).

[0097] (2. Other embodiments) The processing according to each of the above-described embodiments may be implemented in various different forms (modifications) other than the above-described embodiments.

[0098] [2-1. Variations in learning and generation processes] In the above embodiment, a trained model having two encoders and one decoder has been described. Here, the information processing device 100 may train various models having configurations different from those in the above embodiment. This point will be described with reference to FIG. 8 and subsequent figures.

[0099] Fig. 8 is a diagram showing an example of a learning process according to the first modified example of the present disclosure. In the example shown in Fig. 8, the model is different from the example shown in Fig. 1 in that it has a first decoder 61 and a second decoder 62.

[0100] In the example shown in FIG. 8, the information processing device 100 executes the extraction operation 40 on the data x1 (step S15) and extracts the feature information f1 (step S16), similar to step S1 shown in FIG. 1. Also, the information processing device 100 inputs the data x1 to the first encoder 50 and inputs the feature information f1 to the second encoder 55 (step S17), similar to step S5 shown in FIG. 1. As a result, the information processing device 100 obtains the feature amount z1 and the feature amount z2 (step S18). In other words, this operation is the information processing device 100 performing a function operation corresponding to the first encoder 50 or the second encoder 55 on the data x1 and the feature information f1, and outputting the feature amount z1 and the feature amount z2.

[0101] Next, the information processing device 100 inputs the feature z1 and the feature z2 to the first decoder 61, and inputs the feature z2 to the second decoder 62 (step S19). In other words, this operation means that the information processing device 100 performs a function operation corresponding to the first decoder 61 on the feature z1 and the feature z2, and performs a function operation corresponding to the second decoder 62 on the feature z2.

[0102] Through the calculation in step S19, the information processing device 100 obtains data x' from the first decoder 61. Moreover, the information processing device 100 obtains feature information f' from the second decoder 62 (step S20). Note that the data x' is data having a similar format to the data x1, and is a conceptual representation of data output via a decoder. Moreover, the feature information f' is data having a similar format to the feature information f1, and is a conceptual representation of feature information output via a decoder.

[0103] After that, the information processing device 100 adjusts the parameters of each encoder and decoder so that the data x' approaches the data x1 and the feature information f' approaches the feature information f1. That is, the information processing device 100 performs a model optimization process similar to step S10 shown in FIG.

[0104] In this manner, the information processing device 100 may train a model having two decoders. That is, the information processing device 100 generates a trained model having a first decoder 61 that decodes data corresponding to the first content (data x1) based on the first feature amount (feature amount z1) and the second feature amount (feature amount z2), and a second decoder 62 that decodes data (feature information f2) corresponding to the first data (feature information f1) extracted from the elements constituting the first content based on the second feature amount.

[0105] According to this configuration, the feature z2 learned by the second encoder 55 and the second decoder 62 reliably includes the information of the feature information f1, as compared to the configuration shown in Fig. 1. Therefore, the information processing device 100 can stably perform learning with the features separated. More specifically, the information processing device 100 can more reliably learn the information (e.g., the rhythm of a song) separated as the feature information f1 with the configuration shown in Fig. 8.

[0106] It should be noted that the model shown in Fig. 8 may have a further different configuration. This will be described with reference to Fig. 9. Fig. 9 is a diagram showing an example of a learning process according to a second modified example of the present disclosure.

[0107] The model in the example shown in FIG. 9 differs from the model in the example shown in FIG.

[0108] In the example shown in FIG. 9, the information processing device 100 executes the extraction operation 40 on the data x1 (step S31) and extracts the feature information f1 (step S32), similarly to step S15 shown in FIG. 8. Here, the information processing device 100 inputs the data x1 to the first encoder 50 (step S33) and obtains the feature amount z1 and the feature amount z2 (step S34). In this case, the information processing device 100 sets a predetermined rule in advance for the value output from the first encoder 50, and outputs the feature amount z1 and the feature amount z2. For example, the information processing device 100 sets a rule in advance, such as setting the first 32 dimensions of the output 64-dimensional vector as the feature amount z1 and the latter 32 dimensions as the feature amount z2.

[0109] Next, the information processing device 100 inputs the feature z1 and the feature z2 to the first decoder 61, and inputs the feature z2 to the second decoder 62 (step S35). Through the calculation in step S35, the information processing device 100 obtains data x' from the first decoder 61. In addition, the information processing device 100 obtains feature information f' from the second decoder 62 (step S36).

[0110] After that, the information processing device 100 adjusts the parameters of the respective encoders and decoders so that the data x' approaches the data x1 and the feature information f' approaches the feature information f1.

[0111] In this way, even in a configuration in which the second encoder 55 is omitted, the information processing device 100 can perform learning with separated features as shown in Fig. 1 and Fig. 8. This allows the information processing device 100 to simplify the configuration of a model, and therefore, for example, can quickly perform learning processing and reduce the processing load.

[0112] Next, a modified example of the generation process for generating new content will be described with reference to Fig. 10. Fig. 10 is a diagram showing an example of the generation process according to a third modified example of the present disclosure.

[0113] Fig. 10 shows a process flow when generating new content from the trained first encoder 50 and first decoder 61. Note that the first encoder 50 and the first decoder 61 are an encoder and a decoder whose parameters have been trained through the training shown in Fig. 9, for example.

[0114] The information processing device 100 inputs arbitrary data x1 to the first encoder 50 (step S41), and obtains feature amount z1 and feature amount z2 (step S42). Then, the information processing device 100 inputs feature amount z1 and feature amount z2 to the first decoder 61 (step S43), and obtains data x' (step S44). Note that the information processing device 100 may obtain the feature amount by sampling from a distribution of pre-learned features, instead of obtaining the feature amount to be input to the first decoder 61 from the first encoder 50.

[0115] In this way, the information processing device 100 can generate new content, data x', from a trained model with a simple configuration (in this example, the first encoder 50 and the first decoder 61). With this configuration, the information processing device 100 inputs both the feature z1 and the feature z2 into one decoder, and can obtain, for example, data x' including features that are not completely separated. This allows the information processing device 100 to generate new content including, for example, rhythm information that has not been completely separated into the feature z2 and is assumed to remain in the feature z1.

[0116] [2-2. Variations in extraction procedures] In the above embodiment, as an example of the extraction operation 40, a specific calculation is performed to extract information (e.g., rhythm information) that is invariant from the music piece 30. The information processing device 100 may perform various operations as the extraction operation 40 in addition to this example.

[0117] For example, in the above embodiment, an example was shown in which the information processing device 100 extracts chord constituent tones in the song 30, but the information processing device 100 may extract not only chord constituent tones but also melody constituent tones and drum constituent tones. The melody constituent tones refer to the types of tones that appear in a certain section, such as a measure of a song. The drum constituent tones refer to the types of sounds (snare drum, bass drum, etc.) of a drum set that appear in a certain section, such as a measure of a song.

[0118] The information processing device 100 may also extract information indicating the flow of pitches, such as whether the pitches rise, fall, or remain constant in the flow of a song (so-called Melodic Contour). The information processing device 100 may also extract the scale of the notes that make up a song (information indicating which of the 12 notes is used to make up the song). The information processing device 100 may also extract information that assigns an order to the notes that make up the song (ordered constituent notes). For example, the information processing device 100 may extract information that orders the pitches of adjacent notes (for example, a list of pitches such as 1st, 1st, 2nd).

[0119] Furthermore, the information processing device 100 may extract a pattern-classified sequence (such as an Implication / Realization Model, for example) in a song. Furthermore, the information processing device 100 may employ a method for extracting an important sequence of notes determined by the depth of a time-span tree of the Generative Theory of Tonal Music (GTTM), for example.

[0120] Furthermore, the information processing device 100 may extract information on lead notes, chord notes, chord types, root notes of chords, bass lines, and the like, from among the notes that make up a song.

[0121] Furthermore, when a song is composed of MIDI information, the information processing device 100 may extract strength information (velocity) in a certain section.

[0122] The information processing device 100 may also extract sounds for each instrument or each instrument group that constitutes a song. The information processing device 100 may also extract style features calculated by calculating correlations between features of a certain layer when features of a certain song are learned using a DNN (Deep Neural Network). The information processing device 100 may also extract self-similarity in a song.

[0123] The information processing device 100 may also learn the features of a song and extract the features based on the learning results. For example, the information processing device 100 may learn the task of modulating a song using a sequence-to-sequence network (model) and may extract features that are invariant to modulation by setting conditions such as modulation in the decoder.

[0124] [2-3. Application variations] As described in the above embodiment, the information processing according to the present disclosure makes it possible to perform learning that isolates a specific feature, and therefore makes it possible to selectively strengthen or weaken the influence of a feature that appears in new content. This point will be described with reference to Fig. 11. Fig. 11 is a diagram for conceptually explaining the generation processing according to the present disclosure.

[0125] 11 shows a graph 64 conceptually illustrating an image (tone interval and duration) of a piece of music generated by the generation process according to the present disclosure. As described above, the information processing device 100 can extract rhythm as one of the characteristics that represents a piece of music, and generate a new piece of music that reflects the influence of fixing or changing the rhythm.

[0126] For example, when the information processing device 100 keeps the factors that determine the rhythmic characteristics of a song (referred to as "rhythm factors") fixed and changes characteristics other than the rhythm factors, it generates a song along the axis of fixing the rhythm factors shown in Fig. 11. On the other hand, when the information processing device 100 fixes characteristics other than the rhythm factors and changes the rhythm factors, it generates a song along the axis of fixing the other factors than the rhythm factors shown in Fig. 11.

[0127] Generally, when generating content, if the feature amount cannot be separated, some change can be made to the original data, but it is difficult to adjust the direction of the change. For example, the information processing device 100 can change the feature amount of a piece of music 65 that indicates the sound information of one bar to generate a new piece of music 66, but if the feature amount is not separated, the change will be in one diagonal direction (so-called 1D-interpolation).

[0128] However, according to the information processing device 100, since it is possible to separate the rhythm factor element from among the characteristics of a piece of music, it is possible to change the piece of music 65 in a direction that fixes the rhythm factor and in a direction that fixes elements other than the rhythm factor (so-called 2D-interpolation). For example, the information processing device 100 can change the piece of music 65 to the image of all the pieces of music shown in the graph 64 according to the user's request. In this way, the information processing device 100 can generate new content by adjusting the blending ratio of the feature amounts.

[0129] That is, the information processing device 100 can generate new content or variations by not only fixing the rhythm, degree of modulation, scale, etc. of a song, but also controlling the degree of change. As a specific method, a variation of the feature can be generated by obtaining two noises and adding each of them to two feature quantities z1 and z2. In this case, if the noise is scaled, the degree of variation can be controlled for each of the two feature quantities z1 and z2. For example, there are two methods for obtaining noise: (1) obtaining noise from a certain fixed distribution such as a normal distribution, and (2) using a VAE to train an encoder and using the noise output by the encoder. In addition, the information processing device 100 can perform flexible generation processing, such as exchanging the features of two songs to generate new content.

[0130] [2-4. Variation of Features] In the above embodiment, an example has been shown in which the information processing device 100 learns two types of feature amounts (rhythm and elements other than rhythm) in a song, but the information processing device 100 may learn two or more types of feature amounts.

[0131] This point will be described with reference to FIG. 12 and subsequent figures. FIG. 12 is a diagram (1) showing an example of a learning process according to a fourth modified example of the present disclosure. In FIG. 12, a feature quantity z o , z a , z b , z c This conceptually shows the variations in the relationships between these features.

[0132] A conceptual diagram 67 shown in Fig. 12(a) shows three types of feature quantities as mutually exclusive feature sets. A conceptual diagram 68 shown in Fig. 12(b) shows four types of feature quantities as feature sets that form spaces with common parts. A conceptual diagram 69 shown in Fig. 12(c) shows three types of feature quantities as feature sets that form nested spaces.

[0133] Fig. 13 shows an example of learning the feature group shown in Fig. 12(a) Fig. 13 is a diagram (2) showing an example of the learning process according to the fourth modified example of the present disclosure.

[0134] 13, the information processing device 100 executes a first extraction operation 41 and a second extraction operation 42 on data x1 (step S51). Then, the information processing device 100 obtains feature information f1 by the first extraction operation 41, and obtains feature information f2 by the second extraction operation 42 (step S52). Next, the information processing device 100 inputs the data x1 to the first encoder 50, inputs the feature information f1 to the second encoder 55, and inputs the feature information f2 to the third encoder 56 (step S53).

[0135] The information processing device 100 receives the feature quantity z o and the feature quantity z a and the feature quantity z bNext, the information processing device 100 obtains the feature quantity z o , feature z a and feature z b is sent to the first decoder 61, and the feature quantity z a is sent to the second decoder 62, and the feature quantity z b to the third decoder 63 (step S55). Then, the information processing device 100 obtains data x' from the first decoder 61, feature information f1' from the second decoder 62, and feature information f2' from the third decoder 63 (step S56). Next, the information processing device 100 optimizes the decoder and the encoder so that the data x' approaches the data x1, the feature information f1' approaches the feature information f1, and the feature information f2' approaches the feature information f2, respectively.

[0136] With the configuration shown in Fig. 13, the information processing device 100 can learn a group of features having the relationship shown in Fig. 12(a). As a specific example of the learning shown in Fig. 13, for example, the first decoder 61 executes a task related to the melody of a song, the second decoder 62 executes a task related to the rhythm of the melody of the song, and the third decoder 63 executes a task related to the notes that make up the melody of the song. In other words, the model shown in Fig. 13 is such that the second decoder 62 executes a task related to the feature z a The third decoder 63 captures the feature z b It can be said that the first decoder 61 captures the other features.

[0137] Next, an example of learning the feature group shown in Fig. 12(b) is shown in Fig. 14. Fig. 14 is a diagram (3) showing an example of a learning process according to the fourth modified example of the present disclosure.

[0138] As shown in FIG. 14, the information processing device 100 inputs data x1 to the first encoder 50 (step S61). Then, the information processing device 100 receives a feature quantity z o , z a , z b , z c(Step S62). In this case, the information processing device 100 may have a predetermined rule for separating the feature amounts, similar to step S33 in FIG.

[0139] Next, the information processing device 100 calculates the feature quantity z o , z a , z b , z c is sent to the first decoder 61, and the feature quantity z a and feature z c is sent to the second decoder 62, and the feature quantity z b and z c These are input to the third decoder 63 (step S63). c is the feature z a and feature z b This is because it has similarities with

[0140] Then, the information processing device 100 obtains data x' from the first decoder 61, feature information f1' from the second decoder 62, and feature information f2' from the third decoder 63 (step S56). Next, the information processing device 100 optimizes the decoder and the encoder so that the data x' approaches the data x1, the feature information f1' approaches the feature information f1, and the feature information f2' approaches the feature information f2, respectively.

[0141] With the configuration shown in FIG. 14, the information processing device 100 can learn the feature group having the relationship shown in FIG. 12(b).

[0142] Next, an example of learning the feature group shown in Fig. 12(c) is shown in Fig. 15. Fig. 15 is a diagram (4) showing an example of a learning process according to the fourth modified example of the present disclosure.

[0143] Of the processes shown in FIG. 15, steps S71 to S74 are common to steps S51 to S54 shown in FIG. 13, and therefore a description thereof will be omitted.

[0144] The information processing device 100 uses the feature quantity z obtained in step S74o , feature z a and feature z b is sent to the first decoder 61, and the feature quantity z a and feature z b is sent to the second decoder 62, and the feature quantity z b These are input to the third decoder 63 (step S75). b is the feature z a and feature z o This is because it has a nested structure.

[0145] Then, the information processing device 100 obtains data x' from the first decoder 61, feature information f1' from the second decoder 62, and feature information f2' from the third decoder 63 (step S76). Next, the information processing device 100 optimizes the decoder and the encoder so that the data x' approaches the data x1, the feature information f1' approaches the feature information f1, and the feature information f2' approaches the feature information f2, respectively.

[0146] With the configuration shown in Fig. 15, the information processing device 100 can learn the feature groups having the relationships shown in Fig. 12(c). As described in Fig. 12 to Fig. 15, even if a plurality of feature amounts (feature groups) have various relationships, the information processing device 100 can separate each feature and perform learning.

[0147] [2-5. Variations in information processing] When two feature amounts (for example, the feature amount z1 and the feature amount z2) have a dependency relationship, the information processing device 100 may make the encoders and prior distributions used in the respective processes dependent on each other.

[0148] Furthermore, the information processing device 100 may share an embedding vector between two paths of the encoder. For example, when the information processing device 100 extracts constituent notes of a song and performs learning, the information processing device 100 shares an embedding vector between two paths of the encoder.

[0149] Furthermore, the information processing device 100 may separate the features using adversarial learning. Specifically, in order to separate two features (for example, the feature z1 and the feature z2), the information processing device 100 may perform adversarial learning in which a classifier that classifies the feature z2 and noise of normal distribution is trained and the feature z1 is trained so that the discrimination becomes impossible.

[0150] [2-6. Variations in model structure] The information processing device 100 may learn a model having a different structure with respect to the handling of feature amounts and feature information. This will be described with reference to Fig. 16 and subsequent figures. Fig. 16 is a diagram showing an example of a learning process according to a fifth modified example of the present disclosure.

[0151] 16, the information processing device 100 executes the extraction operation 40 on data x1 (step S81) and extracts feature information f1 (step S82). After that, the information processing device 100 inputs the feature information f1 together with the data x1 to the first encoder 50, and inputs the feature information f1 to the second encoder 55 (step S83). As a result, the information processing device 100 obtains feature amount z1 and feature amount z2 (step S84).

[0152] Next, the information processing device 100 inputs the feature z1 and the feature information f1 to the first decoder 61, and inputs the feature z2 to the second decoder 62 (step S85). Then, the information processing device 100 obtains the data x' from the first decoder 61, and obtains the feature information f' from the second decoder 62 (step S86).

[0153] After that, the information processing device 100 adjusts the parameters of each encoder and decoder so that the data x' approaches the data x1 and the feature information f' approaches the feature information f1. That is, the information processing device 100 performs a model optimization process.

[0154] In this way, the information processing device 100 may generate a trained model having a first decoder 61 that decodes data corresponding to the first content (data x1) based on the first feature (feature z1) and the first data (feature information f1), and a second decoder 62 that decodes data (feature information f') corresponding to the first data based on the second feature (feature z2).

[0155] Specifically, unlike the model shown in FIG. 8, the information processing device 100 inputs the feature information f1 to the first decoder 61 in step S85. This allows the information processing device 100 to train the encoder and decoder after conditioning the feature information f1. That is, the information processing device 100 can perform training that eliminates uncertainty regarding the feature information f1 by performing training that includes the feature information f1. Specifically, the information processing device 100 can prevent a situation in which training that separates features is not performed due to some factor (for example, only rhythm information of a song is extracted as a feature).

[0156] More specifically, according to the structure of the model in FIG. 16, the uncertainty of the feature information f1 is not learned for the feature z1, and only the uncertainty of the part of the data x1 other than the feature information f1 is learned. In other words, a space unrelated to the feature information f1 is learned for the feature z1. Meanwhile, a space related to the feature information f1 is learned in the other encoder (the second encoder 55 in the example of FIG. 16). As a result, the feature z1 and the feature z2 are learned as clearly separated spaces. That is, according to the learning process of the fifth modified example, the information processing device 100 can improve the stability of learning.

[0157] Next, a modified example of the generation process for generating new content will be described with reference to Fig. 17. Fig. 17 is a diagram (1) showing an example of the generation process according to the fifth modified example of the present disclosure.

[0158] Fig. 17 shows a process flow when generating new content from the trained first encoder 50, first decoder 61, and second decoder 62. Note that the first encoder 50, first decoder 61, and second decoder 62 are encoders and decoders whose parameters have been trained through the training shown in Fig. 16, for example.

[0159] The information processing device 100 executes the extraction operation 40 on any data x1 (step S87) and extracts feature information f1 (step S88). After that, the information processing device 100 inputs the feature information f1 together with the data x1 to the first encoder 50 (step S89). As a result, the information processing device 100 obtains a feature z1 (step S90). In addition, the information processing device 100 obtains an arbitrary feature z2 by sampling from the distribution of pre-learned features (space).

[0160] Next, the information processing device 100 inputs the obtained feature amount z2 to the second decoder 62 (step S91), and inputs the obtained feature information f', feature amount z1, and feature information f1 to the first decoder 61 (step S91). As a result, the information processing device 100 obtains data x', which is new content (step S92).

[0161] With the configuration in FIG. 17, the information processing device 100 can generate data x' that is new content reflecting the influence of the feature extracted by the extraction operation (feature information f1 in the example in FIG. 17).

[0162] Note that the information processing device 100 may employ a different configuration for generation. This will be described with reference to FIG.

[0163] Fig. 18 shows a process flow when generating new content from the trained first encoder 50, second encoder 55, first decoder 61, and second decoder 62. Note that the first encoder 50, second encoder 55, first decoder 61, and second decoder 62 are encoders and decoders whose parameters have been trained through the training shown in Fig. 16, for example.

[0164] The information processing device 100 executes the extraction operation 40 on any data x1 (step S93) and extracts feature information f1 (step S94). After that, the information processing device 100 inputs the feature information f1 together with the data x1 to the first encoder 50, and inputs the feature information f1 to the second encoder 55 (step S95). As a result, the information processing device 100 obtains the feature z1 from the first encoder 50 and the feature z2 from the second encoder 55 (step S96). At this time, the information processing device 100 may obtain the any feature z1 by sampling from a distribution of pre-learned features (space).

[0165] Next, the information processing device 100 inputs the obtained feature amount z2 to the second decoder 62 (step S97), and inputs the obtained feature information f' and feature amount z1 to the first decoder 61. In this way, the information processing device 100 obtains data x', which is new content (step S98).

[0166] With the configuration of FIG. 18 as well, the information processing device 100 can generate data x' that is new content reflecting the influence of the feature extracted by the extraction operation (feature information f1 in the example of FIG. 18).

[0167] [2-7. Content Variations] In the above embodiment, a song (music) is given as an example of content. However, the information processing device 100 according to a modification may perform information processing according to the present disclosure using images, text data, or the like as content.

[0168] 19 to 21, an example in which the information processing device 100 handles images as content will be described. Fig. 19 is a diagram (1) showing an example of information processing according to a sixth modified example of the present disclosure.

[0169] When handling an image as content, the information processing device 100 performs an operation of extracting high-resolution information and low-resolution information from the image, for example.

[0170] For example, the information processing device 100 learns the image 80 shown in FIG. 19 as content and generates a learned model. It is assumed that the number of pixels constituting the image 80 is "128 x 128 pixels". In this case, the image 80 is pooled in a rough range that occupies a certain size, such as "32 x 32 pixels". This results in a low-resolution image (an image obtained by mosaic processing the image 80) in which the image 80 is represented by one pixel for every "32 x 32 pixels".

[0171] Then, the information processing device 100 learns the original data of the image 80 using the first encoder 50, and learns the low-resolution image using the second encoder 55. Then, the feature z1 output from the first encoder 50 becomes a feature indicating a feature in a range finer than 32 × 32 pixels. On the other hand, the feature z2 output from the second encoder 55 becomes a feature indicating a feature in a wider range other than the feature indicated by the feature z1.

[0172] For example, a feature represented by 32×32 pixels in image 80 corresponds to a wide range of colors such as the background rather than the person. On the other hand, a feature represented by a range smaller than 32×32 pixels in image 80 corresponds to information that characterizes a finer range, such as the person's eyes, nose, and other body parts.

[0173] For this reason, if the information processing device 100 generates a plurality of second contents by varying the feature amount z2 while keeping the feature amount z1 fixed (for example, by sampling the feature amount z2 from a prior distribution while fixing the feature amount z1), it is possible to generate an image group 82 shown in Fig. 19. Each image included in the image group 82 shown in Fig. 19 is an image in which only the background color and ambient light have changed, with small features such as the eyes and nose of the person included in the image 80 remaining almost unchanged.

[0174] The image group 82 thus generated can be said to be a plurality of images of the same person captured under various backgrounds and environments. That is, according to the information processing device 100, from one image 80 capturing an image of a certain person, a large number of images equivalent to images of the same person captured under different environments can be obtained. This allows the information processing device 100 to obtain a large number of images useful for generating a machine learning model that performs a specific process (for example, a classifier for face recognition or facial expression recognition, etc.) from an extremely small number of original images. In other words, the information processing device 100 can appropriately pad the images used for learning.

[0175] Moreover, the information processing device 100 can also perform learning different from that shown in Fig. 19. An example of the different learning will be described with reference to Fig. 20. Fig. 20 is a diagram (2) showing an example of information processing according to a sixth modified example of the present disclosure.

[0176] The original data for learning is the same image 80 as in Fig. 19. In the example of Fig. 20, each pixel information of the image 80 is pooled within a range that occupies a smaller range compared to Fig. 19, such as "8 x 8 pixels".

[0177] Then, the information processing device 100 learns the original data of the image 80 using the first encoder 50, and learns the pooled image using the second encoder 55. Then, the feature z1 output from the first encoder 50 becomes a feature indicating a feature in a range finer than 8×8 pixels. On the other hand, the feature z2 output from the second encoder 55 becomes a feature indicating a feature in a wider range other than the feature indicated by the feature z1.

[0178] For example, features that are shown in an area finer than 8×8 pixels in the image 80 include information that characterizes a finer area, such as the shapes of features of a person's eyes, nose, etc.

[0179] In the example of Fig. 20, if the information processing device 100 generates a plurality of second contents by varying the feature z1 while keeping the feature z2 fixed (for example, sampling the feature z1 from a prior distribution while fixing the feature z2), it is possible to generate an image group 87 shown in Fig. 20. Each image included in the image group 87 shown in Fig. 20 is an image in which the background color and ambient light are hardly changed, and the shapes of small features such as the eyes and nose of the person included in the image 80 are changed, contrary to Fig. 19.

[0180] The image group 87 generated in this manner can be a large number of images that are useful for generating a highly accurate face recognition model, for example. That is, the image group 87 can be used to generate a model that does not mistakenly recognize the person captured in the image 80 as the person himself / herself when the person captured in the image 80 is changed in detail. This allows the information processing device 100 to improve the performance of the face recognition model.

[0181] It should be noted that the information processing device 100 can also use a different learning method. This will be described with reference to Fig. 21. Fig. 21 is a diagram (3) showing an example of information processing according to a sixth modified example of the present disclosure.

[0182] The original data for learning is the same image 80 as in FIG. 19. Here, in the example of FIG. 21, the second encoder 55 is trained using data that appeared in the intermediate layer while encoding the image 80. For example, the information processing device 100 performs a GAP (global average pooling) operation on the intermediate layer data (assuming that the data is 4×4 pixels) of the second encoder 55, and trains the second encoder 55 to output the feature z2. As a result, the feature z2 learns the image 80 from which the location information (background, etc.) has disappeared, and the other feature z1 learns the location information. In this case, by fixing the feature z1 and sampling the feature z2, it is possible to manipulate the image other than the location information. For example, as shown in FIG. 21, a group of images 92 is generated in which the general atmosphere of the background is maintained, but the faces and hairstyles of the people included in the images are variously different. In other words, the information processing device 100 can generate various images including various people with different characteristic attributes (alignment) of the faces. According to this process, the information processing device 100 can generate a large number of learning images labeled as "person" from, for example, one image.

[0183] As described above, the information processing device 100 may extract the first data by executing an extraction operation on the digital data constituting the first content, which is an image. For example, the information processing device 100 extracts the first data by performing order-invariant arithmetic processing (e.g., pooling) on ​​pixel information of each pixel included in a specific area of ​​the image as the extraction operation.

[0184] In this way, even when the content is not a song but an image, the information processing device 100 can perform learning by isolating the features targeted by the user. Furthermore, the information processing device 100 can automatically generate an image that maintains the features specified by the user, such as an image in which a certain feature (such as the background or a person's face) is fixed and other features are changed, by using the generated trained model.

[0185] Furthermore, the information processing device 100 may use text data, video, etc. as content. That is, the information processing device 100 can perform learning to separate a targeted feature amount as long as it is possible to learn information extracted by some operation and the original content data using separate encoders, regardless of the type of content.

[0186] Furthermore, the information processing device 100 may apply the information processing according to the present disclosure to perform processing for detecting substances that have similar tastes based on human taste information. For example, the information processing device 100 can learn features that separate only the characteristics of a specific taste from a certain substance by separating elements having a certain taste from a certain substance and inputting the information that has been digitized for each into separate encoders.

[0187] In the above embodiment, an example was shown in which the information processing device 100 performs an operation to extract feature information by performing order-invariant calculations or by losing certain information. However, the extraction operation is not limited to this example. For example, the information processing device 100 learns a supervised regression function for determining which phrases have a high "excitement level" based on the "excitement level in a song" that is artificially set in advance for each bar of a song. Then, the information processing device 100 calculates the excitement level of each bar of the song to be processed based on the learning result. Then, the information processing device 100 extracts only bars whose excitement level exceeds a predetermined threshold and learns them in the second encoder 55. This allows the information processing device 100 to generate a learned model that can separate artificially set features such as the "excitement level".

[0188] That is, the information processing device 100 does not necessarily extract feature information from content by arithmetic operations, but can also extract feature information from content based on artificially set features. With this configuration, for example, features corresponding to emotions can be separated, so that content having some features felt by humans, such as content that evokes happy emotions, can be automatically generated. As an example, the information processing device 100 can automatically generate content (sentences) having features that evoke certain emotions, such as "sentences that evoke happy emotions," by applying this configuration to natural language processing.

[0189] [2-8.Other] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

[0190] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0191] Furthermore, the above-described embodiments and modifications can be appropriately combined as long as the processing contents are not contradictory.

[0192] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0193] (3. Hardware Configuration) An information device such as the information processing device 100 according to each embodiment described above is realized by a computer 1000 having a configuration as shown in FIG. 11, for example. The information processing device 100 according to the embodiment will be described below as an example. FIG. 22 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the information processing device 100. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. Each unit of the computer 1000 is connected by a bus 1050.

[0194] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processes corresponding to the various programs.

[0195] The ROM 1300 stores boot programs such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000.

[0196] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records an information processing program according to the present disclosure, which is an example of program data 1450.

[0197] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0198] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading a program or the like recorded on a predetermined recording medium. The medium may be, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0199] For example, when the computer 1000 functions as the information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes an information processing program loaded onto the RAM 1200 to realize functions of the control unit 130, etc. Also, the information processing program according to the present disclosure and data in the storage unit 120 are stored in the HDD 1400. Note that the CPU 1100 reads and executes the program data 1450 from the HDD 1400, but as another example, these programs may be obtained from other devices via an external network 1550.

[0200] The present technology can also be configured as follows. (1) an extraction unit that extracts first data from elements that constitute the first content; a model generation unit that generates a trained model having a first encoder that calculates a first feature amount that is a feature amount of the first content and a second encoder that calculates a second feature amount that is a feature amount of the extracted first data; An information processing device comprising: (2) The model generation unit generating the trained model having a decoder that decodes data corresponding to the first content based on the first feature amount and the second feature amount; The information processing device according to (1). (3) The extraction unit is Accepting a user's designation of an operation on the first content and a range within which the operation is to be performed, and executing the operation within the accepted range. The information processing device according to (1) or (2). (4) The extraction unit is As the operation, an order-invariant arithmetic process is performed on the elements included in the range to extract the first data. The information processing device according to (3). (5) The extraction unit is The first data is extracted by losing information that is not time-dependent or order-dependent among the elements included in the range. The information processing device according to (3) or (4). (6) The extraction unit is performing said operation on digital data constituting said first content, which is a song, to extract said first data; The information processing device according to any one of (3) to (5). (7) The extraction unit is An operation is performed to lose information about the pitch of the sound information included in the range in the first content, and the first data is extracted. The information processing device according to (6). (8) The extraction unit is performing said operation on digital data constituting said first content, said first content being an image, to extract said first data; The information processing device according to any one of (3) to (5). (9) The extraction unit is As the operation, the first data is extracted by performing order-invariant arithmetic processing on pixel information of each pixel included in a specific region of the image. The information processing device according to (8). (10) a content generation unit that generates a second content, which is a new content having the same format as the first content, by inputting a value obtained by combining either a feature quantity output from the first encoder or the second encoder that has been trained, or a feature quantity generated based on the feature quantity, to the decoder; The information processing device according to any one of (2) to (9), further comprising: (11) The content generation unit The second content is generated by inputting a combined value of the feature quantity output from the trained first encoder and the feature quantity output from the trained second encoder to the decoder. The information processing device according to (10). (12) The content generation unit Among the combined values, a value corresponding to any one of the feature quantities is fixed, and values ​​corresponding to the other feature quantities are sequentially varied to sequentially generate a plurality of the second contents. The information processing device according to (10) or (11). (13) The computer Extracting first data from elements constituting the first content; A trained model is generated that includes a first encoder that calculates a first feature amount that is a feature amount of the first content, and a second encoder that calculates a second feature amount that is a feature amount of the extracted first data. Information processing methods. (14) Computer, an extraction unit that extracts first data from elements that constitute the first content; a model generation unit that generates a trained model having a first encoder that calculates a first feature amount that is a feature amount of the first content and a second encoder that calculates a second feature amount that is a feature amount of the extracted first data; An information processing program that functions as a (15) The model generation unit A trained model is generated, the trained model having a first decoder that decodes data corresponding to the first content based on the first feature amount and the second feature amount, and a second decoder that decodes data corresponding to the first content based on the second feature amount. The information processing device according to any one of (1) or (3) to (12). (16) The model generation unit A trained model is generated, the trained model having a first decoder that decodes data corresponding to the first content based on the first feature and the first data, and a second decoder that decodes data corresponding to the first data based on the second feature. The information processing device according to any one of (1) or (3) to (12). [Explanation of symbols]

[0201] 100 Information processing device 110 Communications Department 120 Storage section 121 Model Memory Section 122 Song data storage section 130 Control section 131 Extraction part 132 Model Generation Unit 133 Acquisition Department 134 Content Generation Division

Claims

1. an acquisition unit that, when receiving from a user a designation regarding a first feature among features included in a first content, acquires, via a trained encoder trained to represent the features of the content, a first feature amount including an expression of the first feature and a second feature amount including an expression of a second feature that is a feature other than the first feature; a content generating unit that, when acquiring another content to be processed, generates a second content, which is a new content having the same format as the first content, based on a third feature amount including an expression corresponding to the second feature and including an expression of a third feature that is a feature of the other content, and based on the first feature amount; An information processing device comprising:

2. The first feature amount is Regarding a time-dependent or order-dependent feature amount among the features included in the first content, The information processing device according to claim 1 .

3. The third feature amount is Regarding a feature amount that is not time-dependent or order-dependent among the features included in the second content, The information processing device according to claim 1 .

4. The first feature amount is A feature amount relating to rhythm contained in the first content, The information processing device according to claim 2 .

5. The third feature amount is A feature amount other than the rhythm among the features included in the second content, The information processing device according to claim 3 .

6. The content generation unit generating the second content by inputting the first feature amount and the third feature amount to a trained decoder; 6. An information processing apparatus according to claim 1.

7. The acquisition unit is acquiring the second feature amount extracted by inputting the first content, on which a predetermined calculation has been performed based on the designation, into the trained encoder; 6. An information processing apparatus according to claim 1.

8. The acquisition unit is acquiring the third feature amount extracted by inputting the other content, on which a predetermined calculation has been performed based on the designation, into the trained encoder; 6. An information processing apparatus according to claim 1.

9. The content generation unit generating a third content, which is a new content having the same format as the second content, by fixing the first feature amount and arbitrarily changing the data corresponding to the third feature amount and inputting the changed data to the trained decoder; The information processing device according to claim 6.

10. The computer When a designation regarding a first feature among features included in a first content is received from a user, a first feature amount including a representation of the first feature and a second feature amount including a representation of a second feature that is a feature other than the first feature are acquired via a trained encoder trained to represent the features of the content; when acquiring another content to be processed, generating a second content, which is a new content having the same format as the first content, based on a third feature amount including an expression corresponding to the second feature and including an expression of a feature of the other content, and the first feature amount; Information processing methods.

11. Computer, an acquisition unit that, when receiving from a user a designation regarding a first feature among features included in a first content, acquires, via a trained encoder trained to represent the features of the content, a first feature amount including an expression of the first feature and a second feature amount including an expression of a second feature that is a feature other than the first feature; a content generating unit that, when acquiring another content to be processed, generates a second content, which is a new content having the same format as the first content, based on a third feature amount including an expression corresponding to the second feature and including an expression of a feature of the other content, and the first feature amount; An information processing program that functions as a

Images