Information processing device, information processing method, and information processing program

Abstract

According to the information processing device (100) of the present disclosure, it includes: an acquisition unit (132), which uses a trained encoder to acquire feature values ​​of each partial data constituting the first content; a calculation unit (133), which calculates relative feature values ​​as feature values ​​between partial data from the feature values ​​of each acquired partial data to calculate a relative feature value sequence indicating features of the construction of the first content; and a generation unit (134), which generates the second content based on the relative feature value sequence of the first content and the feature values ​​of the optional data.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, an information processing method and an information processing program, and more particularly to a generation process of a training model generated by machine learning. Background Art

[0002] Information processing using machine learning has been used in various technical fields. For example, by using a neural network representing the mechanism of the brain nervous system to learn the characteristics of content (images, music, etc.), it is possible to automatically generate new content, for example.

[0003] For example, a technology is disclosed in which, by learning the features of existing songs, an appropriate song suitable for lyrics can be automatically created without the user inputting parameters other than lyrics.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2011-175006 Summary of the invention

[0007] Technical issues

[0008] According to the conventional technology, language feature values ​​calculated from lyrics data indicating lyrics of each song and attribute data indicating attributes of the song are learned, and therefore, when new lyrics data is provided, a song suitable for the new lyrics data can be automatically generated.

[0009] However, according to the conventional technology, a melody or code is simply generated based on the lyrics, and in terms of the structure (structure) of the entire song, a natural song like a song composed by a person is not always generated. That is, according to the conventional technology, it is difficult to learn the characteristics of the overall structure of the content or to generate new content while maintaining the overall structure.

[0010] Therefore, the present disclosure provides an information processing device, an information processing method, and an information processing program with which the characteristics of the overall structure of content can be learned.

[0011] Solution to the problem

[0012] In order to solve the above-mentioned problem, an information processing device includes: an acquisition unit, which uses a trained encoder to acquire feature values ​​of each partial data forming a first content; a calculation unit, which calculates relative feature values ​​as relative feature values ​​between partial data from the acquired feature values ​​of each partial data to calculate a relative feature value sequence indicating features of the structure of the first content; and a generation unit, which generates second content based on the relative feature value sequence of the first content and the feature values ​​of optional data. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a diagram illustrating an example of information processing according to an embodiment of the present disclosure.

[0014] Figure 2 is a diagram showing an example of a configuration of an information processing device according to an embodiment of the present disclosure.

[0015] Figure 3 is a diagram illustrating an example of a song data storage unit according to an embodiment of the present disclosure.

[0016] Figure 4 is a flowchart (1) showing the steps of information processing according to an embodiment of the present disclosure.

[0017] Figure 5 is a flowchart (2) showing the steps of information processing according to an embodiment of the present disclosure.

[0018] Figure 6 is a hardware configuration diagram showing an example of a computer that executes the function of the information processing apparatus. DETAILED DESCRIPTION

[0019] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings. In addition, in each of the following embodiments, the same components are represented by the same reference numerals, and repeated descriptions are omitted.

[0020] The present disclosure is described according to the order of the parts described below.

[0021] 1. Example

[0022] 1-1. Overview of information processing according to an embodiment of the present disclosure

[0023] 1-2. Configuration of information processing device according to the present embodiment

[0024] 1-3. Steps of information processing according to this embodiment

[0025] 2. Other embodiments

[0026] 3. Hardware Configuration

[0027] (1. Example)

[0028] [1-1. Overview of information processing according to an embodiment of the present disclosure]

[0029] Figure 1 1 is a diagram showing an example of information processing according to an embodiment of the present disclosure. Figure 1 The information processing device 100 shown executes.

[0030] The information processing device 100 is a device that performs information processing according to the present disclosure, and is, for example, an information processing terminal or a server device.

[0031] According to the present embodiment, the information processing device 100 includes a training model for extracting features of content. According to the present embodiment, the content includes digital data in a predetermined format, such as music (songs), images, or moving images. Figure 1 In the illustrated example, the information processing apparatus 100 uses songs as an example of content for processing.

[0032] The training model according to the present embodiment includes: an encoder that extracts feature values ​​from data forming the content; and a decoder that reconstructs the content from the extracted feature values. For example, the information processing device 100 trains the encoder through unsupervised learning such as VAE (Variational Autoencoder) or GAN (Generative Adversarial Network). Specifically, the information processing device 100 inputs the content to the encoder, reconstructs the content from the extracted feature values, compares the original content with the reconstructed content, and adjusts the parameters of the encoder and the decoder. The information processing device 100 repeats the process to optimize the parameters of the encoder and the decoder, thereby generating a training model. This allows the information processing device 100 to generate a training model that can obtain appropriate feature values ​​from the data forming the content. In addition, the feature values ​​are represented by using, for example, a vector with a lower dimension compared to the data of the input content.

[0033] Furthermore, the training model is not limited to the above example, but may be any type of model as long as feature values ​​can be extracted and content can be reconstructed from the extracted feature values.

[0034] exist Figure 1In the example of , the information processing device 100 inputs a song (in other words, digital data representing sounds forming the song) to the encoder to extract elements representing features of the song (such as a melody line, constituent sounds, rhythm (a time structure of music, such as the number of included notes and rests, or the order in which sounds are reproduced), tempo, or beat) as feature values ​​of the song. In addition, according to the present embodiment, data on the song is represented by using data indicating pitch (data indicating the level at which sound is generated), data indicating how long a generated note is to be retained, and data (e.g., in a vector format) of the timing of rests.

[0035] Here, for the automatic generation of new songs using the feature values ​​of existing songs as described above, one of the challenges may be to generate songs that have a natural structure like artificial existing songs. However, although extracting the feature values ​​of an optional song without changing it can reproduce the characteristics of the song, it will make it difficult to generate a song that has a natural structure as a whole. Specifically, even if a similar characteristic melody line or sound structure (also called a theme (motif)) that is part of the song is reproduced in the automatically generated song, it is difficult to reproduce the structure of the song itself, such as how the characteristics of the theme change throughout the song. That is, it is conventionally difficult to automatically generate a song that has a natural structure like an artificial existing song and has a specific length like an existing song.

[0036] Therefore, through the information processing described below, the information processing device 100 according to the present disclosure can calculate the feature value indicating the structure of the entire song, and automatically generate a natural song by using the feature value. Specifically, the information processing device 100 uses the above-mentioned trained encoder to obtain the feature value of each partial data (e.g., each measure) forming the song. In addition, the information processing device 100 calculates a relative feature value from the feature value of each partial data, which is a relative feature value between the partial data, so as to calculate a relative feature value sequence indicating the feature of the structure of the song. As described above, the information processing device 100 regards a song with a certain length as a sequence of arranged partial data, calculates their relative feature values ​​to obtain the feature value that is transformed in the entire song, and then expresses the structure of the entire song. In other words, the information processing device 100 models the structure of the entire song by using a relative feature value sequence. Then, the information processing device 100 generates a new song using a relative feature value sequence indicating the feature of the structure of the entire song. Therefore, the information processing device 100 can automatically generate a new song that maintains the features of the structure of the existing song and has a natural structure. In addition, the sequence of feature values ​​is a sequence of feature values ​​arranged in order for the individual partial data forming the song.

[0037] refer to Figure 1, the following describes in order an overview of information processing according to the present disclosure. In addition, in the following description, the target content from which the feature value is calculated is referred to as "first content", and the new content generated based on the relative feature value sequence of the first content is sometimes referred to as "second content". In addition, Figure 1 The trained encoders and the like shown are simply referred to as “encoders.” Also, the trained decoders are simply referred to as “decoders.”

[0038] like Figure 1 As shown, the information processing device 100 first acquires the song 30 as the first content and divides the song 30 into partial data (step S1). For example, the information processing device 100 divides the song 30 on a per-bar basis. Figure 1 The example in shows that song 30 has six bars, but song 30 can have more bars.

[0039] The song 30 includes a symbol string (digital data) indicating, for example, a pitch, a length, or a rest. For example, the pitch indicates the frequency indicating the pitch by using a predetermined level (e.g., 128 levels). In addition, the length indicates the length of the reproduced sound being retained. In addition, the rest indicates the timing of stopping the sound reproduction. In addition, the data representing the song 30 may include information such as a symbol indicating the beat or rhythm of the song 30 or the interruption of a measure, a code at a specific timing, or a constituent sound forming a code.

[0040] For example, the information processing device 100 converts the above-mentioned string of symbols for manipulation in the model. For example, the information processing device 100 uses the vector representation of the above-mentioned string of symbols (for example, an embedding vector having a pitch, a length, etc. assigned to each dimension and representing sound information) for processing. The embedding vector is, for example, a d-dimensional (d is an arbitrary integer) vector, in which "1" is input to the dimension corresponding to the corresponding sound (for example, information indicating the pitch level such as "C4"), and "0" is input to other dimensions. In addition, the data representing the sound can be represented in MIDI (Musical Instrument Digital Interface) (registered trademark) format, can be digital data having a known format and reproducible by a general sequencer, or can be represented as waveform data such as WAV format.

[0041] The information processing device 100 may use various known techniques to divide the song 30 into partial data. For example, the information processing device 100 detects the interruption of a preset measure to divide the song 30 into partial data (measures). Alternatively, the information processing device 100 may divide the song 30 into partial data based on the relationship between beats and notes. For example, in the case of a song 30 having a 4 / 4 time signature, the information processing device 100 detects the time when four sounds having a length corresponding to a quarter note are reproduced as an interruption to divide the song 30 into measures.

[0042] In addition, the information processing device 100 may use breaks other than measures for the partial data. For example, the information processing device 100 may detect a break in the melody line of the song 30 (e.g., a position where a rest having a length greater than a predetermined threshold appears) to divide it into partial data. In this case, the partial data does not necessarily need to match the measures.

[0043] The information processing device 100 divides the song 30 into measures in step S1 and extracts each section data (step S2). Figure 1 In the example, each part of the data is shown as "x n (n is an arbitrary natural number)". For example, "x1" indicates data included in the first measure of song 30.

[0044] Then, the information processing device 100 sequentially inputs the extracted partial data to the encoder 50 (step S3). Thus, the information processing device 100 acquires a feature value of each partial data (step S4).

[0045] exist Figure 1 In the example, the eigenvalue of each part of the data is represented by "z n ” indicates. For example, “z1” indicates the feature value of the first measure of song 30.

[0046] Here, the information processing device 100 calculates a relative feature value as a relative feature value between the partial data from the feature value of each partial data obtained (step S5). For example, as a relative feature value between the partial data, the information processing device 100 subtracts the feature value of a specific common partial data from the feature value of each partial data to calculate the relative feature value.

[0047] exist Figure 1 In the example of , the information processing device 100 subtracts "z1" which is the feature value of the first measure from the feature value of each partial data to calculate the relative feature value. In addition, this calculation is an example, and the information processing device 100 may perform addition, multiplication, or division on the feature value to calculate the relative feature value. In addition, the information processing device 100 may subtract partial data of a different measure from each partial data instead of "z1".

[0048] The information processing device 100 calculates a relative feature value sequence based on the calculated relative feature values, which is a sequence of feature values ​​indicating the structure of the song 30 (step S6). The relative feature value sequence is, for example, a sequence of relative feature values ​​arranged in order so that they correspond to the structure of the song 30. Figure 1 In the example, n+1 -z1" is represented by "rn " means. That is, in Figure 1 In the example of , the relative feature value sequence corresponding to song 30 is expressed as “(r1, r2, r3, r4, r5)”.

[0049] As described above, during the process of step S6, the information processing device 100 calculates a relative feature value sequence indicating the structure of the song 30. Then, the information processing device 100 acquires optional information as a theme of a new song (second content) to be generated.

[0050] For example, the information processing device 100 acquires data included in the first measure of the predetermined song 35 as information on the theme of the new song to be generated (step S7). In addition, as information on the theme of the new song to be generated, the information processing device 100 does not always need to acquire data included in one measure, but may acquire data on the entire optional song, for example.

[0051] exist Figure 1 In the example of , the obtained optional data (ie, the data included in the first measure of song 35) is represented by "x a "express.

[0052] Then, the information processing device 100 will a ” is input to the encoder 50 (step S8). Therefore, the information processing device 100 obtains a "The corresponding eigenvalue. Figure 1 In the example, a The corresponding eigenvalue of "z a "express.

[0053] Then, the information processing device 100 obtains the feature value “z a " and the relative feature value indicating the structure of the song 30 to generate a feature value sequence corresponding to the newly generated second content (step S9). For example, the information processing device 100 generates a feature value sequence in which "z a " is at the beginning and "z a " is attached to each relative feature value of the song 30. Specifically, the information processing device 100 generates a feature value sequence having a structure such as "(z a , z a +r1,z a +r2,z a +r3,z a +r4,z a That is, the information processing device 100 generates a feature value sequence in which “z a ” also adds to the characteristics of the structure of song 30.

[0054] The information processing device 100 inputs the feature value sequence generated at step S9 to the decoder 60 (step S10). The decoder 60 is a decoder trained to reconstruct content based on the feature values ​​extracted by the encoder 50. Figure 1 In the example of , the decoder 60 reconstructs the song (precisely, the digital data for reproducing the sound) based on the feature values ​​extracted by the encoder 50 .

[0055] The information processing device 100 acquires the sound data of each measure corresponding to the feature value sequence from the output of the decoder 60. The information processing device 100 arranges the acquired data in the order of the sequence to generate the song 40 (step S11). The song 40 is a song that uses the first measure of the song 35 as a theme and also maintains the characteristics of the structure of the song 30.

[0056] Therefore, the information processing device 100 according to the present disclosure uses the encoder 50 to obtain the first content ( Figure 1 Then, the information processing device 100 calculates the relative feature value of each partial data as the relative feature value between the partial data based on the acquired feature value of each partial data to calculate a relative feature value sequence indicating the feature of the structure of the first content. That is, the information processing device 100 obtains a sequence in which the feature values ​​of the partial data forming the first content are arranged in sequence, instead of calculating the feature value of the first content itself. Therefore, the information processing device 100 can extract the structure of the first content (if the first content is a song, the flow of the entire song corresponding to the time direction, excitement points, etc.) as a feature.

[0057] In addition, the information processing device 100 according to the present disclosure generates a feature value sequence based on the relative feature value sequence of the first content and the feature value of the optional data (“z a ",exist Figure 1 In the example, the feature value of the first section of song 35) is used to generate the second content (in Figure 1 That is, the information processing device 100 generates a new feature value sequence based on the feature value sequence indicating the structure of the first content and the feature value of the new data, and reconstructs the content based on the new feature value sequence. Therefore, the information processing device 100 can generate a new song as the second content, which maintains the structure of the first content and adopts a new theme or constituent sound. In addition, Figure 1In the example shown, the information processing device 100 subtracts the feature value "z1" from the feature value of each partial data to calculate the relative feature value. However, this example is not limiting, and the information processing device 100 can calculate the relative feature value by using a feature value extractor (such as addition, subtraction, multiplication, division, or coherence) that extracts the feature value. In addition, the information processing device 100 can extract the graph structure based on the similarity or causal relationship of the feature value of each partial data, and calculate the relative feature value sequence for the graph data by using machine learning technology, etc.

[0058] [1-2. Configuration of Information Processing Device According to the Present Embodiment]

[0059] Next, a configuration of the information processing apparatus 100 as an example of an information processing apparatus that performs information processing according to the present embodiment is described. Figure 2 is a diagram showing an example of the configuration of the information processing apparatus 100 according to an embodiment of the present disclosure.

[0060] like Figure 2 As shown, the information processing device 100 includes a communication unit 110, a storage unit 120, and a control unit 130. In addition, the information processing device 100 may include an input unit (e.g., a keyboard or a mouse) for receiving various operations from an administrator or the like who manages the information processing device 100, and a display unit (e.g., a liquid crystal display) for displaying various information.

[0061] The communication unit 110 is realized by using, for example, a NIC (Network Interface Card). The communication unit 110 is connected to a network N such as the Internet via wired or wireless communication to transmit / receive information to / from other devices or the like via the network N.

[0062] The storage unit 120 is implemented by using a semiconductor memory device such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 120 includes a model storage unit 121 and a song data storage unit 122.

[0063] The model storage unit 121 stores a pre-trained training model. Specifically, the model storage unit 121 includes an encoder 50 that extracts feature values ​​of content and a decoder 60 that reconstructs content. In addition, the model storage unit 121 may store training data such as content used for training.

[0064] The song data storage unit 122 stores data on the content (song) input to the model. Figure 3 An example of the song data storage unit 122 according to the present embodiment is shown. Figure 3 is a diagram showing an example of the song data storage unit 122 according to an embodiment of the present disclosure. Figure 3 In the example shown, the song data storage unit 122 has items such as “song ID”, “part data ID”, “pitch information”, “note length rest information”, “code information”, and “rhythm information”.

[0065] "Song ID" is identification information for identifying a song. "Part data ID" is identification information for identifying part data. Part data corresponds to, for example, one or more bars forming a song.

[0066] "Pitch information" indicates information about the pitch (scale) of the sound included in the partial data. "Note length rest information" indicates the length of the sound included in the partial data (reproduction time period or number of beats to be reproduced), and the length and timing of the rest. "Code information" indicates the type of code included in the partial data, the constituent sounds of the code, the switching of the code within a measure, etc. "Rhythm information" indicates the beat or rhythm in the measure, the position of the strong beat and the weak beat, etc.

[0067] although Figure 3 Although there is a conceptual description in which an item such as pitch information is "C01", each item actually stores specific data indicating a sound as described above. Although, for the purpose of description, Figure 3 An example is shown in which "pitch information", "note length rest information", etc. are stored as different items, but these information may also be collectively stored in one item, etc. as information indicating the notes included in the measure. That is, the data format representing the song is not limited to Figure 3 The data format is shown, and can be in any format that can be manipulated in the model.

[0068] For example, Figure 3 The example shown indicates that the song identified with the song ID "A01" includes the partial data identified with the partial data IDs "B01" and "B02." In addition, it is indicated that the partial D data with the partial data ID "B01" includes the sound data represented with the pitch information "C01", the length rest information "D01", the code information "E01" and the rhythm information "F01".

[0069] Return to reference Figure 2 , the description is continued. When, for example, a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) uses a RAM (Random Access Memory) or the like as a work area and executes a program (for example, an information processing program according to the present disclosure) stored in the information processing device 100, the control unit 130 is implemented. In addition, the control unit 130 is a controller and can be implemented by using an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0070] like Figure 2 As shown, the control unit 130 includes a training unit 131, an acquisition unit 132, a calculation unit 133, and a generation unit 134 to implement or execute the functions and operations for the following information processing. In addition, the internal configuration of the control unit 130 is not limited to Figure 2 The configuration shown, and the internal configuration may be a different configuration as long as the configuration allows the information processing described later.

[0071] The training unit 131 performs a predetermined training process using the content as training data to generate a training model.

[0072] For example, in order to learn sound data about a song or the like, the training unit 131 inputs the sound data to the encoder 50 to extract feature values ​​of the sound data. Subsequently, the training unit 131 inputs the feature values ​​of the sound data to the decoder 60 to reconstruct the sound data that has been input to the encoder 50. Then, the training unit 131 adjusts the parameters of the encoder 50 and the decoder 60 so as to reduce the difference between the continuous sound data. The training unit 131 repeats this process to generate a training model using the optimized encoder 50 and the decoder 60. As described above, the training unit 131 can generate a model by using various known techniques such as VAE or GAN.

[0073] The acquisition unit 132 acquires various types of information. For example, the acquisition unit 132 acquires the first content to be input to the model trained by the training unit 131.

[0074] In addition, the acquisition unit 132 divides the acquired first content to acquire partial data forming the first content. For example, when the first content is a song, the acquisition unit 132 detects the interruption of a measure of the song by using the above-mentioned technology and sets the detected measure as partial data.

[0075] Alternatively, the acquisition unit 132 can detect a rest of a length greater than a predetermined threshold in the first content, and divide the song into partial data based on the detected rest. In this case, the length of the predetermined threshold can be a time length, a percentage of the rest in a measure, etc.

[0076] Furthermore, the acquisition unit 132 acquires a feature value of each partial data forming the first content using the encoder 50 trained by the training unit 131 .

[0077] Specifically, the acquisition unit 132 inputs a symbol string indicating pitch, duration, and rest as data indicating the sound included in the partial data to the encoder 50 to acquire a feature value corresponding to the partial data. The feature value is represented as a vector having a lower dimension than the vector representing the original partial data, for example.

[0078] The calculation unit 133 calculates relative feature values, which are relative feature values ​​between the partial data, from the feature value of each partial data acquired by the acquisition unit 132 to calculate a relative feature value sequence indicating the features of the structure of the first content.

[0079] For example, the calculation unit 133 performs addition, subtraction, multiplication or division on the characteristic value of the specific partial data of the first content and the characteristic value of each partial data other than the specific partial data to calculate the relative characteristic value, so as to calculate the relative characteristic value sequence. The specific partial data of the first content indicates specific partial data among the partial data forming the first content, and Figure 1 In the example of , it corresponds to the feature value "z1" of the first measure of song 30. Therefore, the calculation unit 133 performs an operation such as addition, subtraction, multiplication or division on a plurality of partial data having specific partial data to calculate relative feature values ​​having a relationship.

[0080] also, Figure 1 An example is shown in which the feature value "z1" of the first measure of the song 30 is subtracted from the feature value of each partial data of the song 30; however, the operation is not limited to this example. For example, the calculation unit 133 may calculate a relative feature value between the feature value of the specific partial data of the first content and the feature value of each partial data other than the specific partial data and having a causal relationship with the specific partial data, so as to calculate a relative feature value sequence.

[0081] Here, partial data having a causal relationship with specific partial data refers to partial data having a certain corresponding relationship with the specific partial data. For example, if the content is a song, there may be a bar in the song corresponding to a specific bar (for example, called a call-response relationship). In this case, as described above, the calculation unit 133 may preferably perform an addition (or subtraction) operation on the feature value of the bar with a certain causal relationship in order to emphasize or weaken the structure of the song. In addition, the causal relationship between the partial data of the content can be obtained by using a technique in which the song is analyzed by using a known machine learning technique such as causal inference, and the relationship between the specific bar and the factor bar is quantified.

[0082] Furthermore, the calculation unit 133 may calculate the relative feature value by using various known techniques, such as generating partial data in the content as a graph based on a predetermined relationship, and adding or subtracting partial data having a close relationship in the graph.

[0083] As described above, the calculation unit 133 calculates the relative feature value as the relative feature value between the partial data by using various techniques and arranges the calculated relative feature values ​​in order to calculate the relative feature value sequence indicating the feature of the structure of the first content. The calculation unit 133 sends the calculated relative feature value to the generation unit 134.

[0084] The generating unit 134 generates the second content based on the relative feature value sequence of the first content and the feature value of the optional data.

[0085] For example, the generating unit 134 calculates a new feature value sequence based on the relative feature value sequence of the first content and the feature value of the optional data calculated by the calculating unit 133. Then, the generating unit 134 inputs each feature value included in the new feature value sequence to the decoder 60, and reconstructs the partial data corresponding to each feature value. In addition, the generating unit 134 combines the reconstructed partial data in the order of the sequence to generate the second content as the new content.

[0086] When the relative feature value sequence of the first content as a song has been acquired, the generation unit 134 can generate an optional new song as the second content. In this case, the acquisition unit 132 inputs a symbol string indicating pitch, length, and rest as data indicating the sound included in the partial data and the optional data to the encoder 50 to acquire feature values ​​corresponding to the partial data and the optional data. Then, based on the feature values ​​acquired by the acquisition unit 132, the calculation unit 133 calculates the relative feature value sequence corresponding to each partial data. The generation unit 134 generates a new feature value sequence based on the relative feature value sequence calculated by the calculation unit 133 and the feature value corresponding to the optional data (for example, the sound data as the theme of the new content), and generates the second content as a song based on the generated new feature value sequence.

[0087] [1-3. Steps of information processing according to the present embodiment]

[0088] Next, refer to Figure 4 and Figure 5 Describe the steps of information processing according to this embodiment. First, refer to Figure 4 The flow of a training process according to an embodiment of the present disclosure is described. Figure 4 is a flowchart (1) showing the steps of information processing according to an embodiment of the present disclosure.

[0089] like Figure 4 As shown, the information processing device 100 determines whether training data (content) has been acquired (step S101). When training data has not been acquired (step S101; No), the information processing device 100 stands by until the training data is acquired.

[0090] On the contrary, when the training data has been acquired (step S101; yes), the information processing device 100 generates a model using the training data (step S102). Then, the information processing device 100 stores the trained model (encoder and decoder) in the storage unit 120 (step S103).

[0091] Next, refer to Figure 5 The flow of the generation process according to the embodiment of the present disclosure is described. Figure 5 is a flowchart (2) showing the steps of information processing according to an embodiment of the present disclosure.

[0092] like Figure 5 As shown, the information processing device 100 determines whether the first content has been acquired (step S201). When the first content has not been acquired (step S201; No), the information processing device 100 stands by until the first content is acquired.

[0093] On the contrary, when the first content has been acquired (step S201; Yes), the information processing device 100 divides the first content into partial data (step S202). For example, when the first content is a song, the information processing device 100 divides the song into partial data for each measure.

[0094] Subsequently, the information processing device 100 inputs each partial data to the encoder 50 to calculate the feature value of each partial data (step S203). Furthermore, the information processing device 100 performs a predetermined operation on the feature value of the partial data to calculate a relative feature value (step S204).

[0095] Based on the relative feature values, the information processing apparatus 100 calculates a relative feature value sequence indicating features of the structure of the first content (step S205 ).

[0096] Then, the information processing device 100 determines whether the optional data has been acquired, which is the theme of the second content to be generated, etc. (for example, a section of a certain song) (step S206). When the optional data is not acquired (step S206; No), the information processing device 100 stands by until the optional data is acquired.

[0097] In contrast, when the optional data has been acquired (step S206; Yes), the information processing device 100 inputs the optional data to the encoder 50 to calculate the feature value of the optional data (step S207).

[0098] Subsequently, the information processing device 100 calculates a new feature value sequence as a source of the second content based on the relative feature value sequence of the first content and the feature values ​​of the optional data (step S208).

[0099] Then, the information processing device 100 inputs the new feature value sequence to the decoder 60 to generate second content from the new feature value sequence (step S209 ).

[0100] (2. Other embodiments)

[0101] The processing according to each of the above-described embodiments may be implemented in various different embodiments other than the above-described embodiments.

[0102] According to the above-described embodiment, examples of content include songs (music). However, the information processing device 100 may perform information processing according to the present disclosure using text data, moving images, and the like as content.

[0103] For example, the information processing device 100 acquires the feature value of each partial data forming the first content (which is text data). Then, the information processing device 100 generates the second content as text data based on the relative feature value sequence of the first content and the feature value of the optional data (which is text data), the relative feature value sequence of the first content being obtained based on the acquired feature value of each partial data. In this case, it is assumed that the information processing device 100 generates a training model for outputting the feature value of the text data.

[0104] In this case, examples of text data include poetry or a thirty-one-syllable poem. For example, the information processing device 100 acquires a poem including a plurality of sentences (e.g., text divided by line break symbols). Then, the information processing device 100 detects line break symbols included in the poem to divide the poem into partial data (sentences per line). Subsequently, the information processing device 100 calculates relative feature values ​​of the partial data to obtain a relative feature value sequence corresponding to the poem.

[0105] When acquiring new optional data (e.g., a desired phrase, sentence, etc. that the user wants to include in a poem), the information processing device 100 calculates feature values ​​of the optional data, and calculates a new feature value sequence based on the calculated feature values ​​and a relative feature value sequence corresponding to the poem. Then, the information processing device 100 generates a new poem as the second content from the new feature value sequence.

[0106] This allows the information processing device 100 to generate a new poem having a structure similar to that of an existing poem (eg, the number of characters (rhythm), factor (phoneme) at a break, or the expression of a phrase appearing in the next sentence).

[0107] In addition, the information processing device 100 may generate new moving image content based on moving image content instead of text data. In this case, it is assumed that the information processing device 100 generates a trained model that outputs feature values ​​of an image forming the moving image content.

[0108] For example, the information processing device 100 acquires a feature value of each partial data forming a first content as a moving image content. In this case, the partial data is, for example, an image corresponding to each frame forming the moving image content. In addition, the partial data may be, for example, image data obtained by collecting and averaging several frames, and a single still image. Then, the information processing device 100 generates a second content as a moving image content based on a relative feature value sequence of the first content, the relative feature value sequence of which is obtained based on the feature value of each partial data acquired, and a feature value of optional data as a moving image or moving image content.

[0109] This allows the information processing device 100 to generate new moving image content having a structure similar to that of existing moving image content. For example, based on moving image content capturing a series of movements of a blooming flower, moving image content capturing moving images of a walking person, etc., the information processing device 100 generates new moving image content of different objects performing similar actions. Therefore, the information processing device 100 can automatically generate various types of new content having a similar content structure as a whole, not limited to songs, by information processing according to the present disclosure.

[0110] In addition, the information processing device 100 can apply information processing according to the present disclosure to perform processing for detecting, for example, a substance that causes a similar taste based on human taste information. For example, with respect to human taste, it is assumed that if a human perceives elements in the order of a first element, a second element, and then a third element, he tends to feel a specific taste (e.g., sweetness). In this case, the information processing device 100 can calculate the relative characteristic value of a substance having a structure that causes a human to perceive elements in the order of a first element, a second element, and then a third element, thereby generating a new substance having a similar structure.

[0111] In addition, in the example described according to the above embodiment, the song 30 is divided into sections to calculate the feature value sequence; however, for example, the information processing device 100 may regard multiple songs as the first content, and regard each song as a partial data to calculate the feature value sequence indicating the song. In this case, the information processing device 100 can express features such as the structure of the sequence of songs. Specifically, as the order of the sequence of songs, the information processing device 100 expresses the characteristics of the structure in a manner such that positive songs (e.g., fast-paced songs) are in front and relatively negative songs are in the back. As described in the above embodiment, the information processing device 100 can transform the characteristics of the structure into different content. Therefore, for example, when rearranging other songs, the information processing device 100 can generate a list having a structure similar to the structure of the first content (i.e., a playlist in which the order of the songs is automatically rearranged).

[0112] In addition, among the processes described in each of the above embodiments, all or some of the processes automatically performed as described above may be performed manually, or all or part of the processes manually performed as described above may be performed automatically by using a known method. In addition, unless otherwise specified, the processing steps, specific names, and information including various types of data and parameters as described in the above description and the accompanying drawings may be optionally changed. For example, the various types of information shown in each figure are not limited to the information shown.

[0113] In addition, the components of each device shown are conceptual in function and do not necessarily need to be physically configured as shown in the drawings. Specifically, the specific forms of separation and combination of devices are not limited to those depicted in the drawings, and the configuration may be such that all or some of them are functionally or physically separated or combined in any unit according to various types of loads or uses.

[0114] Furthermore, the above-described embodiments and modifications may be appropriately combined as long as consistency is ensured in processing details.

[0115] Furthermore, since the advantages described in this specification are merely examples, there is no limitation and other advantages may occur.

[0116] (3. Hardware Configuration)

[0117] By using a Figure 6 The computer 1000 of the illustrated configuration realizes an information device such as the information processing device 100 according to the above-described embodiment. The information processing device 100 according to the present embodiment is described below as an example. Figure 61050 is a hardware configuration diagram showing an example of a computer 1000 that performs the functions of the information processing apparatus 100. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, an HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The units of the computer 1000 are connected via a bus 1050.

[0118] The CPU 1100 operates to control each unit based on the program stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the program stored in the ROM 1300 or the HDD 1400 into the RAM 1200 to execute processing corresponding to various programs.

[0119] The ROM 1300 stores, for example, a boot program such as BIOS (Basic Input Output System) executed by the CPU 1100 at the time of startup of the computer 1000 and programs depending on the hardware of the computer 1000 .

[0120] HDD 1400 is a computer-readable recording medium that records programs to be executed by CPU 1100 , data to be used by the programs, etc. in a non-transitory manner. Specifically, HDD 1400 is a recording medium that records an information processing program according to the present disclosure, which is an example of program data 1450 .

[0121] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (eg, the Internet). For example, the CPU 1100 receives data from various devices or transmits data generated by the CPU 1100 to various devices via the communication interface 1500.

[0122] The input / output interface 1600 is an interface for connecting the input / output device 1650 with the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. In addition, the CPU 1100 sends data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. In addition, the input / output interface 1600 can be used as a medium interface for reading a program recorded in a predetermined recording medium (medium), etc. Examples of the medium include optical recording media such as DVDs (digital versatile discs) and PDs (phase change rewritable discs), magneto-optical recording media such as MOs (magneto-optical discs), magnetic tape media, magnetic recording media, or semiconductor memories.

[0123] For example, when the computer 1000 is used as the information processing apparatus 100 according to the present embodiment, the CPU 1100 of the computer 1000 executes the information processing program loaded into the RAM 1200 to perform the functions of the control unit 130, etc. In addition, the HDD 1400 stores the information processing program and data according to the present disclosure in the storage unit 120. In addition, although the CPU 1100 reads and executes the program data 1450 from the HDD 1400, it may acquire the program from a different device via the external network 1550 according to another example.

[0124] In addition, the technology may also include the following configurations. (1)

[0126] An information processing device, comprising:

[0127] an acquisition unit, using a trained encoder to acquire a feature value of each portion of data forming the first content;

[0128] a calculation unit that calculates a relative feature value, which is a relative feature value between the partial data, from the feature value of each acquired partial data to calculate a relative feature value sequence indicating a feature of a structure of the first content; and

[0129] A generating unit generates second content based on the relative feature value sequence of the first content and the feature value of the optional data. (2)

[0131] An information processing device according to (1), wherein the calculation unit calculates the relative characteristic value by performing addition, subtraction, multiplication or division on the characteristic value of a specific partial data of the first content and the characteristic value of each partial data other than the specific partial data, so as to calculate a relative characteristic value sequence. (3)

[0133] An information processing device according to (1) or (2), wherein a calculation unit calculates relative characteristic values ​​between characteristic values ​​of a specific partial data of the first content and characteristic values ​​of each partial data other than the specific partial data and having a causal relationship with the specific partial data, so as to calculate a relative characteristic value sequence. (4)

[0135] An information processing device according to any one of (1) to (3), wherein the generation unit inputs a feature value sequence calculated based on a relative feature value sequence of the first content and feature values ​​of the optional data into a trained decoder to generate the second content. (5)

[0137] The information processing apparatus according to any one of (1) to (4), wherein the generation unit generates the optional song as the second content based on the relative feature value sequence of the first content as the song. (6)

[0139] The information processing device according to (5), wherein:

[0140] The acquisition unit inputs a symbol string indicating a pitch, a tone length, and a rest as data indicating sounds included in the partial data and in the optional data to the trained encoder to acquire feature values ​​corresponding to the partial data and the optional data, and

[0141] The generating unit inputs a relative feature value sequence obtained based on the acquired feature values ​​of the optional data and the partial data into the trained decoder to generate the second content. (7)

[0143] The information processing device according to any one of (1) to (4), wherein:

[0144] The acquiring unit acquires a feature value of each partial data forming the first content as text data, and

[0145] The generating unit generates second content as text data based on a relative feature value sequence of the first content obtained based on the acquired feature value of each partial data and a feature value of the optional data as text data. (8)

[0147] The information processing device according to any one of (1) to (4), wherein

[0148] The acquisition unit acquires a feature value of each partial data forming the first content as the moving image content, and

[0149] The generating unit generates second content as moving image content based on a relative feature value sequence of first content obtained based on the acquired feature value of each partial data and a feature value of optional data as moving image or image content. (9)

[0151] The information processing apparatus according to any one of (1) to (6), wherein the acquisition unit detects a break of a measure as the first content of the song, and acquires the feature value of each detected measure. (10)

[0153] An information processing device according to any one of (1) to (6), wherein an acquisition unit detects a rest having a length greater than a predetermined threshold value in the first content of a song, divides the song into partial data based on the detected rest, and acquires a feature value of each divided partial data. (11)

[0155] A method of information processing that enables a computer to:

[0156] Using the trained encoder, obtain a feature value of each partial data forming the first content;

[0157] calculating a relative feature value from the acquired feature value of each partial data, the relative feature value being a relative feature value between the partial data, to calculate a relative feature value sequence indicating a feature of a structure of the first content; and

[0158] The second content is generated based on the relative feature value sequence of the first content and the feature value of the optional data. (12)

[0160] An information processing program that enables a computer to:

[0161] an acquisition unit, using a trained encoder to acquire a feature value of each portion of data forming the first content;

[0162] a calculating unit that calculates relative feature values ​​as relative feature values ​​between the partial data based on the acquired feature values ​​of the partial data to calculate a relative feature value sequence indicating features of the structure of the first content; and

[0163] A generating unit generates second content based on the relative feature value sequence of the first content and the feature value of the optional data.

[0164] Reference Symbols List

[0165] 100 Information processing equipment

[0166] 110 Communication unit

[0167] 120 storage units

[0168] 121 Model storage unit

[0169] 122 Song data storage unit

[0170] 130 Control unit

[0171] 131 Training Units

[0172] 132 Get Unit

[0173] 133 Computing Unit

[0174] 134 Generate unit.

Claims

1. An information processing device, comprising: an acquisition unit, which uses a trained encoder to acquire a feature value of each partial data forming the first content and a feature value of optional data, wherein the optional data is data of a subject of the second content to be generated, and the first content and the second content both include digital data of songs, or the first content and the second content both include digital data of images; a calculation unit that performs addition, subtraction, multiplication, or division on a feature value of a portion of data of the first content and a feature value of each portion of data other than the portion of data to calculate a relative feature value as a relative feature value between the portion of data to calculate a relative feature value sequence indicating features of a structure of the first content; and a generating unit, generating the second content based on the relative feature value sequence of the first content and the feature value of the optional data, The generating unit inputs a new feature value sequence calculated from the relative feature value sequence of the first content and the feature value of the optional data into a trained decoder to generate the second content.

2. The information processing device according to claim 1, wherein: The calculation unit calculates relative feature values ​​between feature values ​​of the partial data of the first content and feature values ​​of each partial data other than the partial data and having a causal relationship with the partial data, so as to calculate the relative feature value sequence.

3. The information processing device according to claim 1, wherein: The generation unit generates an optional song as the second content based on the relative feature value sequence of the first content as a song represented by using data indicating a pitch, data indicating a tone length, and data indicating a timing of a rest.

4. The information processing device according to claim 3, wherein: The acquisition unit inputs a symbol string indicating a pitch, a tone length, and a rest as data indicating a sound included in the partial data and the optional data to the trained encoder to acquire feature values ​​corresponding to the partial data and the optional data, and The generating unit generates the new feature value sequence based on the relative feature value sequence calculated from the acquired feature values ​​of the partial data and the feature values ​​of the optional data.

5. The information processing device according to claim 1, wherein: The acquiring unit acquires a feature value of each partial data forming the first content as text data, and The generating unit generates the second content as text data based on the relative feature value sequence of the first content and the feature value of the optional data as text data, wherein the relative feature value sequence of the first content is obtained based on the acquired feature value of each of the partial data.

6. The information processing device according to claim 1, wherein: The acquisition unit acquires a feature value of each partial data forming the first content as a moving image content, and The generating unit generates the second content as motion image content based on the relative feature value sequence of the first content and the feature value of the optional data as motion image or image content, wherein the relative feature value sequence of the first content is obtained based on the acquired feature value of each of the partial data.

7. The information processing device according to claim 1, wherein: The acquisition unit detects interruption of a bar as the first content of a song, and acquires a feature value of each detected bar.

8. The information processing device according to claim 1, wherein: The acquisition unit detects a rest having a length greater than a predetermined threshold in the first content as a song, divides the song into partial data based on the detected rest, and acquires a feature value of each divided partial data.

9. An information processing method that enables a computer to: The trained encoder is used to obtain feature values ​​of each partial data and feature values ​​of the optional data forming the first content, wherein: The optional data is data of the subject of the second content to be generated, the first content and the second content both include digital data of songs, or the first content and the second content both include digital data of images; performing addition, subtraction, multiplication or division on a feature value of a portion of data of the first content and a feature value of each portion of data other than the portion of data to calculate a relative feature value as a relative feature value between the portion of data to calculate a relative feature value sequence indicating features of a structure of the first content; and generating the second content based on the relative feature value sequence of the first content and the feature value of the optional data, Wherein, a new feature value sequence calculated from the relative feature value sequence of the first content and the feature value of the optional data is input into a trained decoder to generate the second content.

10. A computer-readable storage medium having an information processing program stored thereon, wherein when the information processing program is executed, the computer is caused to perform the following functions: an acquisition unit, which uses a trained encoder to acquire a feature value of each partial data forming the first content and a feature value of optional data, wherein the optional data is data of a subject of the second content to be generated, and the first content and the second content both include digital data of songs, or the first content and the second content both include digital data of images; a calculation unit that performs addition, subtraction, multiplication, or division on a feature value of a portion of data of the first content and a feature value of each portion of data other than the portion of data to calculate a relative feature value as a relative feature value between the portion of data to calculate a relative feature value sequence indicating features of a structure of the first content; and a generating unit, generating the second content based on the relative feature value sequence of the first content and the feature value of the optional data, The generating unit inputs a new feature value sequence calculated from the relative feature value sequence of the first content and the feature value of the optional data into a trained decoder to generate the second content.

Images