Information processing device, information processing method, and program

Abstract

The present invention relates to: an information processing device that makes it possible to adjust the ordinariness and originality of automatically generated content by a likelihood search while satisfying reality; an information processing method; and a program. Input content comprising a data sequence is encoded and converted into a latent variable, the latent variable is decoded and output content is reconstructed, a loss function is calculated on the basis of the likelihood of the input content serving as an input sequence, the gradient of the loss function is lowered, the latent variable is updated, the updated latent variable is decoded, and output content is reconstructed. The present invention can be applied to a device for automatically generating content.

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program, and in particular to an information processing device, an information processing method, and a program that enable adjustment of the ordinariness or eccentricity of automatically generated content while maintaining realism. [Background technology]

[0002] Information processing using machine learning is being used in various technical fields. For example, a technology has been proposed that automatically generates new content by learning the characteristics of content (images, music, etc.) using a neural network that mimics the mechanism of the brain nervous system.

[0003] For example, a technology has been proposed that makes it possible to automatically compose an appropriate piece of music that matches lyrics by learning the characteristics of existing songs, without the user having to input parameters other than the lyrics (see Patent Document 1).

[0004] This technology learns linguistic features calculated from lyric data representing the lyrics of each song and attribute data representing the attributes of the song, so that when new lyric data is given, a song can be automatically generated to match the new lyric data. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2011-175006 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technology described in Patent Document 1 merely generates melodies and chords that match the lyrics, and there is a risk that the generated music will be too ordinary or too eccentric.

[0007] If the generated music is ordinary, it may be boring, but if it is eccentric, it may be to one's tastes. Furthermore, if it is too eccentric, it may not be recognizable as a song and may lack reality as a song.

[0008] Therefore, it may be possible to adjust the generated music so that it falls somewhere between ordinary and unusual, but it is difficult to achieve an adjustment that falls somewhere between ordinary and unusual.

[0009] This is not limited to automatically generated music, but also applies to automatically generating various content such as images and text.

[0010] The present disclosure has been made in light of such circumstances, and in particular, makes it possible to adjust the banality and eccentricity of automatically generated content while still maintaining realism. [Means for solving the problem]

[0011] An information processing device and program according to one aspect of the present disclosure include an information processing device and program including an encoder that encodes input content consisting of a series of data and converts it into latent variables, a decoder that decodes the latent variables to reconstruct output content, a loss function calculation unit that calculates a loss function based on the likelihood of the input content, and a control unit that descends the gradient of the loss function to update the latent variables and causes the decoder to decode the updated latent variables to reconstruct output content.

[0012] An information processing method of one aspect of the present disclosure is an information processing method for an information processing device including an encoder, a decoder, a loss function calculation unit, and a control unit, the encoder encoding input content consisting of a series of data and converting it into latent variables, the decoder decoding the latent variables to reconstruct output content, the loss function calculation unit calculating a loss function based on the likelihood of the input content, the control unit descending a gradient of the loss function to update the latent variables, and causing the decoder to decode the updated latent variables to reconstruct output content.

[0013] In one aspect of the present disclosure, input content consisting of a sequence of data is encoded and converted into latent variables, the latent variables are decoded to reconstruct output content, a loss function is calculated based on the likelihood of the input content, a gradient of the loss function is descended to update the latent variables, and the updated latent variables are decoded by the decoder to reconstruct output content. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram illustrating an overview of the present disclosure. [Diagram 2] FIG. 1 is a diagram illustrating an example of the configuration of an information processing device according to the present disclosure. [Diagram 3] 3 is a diagram for explaining functions realized by the information processing device of FIG. 2 in the first embodiment. [Figure 4] FIG. 2 is a diagram for explaining real labels and fake labels used for learning of a reality evaluator in the first embodiment. [Diagram 5] FIG. 13 is a diagram illustrating changes in latent variables based on likelihood and reality. [Figure 6] 4 is a flowchart illustrating a content generation process according to the first embodiment. [Figure 7] 10 is a diagram for explaining functions realized by the information processing device of FIG. 2 in the second embodiment. FIG. [Figure 8]FIG. 13 is a diagram for explaining real labels and fake labels used for learning of a reality evaluator in the second embodiment. [Figure 9] 13 is a flowchart illustrating a content generation process according to the second embodiment. [Figure 10] FIG. 13 is a diagram illustrating a first modified example of the present disclosure. [Figure 11] FIG. 11 is a diagram illustrating a second modified example of the present disclosure. [Figure 12] FIG. 1 is a diagram illustrating an example of the configuration of a general-purpose personal computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0016] Hereinafter, an embodiment of the present technology will be described in the following order. 1. Overview of this Disclosure 2. First embodiment 3. Second embodiment 4. Variation 1 5. Variation 2 6. Software implementation examples

[0017] <<1. Overview of this Disclosure>> In particular, the present disclosure makes it possible to adjust the banality and eccentricity of automatically generated content while still satisfying realism.

[0018] When various contents such as music, images, and text are automatically generated, the generated contents are not often generated in a satisfactory state according to the intentions of the creators of the contents, and some dissatisfaction occurs.

[0019] In this case, the creator's dissatisfaction with the generated content often occurs in the form of a satisfaction with a specific part of the content generated under certain conditions, but dissatisfaction with other parts that are different from the specific part.

[0020] In other words, creators are often dissatisfied with a particular part of the content, but are still dissatisfied with the generated content as a whole.

[0021] In such cases, the creator will repeatedly generate content automatically, changing the conditions until content that is satisfactory as a whole is generated, but it is rare that such content is automatically generated.

[0022] As a result, the generated content is not satisfactory as a whole, but some parts are satisfactory, in other words, much of the content that the creator feels is a shame is discarded.

[0023] The most basic case of content that creators feel is a shame is when the series that makes up the content is too ordinary or too unusual.

[0024] When creating content, creators strive to create content that is highly original, but if the content they create is too ordinary and generic, it doesn't feel original.

[0025] Therefore, content that is too ordinary is likely to be boring for both the creator and the people who watch or listen to the content.

[0026] On the other hand, if the generated content is too eccentric, it may feel original, but people's tastes may be divided.

[0027] Therefore, content that is too unusual may not be accepted by the target audience, even if the creator likes it.

[0028] For this reason, it is necessary to adjust the generated content so that it falls somewhere between ordinary and unusual, but it is difficult to adjust the generated content so that it falls somewhere between ordinary and unusual.

[0029] Even if we were to research the content that the general public likes, create a labeled dataset of their preferences, and then train a model that reflects that to automatically generate content, we could reflect the preferences of the general public, but there is a high possibility that the content would become more ordinary and would not be able to reflect the preferences of the target audience.

[0030] In particular, when applied to an application program that automatically generates content, preferences for the generated content will change depending on the purpose for which the content is to be generated and the target audience who will be viewing or listening to the content, so it is necessary to be able to adjust the preferences for the generated content.

[0031] Therefore, in the present disclosure, the likelihood is used to adjust the intermediate level between ordinary and unusual content to be generated.

[0032] Here, the likelihood is the probability that sample content will be obtained.

[0033] For example, if the content is a song, the likelihood is the probability that the automatically generated song is a song in the collected samples.

[0034] Therefore, in this case, a high likelihood of a generated song means that the generated song is close to the songs collected as samples and is highly likely to be a common song, that is, an ordinary song (a song that is highly ordinary).

[0035] Conversely, a low likelihood of a generated song indicates that the generated song is highly likely to be a unique song (a song with a high degree of uniqueness) that is far removed from the songs collected as samples.

[0036] In the present disclosure, as shown in FIG. 1, the likelihood of automatically generated content is adjusted to generate content with a high likelihood, thereby generating content that is highly ordinary, and conversely, content with a low likelihood is adjusted to generate content that is highly unique.

[0037] In this specification, naturalness is defined as an expression indicating a degree of automatically generated content that is intermediate between ordinary and unusual.

[0038] The term "naturalness" is used to express a degree that is intermediate (among the likely range) between ordinary and unusual. In other words, naturalness can be said to be an expression indicating a degree that is neither ordinary nor unusual.

[0039] In other words, in the present disclosure, the likelihood is adjusted to match the preferences of the target audience of the automatically generated content, so that the naturalness of the content is adjusted to a level intermediate between ordinary and unusual.

[0040] However, when adjusting the naturalness by adjusting the likelihood, the more the originality of the generated content is increased, that is, the more the adjustment is made to decrease the likelihood, the more the reality is reduced.

[0041] The reality here refers to the likelihood that the generated content is generated by a human being.

[0042] For example, when the content is a musical piece, a decrease in realism means that the generated music contains dissonant sounds that a human could never create, or rhythms and scale changes that are difficult for a human to recognize as part of a musical piece.

[0043] In other words, the lower the level of realism, the closer the automatically generated content becomes to content that could never be generated by humans, and the content may be unrecognizable to those viewing or listening to it, or in some cases, unpleasant.

[0044] Therefore, in the present disclosure, content is automatically generated by gradually changing the input content while adjusting the naturalness, which is halfway between ordinary and unique, using likelihood exploration, and ultimately changing the content into the content desired by the user.

[0045] In this case, when automatically generating content, as shown in Figure 1, even if the content is too bizarre, realism is maintained by enhancing it, and adjustments are made so that the content does not become content that could not be generated by humans (content that humans cannot recognize as content).

[0046] As a result, in the present disclosure, it becomes possible to automatically generate content while using likelihood search to adjust the naturalness that is somewhere between ordinary and unusual while still satisfying realism.

[0047] <<2. First embodiment>> Next, a configuration of an information processing device 31, which is an example of the hardware configuration of an information processing device according to the present disclosure, will be described with reference to FIG.

[0048] In this specification, an example will be described in which the information processing device 31 automatically generates music as content; however, the same applies to cases in which the information processing device 31 automatically generates various content other than music, such as images and text.

[0049] 2, the information processing device 31 includes a communication unit 51, a control unit 52, and a storage unit 53. The information processing device 31 includes an input / output unit 32 including, for example, a keyboard, a mouse, etc., for accepting various operations from an administrator who manages the information processing device 31, and a liquid crystal display, etc., for presenting various information.

[0050] The communication unit 51 is realized by, for example, a network interface card (NIC) etc. The communication unit 51 is connected to a network such as the Internet by wired or wireless connection, and transmits and receives information to and from other devices etc. via the network.

[0051] The control unit 52 is composed of a memory and a processor, and controls the overall operation of the information processing device 31 .

[0052] More specifically, the control unit 52 includes a learning unit 71, an optimization unit 72, and a generation unit 73.

[0053] The learning unit 71 trains an encoder 91 and a decoder 92 stored in a model memory unit 81 in the memory unit 53 described later, using music data stored as samples in the music DB 82, to configure a VAE (Variational Auto Encoder) as a trained model.

[0054] The optimization unit 72 is controlled by the learning unit 71, and adjusts and optimizes the parameters of the encoder 91 and the decoder 92 so that when the encoder 91 and the decoder 92 repeatedly learn using music data stored as samples in the music DB 82, the reconstruction error is minimized while the posterior distribution is regularized with the prior distribution (normal distribution).

[0055] The generation unit 73 controls the encoder 91, the decoder 92, and the loss function calculation unit 93 stored in the model storage unit 81 in the storage unit 53, adjusts the naturalness of the input content (music) between ordinary and unusual through likelihood search, and converts it into the content (music) desired by the user, thereby generating (automatically generating) the content (music). Note that the automatic generation of content by the generation unit 73 will be described in detail later with reference to FIG. 3.

[0056] The storage unit 53 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 53 has a model storage unit 81 and a music DB (database) 82.

[0057] The model storage unit 81 stores a trained model that has been trained in advance. Specifically, the model storage unit 81 includes an encoder 91 that extracts latent variables that are feature quantities from content, a decoder 92 that reconstructs content based on the latent variables, and a loss function calculation unit 93 that calculates a loss function that is the difference between the likelihood of the content as input data and the likelihood desired by the user.

[0058] The music DB 82 stores data on content (music) as samples to be input to the model. The music DB 82 also stores content that the generator 73 controls to generate (automatically generate) the encoder 91 and the decoder 92.

[0059] <Automatic generation of content by the generation unit> Next, automatic generation of content by the generation unit 73 will be described with reference to FIG.

[0060] The generation unit 73 controls the pre-trained encoder 91 and decoder 92 to cause the encoder 91 to encode the input data (music) X(init) to obtain a latent variable Zinit. The generation unit 73 then causes the decoder 92 to reconstruct the input data based on the obtained latent variable Zinit, thereby generating the output data (music) X(init)'.

[0061] More specifically, as shown in FIG. 3, the encoder 91 encodes a content (music) X(init) consisting of a sequence of partial data consisting of multiple bars, etc., to convert it into a latent variable Zinit as a feature consisting of a vector having fewer dimensions than the content (music) X.

[0062] Then, as shown in FIG. 3, the decoder 92 decodes each of the content based on the latent variable Zinit, which is the feature of the content, and returns it to the original dimension, thereby reconstructing and restoring it as content (music) X(init)' consisting of a series of partial data such as bars.

[0063] Here, the encoder 91 and the decoder 92 are controlled by the learning unit 71, and undergo unsupervised learning by VAE in advance, so that the encoder 91 encodes the input data, that is, the content (music) Xinit, and converts it into a latent variable Zinit, and the decoder 92 is configured to reconstruct the content (music) X(init)' based on the latent variable Zinit. In other words, since the encoder 91 and the decoder 92 have been trained, the content X(init) and the content X(init)' become almost identical.

[0064] When the generation unit 73 receives likelihood information indicating the degree of naturalness, which is intermediate between ordinary and unusual, desired by the creator of the content (music), it controls the loss function calculation unit 93 to calculate the difference between the likelihood of the content (music) X(init)' reconstructed by the decoder 92 and the likelihood desired by the creator as the loss function LLE.

[0065] Then, the generation unit 73 changes the latent variable Zi (i is the number of times the loss function LLE is lowered) so as to gradually lower the calculated loss function LLE by a predetermined value Δ, i.e., to reduce the loss function LLE in stages, and has the decoder 92 decode the changed latent variable Zi, thereby gradually generating content (music) X(i)' with the likelihood desired by the creator.

[0066] The loss function LLE is a function that indicates the difference in likelihood between the content X(init)' reconstructed using the latent variable Zinit obtained from the input data content X(init) and the content with the desired likelihood, as shown in the following equation (1), for example, and is composed of a function F1 constituting a term related to the difference in the likelihood of the contents, and a function F2 constituting a term related to the likelihood of the reality of the content.

[0067] LLE = F1-α×F2 (1)

[0068] Here, LLE is a loss function, F1 is a function constituting a term related to the likelihood of content, F2 is a function constituting a term related to the likelihood of reality, and α is a predetermined coefficient that can be set arbitrarily.

[0069] More specifically, the loss function calculation unit 93 includes a likelihood evaluator 101 and a reality evaluator 102, and calculates a function F1 constituting a term related to the likelihood of the content based on the likelihood calculated by the likelihood evaluator 101, and calculates a function F2 constituting a term related to the likelihood of the reality based on the likelihood of the reality of the content calculated by the reality evaluator 102.

[0070] The likelihood evaluator 101 learns, for example, using an RNN or Transformer as an architecture based on a sequence generation model (language generation model) so as to maximize the log-likelihood, and calculates the likelihood of the reconstructed content (music) X' as a log-likelihood.

[0071] Here, the likelihood of the reconstructed content X′ is the probability that the reconstructed content X′ is a piece of music registered in the music DB 82 as a sample.

[0072] More specifically, when a reconstructed content (music) X' is constructed as sequence data X' (X1', X2', . . . Xn') such as partial data X1', X2', . . . Xn', the likelihood of the content X' is expressed as the product of the probabilities of each of the partial data.

[0073] For example, when the likelihood (probability) of a song X' is expressed as P(X'), the likelihood (probability) P(X') of a song X' is calculated as the following formula (2).

[0074] In the case of sequence data X' (X1', X2', ... Xn'), when an initial value Start is input, the first partial data X1' is generated, when partial data X1' is input, the adjacent partial data X2' is generated, when partial data X2' is input, the adjacent partial data X3' is generated, ... when partial data X(n-1)' is input, the adjacent partial data Xn' is generated.

[0075] Therefore, the likelihood (probability) P(X') of the content X' consisting of this sequence data (X1', X2', . . . Xn') is expressed as the following formula (2).

[0076] P(X')=P(X1'|Start) ×P(X2'|Start,X1') ×P(X3'|Start,X1',X2') ×P(X4'|Start,X1',X2',X3') ×P(X5'|Start,X1',X2',X3',X4') ... ×P(Xn'|Start,X1',X2',...X(n-1)') (2)

[0077] Here, P(X') is the likelihood (probability) of the content (music) X'.

[0078] Furthermore, P(Xn'|Start, X1', X2', ... X(n-1)') is the conditional probability (likelihood) of partial data Xn' when the initial value is Start and the partial data are sequentially X1', X2', ... X(n-1)'.

[0079] The likelihood evaluator 101 converts the likelihood P(X') of the content (music) X' thus determined into a logarithm, and outputs it as a log likelihood EL(X').

[0080] In addition, the reality evaluator 102 pre-learns using, for example, an RNN, Transformer, or other architecture so as to maximize the log-likelihood indicating reality based on the input content consisting of a series of content labeled with a real class, which is content generated by humans, and content consisting of a series of content labeled with a fake class, which is not generated by humans.

[0081] Then, the reality evaluator 102 obtains the likelihood of the reality of the content (music piece) X(init)' as a function of the term related to the likelihood of the reality, and obtains the log-likelihood of the reality as reality ER(X') by logarithmizing it.

[0082] In the following, the likelihood indicating the probability that the reconstructed content (song) X' is song data registered as a sample in the song DB82 will be referred to as "likelihood," and the likelihood of reality will be referred to simply as "reality" for the sake of distinction, although it is still a likelihood conceptually. However, both are equally likely.

[0083] Here, the content in the series labeled with the real class is, for example, sample music data registered in the music DB 82 and used for learning.

[0084] Furthermore, content consisting of a sequence labeled with the fake class is, for example, music data F that is reconstructed by decoding a latent variable Z obtained from a prior distribution by a VAE related to the training of an encoder 91 and a decoder 92, as shown in Figure 4.

[0085] That is, the reality evaluator 102 learns based on a group of contents R consisting of sequences labeled with the real class generated as shown in FIG. 4 and a group of contents F consisting of sequences labeled with the fake class, and calculates the log-likelihood, which is the logarithm of the likelihood that the reconstructed content (music piece) X(init)' is content generated by a human, as reality ER(X(init)').

[0086] The loss function calculation unit 93 calculates the loss function LLEinit shown in the above-mentioned equation (1) from the likelihood (log likelihood) EL(X(init)') of the reconstructed content X' calculated by the likelihood evaluator 101 and the reality (log likelihood) ER(X(init)') calculated by the reality evaluator 102 as the probability that the reconstructed content X' is content generated by a human.

[0087] More specifically, the loss function calculation unit 93 calculates the term relating to the likelihood of the reconstructed content X′, which is the function F1 in equation (1), as shown in the following equation (3).

[0088] F1 = (EL(X(init)')-β×ELinit) 2 (3)

[0089] Here, F1 is a function indicating the term relating to the likelihood of the reconstructed content X(init)' in equation (1), EL(X(init)') is the log-likelihood of the reconstructed content X(init)' calculated by the likelihood evaluator 101, and β×ELinit is the reference likelihood.

[0090] The reference likelihood β×ELinit is a value set by the creator who is trying to automatically generate content, that is, a value that serves as a guideline for the likelihood that is ultimately to be obtained, and is expressed as the product of the coefficient β and the initial value ELinit of the likelihood (a predetermined fixed value).

[0091] For example, when it is desired to increase the likelihood of the input data content X(init) and reconstruct (automatically generate) content with a higher level of mediocrity, the reference likelihood is set to be greater than the initial value ELinit of the likelihood, and therefore the coefficient β is set to a value greater than 1. Also, when there is no specific desired likelihood and it is desired simply to increase the likelihood, the coefficient β may be set to a specific value greater than 1, for example, 1.2 or 1.5.

[0092] On the other hand, when it is desired to lower the likelihood of automatically generated content and increase its originality, the reference likelihood is set to be smaller than the initial value ELinit of the log likelihood EL, and therefore the coefficient β is set to a value smaller than 1. Also, when there is no specific desired likelihood and it is desired simply to lower the likelihood, the coefficient β may be set to a specific value smaller than 1, for example, 0.8 or 0.5.

[0093] Moreover, the loss function calculation unit 93 performs calculation by substituting the log likelihood ER(X(init)') of the reality of the reconstructed content X(init)' into the function F2 of the above-mentioned formula (1).

[0094] From these factors, the loss function calculation unit 93 calculates the loss function LLE as shown in the following equation (4).

[0095] LLE = F1-α×F2 = (EL(X(init)')-β×ELinit) 2 -α × ER(X(init)') (4)

[0096] The loss function calculation unit 93 makes the decoder 92 differentiable by using, for example, Gumbel softmax so that it can receive evaluation signals from the likelihood evaluator 101 and the reality evaluator 102 .

[0097] <Changes in latent variables based on loss functions> As described above, the loss function LLE is composed of a function F1 that depends on the difference between the likelihood of the input data content X(init) (= the likelihood of the reconstructed content X(init)') and the likelihood desired by the creator, and a function F2 that depends on reality.

[0098] Therefore, the generation unit 73 reconstructs the content X(i) so as to reduce the loss function LLE, thereby generating content with the likelihood desired by the creator.

[0099] In other words, since the loss function LLE is configured as shown in equation (4), the generation unit 73 reconstructs the content so that the loss function LEE is small by increasing the reality ER while bringing the likelihood EL closer to the reference likelihood.

[0100] More specifically, the generation unit 73 changes the latent variable Zi based on the above-mentioned loss function LLE so as to gradually decrease the loss function by a predetermined value Δ, and causes the decoder 92 to decode the changed latent variable Zi, thereby sequentially generating new content X'(i).

[0101] Next, a method for changing the latent variable Zi so as to reduce the loss function will be described.

[0102] For example, as shown in Figure 5, consider defining a latent variable Z calculated based on various input contents as a latent variable space represented in a two-dimensional space, and expressing the likelihood of a content when reconstructed using each latent variable Z in the latent variable space as a contour line.

[0103] In addition, in the latent variable space of FIG. 5, an example will be described in which the latent variable Z is two-dimensional for the sake of explanation. However, in reality, the latent variable Z is composed of many more dimensions, and the latent variable space is also expressed in many more dimensions.

[0104] That is, in the latent variable space of Fig. 5, the distribution of latent variables Z expressed in two dimensions is represented by crosses, and the likelihood when the content is reconstructed using each latent variable Z is shown by solid lines L1 to L5 in a concentric circle shape, with the likelihood decreasing toward the outside of the page. That is, the likelihoods shown by the solid lines L1 to L5 are in the order of L1>L2>L3>L4>L5 in the figure. The distribution of the likelihoods in the latent variable space of Fig. 5 is just an example.

[0105] In addition, in the latent variable space of FIG. 5, when the reality when the content is reconstructed is obtained using a similar latent variable Z, the distribution of a predetermined threshold value of the reality is indicated by a dotted line R1.

[0106] Here, the area above the dotted line R1 in the figure is a non-real area where the reality is lower than a predetermined threshold and the reconstructed content is considered to be "non-real" and difficult to recognize as having been generated by a human.

[0107] Furthermore, in the latent variable space of Fig. 7, the area below the dotted line R1 is considered to be the real area, where the reality is higher than a certain threshold and the reconstructed content is recognized as having been generated by a human.

[0108] Incidentally, the reality in the latent variable space in Figure 5 should also be displayed as multiple contour lines, just like the likelihood, but here, only the dotted line R1, which is the distribution of a predetermined threshold that is the boundary between the real area and the non-real area, is displayed.

[0109] Here, for example, consider a case where the latent variable generated by encoding the input data, content Xa, by encoder 91 is represented by position Za in the latent variable space of Figure 5, and the creator wants to lower the likelihood from the current level of solid line L2 to the level of solid line L4.

[0110] If there is no need to be conscious of reality, the generation unit 73 can obtain latent variables by moving the position in the latent variable space in the direction of vector VL perpendicular to the solid line L2, which is a contour line representing the level of likelihood in the latent variable space of Figure 5, toward the solid line L4, and then have the latent variables decoded by the decoder 92, thereby reconstructing content with the likelihood desired by the creator.

[0111] That is, in the latent variable space, latent variables that are close to each other can be said to be similar latent variables, and therefore, when viewed from position Za where the likelihood level is on solid line L2, the latent variable obtained at position Zx on the nearest solid line L4 can be considered to be the latent variable with the likelihood represented by solid line L4 that is most similar to the latent variable at the current position Za.

[0112] However, when taking reality into consideration, in FIG. 7, the region in which the latent variable Za exists is a non-real region, so if the latent variable is obtained by moving the position in the latent variable space while considering only the likelihood, and the content is reconstructed by decoding it using the decoder 92, even if the likelihood is satisfied, the reality will be low, and there is a risk that the subject will not be able to recognize the content as generated by a human when they see or hear it.

[0113] Therefore, the generation unit 73 sets the position Za in the latent variable space to the reality vector VR in the direction closest to the dotted line R1, moves it to a position Zb that is a combination of the vectors VL and VR and that lowers the likelihood gradient by a predetermined value Δ, obtains a latent variable, and has the decoder 92 decode it to reconstruct new content.

[0114] The generation unit 73 repeats the same operations thereafter, for example, to sequentially output the latent variables obtained by changing the position in the latent variable space from Za to Zb, Zc, Zd, Ze, and Zf, to the decoder 92, which decodes the latent variables, thereby generating new content.

[0115] In other words, the content Xb generated by decoding the latent variable at position Zb in the latent variable space in Figure 7 by the decoder 92 has a lower likelihood than the content Xa, and its reality is improved, and it approaches the dotted line R1, which is the boundary with the real area.

[0116] Furthermore, the content Xc generated by decoding the latent variable at position Zc by the decoder 92 has a lower likelihood than the content Xb, and has a higher reality, and thus approaches the dotted line R1.

[0117] Furthermore, the content Xd generated by decoding the latent variable at position Zd by the decoder 92 has a lower likelihood than the content Xc and a higher level of realism, so that the content Xd crosses the dotted line R1 and enters the real area, thereby achieving a state of sufficient realism.

[0118] In addition, the content Xe generated by decoding the latent variable at position Ze by the decoder 92 has a lower likelihood than the content Xd, and since the content Xd is already sufficiently realistic, it is moved in a direction closer to perpendicular to the solid line L3, which is a contour line indicating the likelihood.

[0119] Furthermore, the content Xf generated by decoding the latent variable at position Zf by the decoder 92 has a lower likelihood than the content Xe, and since the content Xd is already sufficient as reality, it is moved in a direction approximately perpendicular to the solid line L3, which is a contour line indicating the likelihood.

[0120] As described above, in the latent variable space shown in Figure 5, the latent variables are changed and the decoding process is repeated so that the likelihood in the loss function LLE is gradually reduced while improving the realism. This makes it possible for the content, which is the input data, to gradually approach the likelihood desired by the creator while improving its realism.

[0121] In the above, we have explained an example of making content more natural by gradually reducing the likelihood of the content, which is the input data, to the desired likelihood. However, when making content ordinary, the process is similar, except that the reference likelihood is set high.

[0122] Furthermore, in the above, we have described a process for gradually approaching the likelihood desired by the creator, but rather than a gradual process, it is also possible to change the latent variables so as to reach the desired likelihood in one go.

[0123] <Content Generation Process in the First Embodiment> Next, the content generation process in the first embodiment will be described with reference to the flowchart in FIG.

[0124] In step S11, the generation unit 73 initializes a counter i to one.

[0125] In step S12, the generating unit 73 sets a reference likelihood. More specifically, the generating unit 73 sets the reference likelihood by, for example, accepting the value of the coefficient β in the above-mentioned formula (4) and setting a specific value of the reference likelihood, or accepting input of information such as wanting to increase or decrease the likelihood and setting the coefficient β to a predetermined value.

[0126] In step S13, the generation unit 73 receives an input of content X(init) which is input data.

[0127] In step S14, the generation unit 73 controls the encoder 91 to encode the partial data Xint and convert it into a latent variable Zinit.

[0128] In step S15, the generation unit 73 controls the decoder 92 to decode the latent variable Zinit to reconstruct the content X(init)'.

[0129] In step S16, the generation unit 73 controls the loss function calculation unit 93 to calculate the loss function LLEinit based on the difference between the likelihood of the content X(init)' and the likelihood desired by the creator, using the above-mentioned equation (4).

[0130] In step S17, the generation unit 73 obtains a loss function LLEi by lowering the loss function LLEinit by a predetermined value Δ.

[0131] In step S18, as described with reference to FIG. 5, the generation unit 73 moves the position of the latent variable Zinit in the latent variable space while maintaining reality and decreasing the likelihood so as to change the loss function LLEinit to a loss function LLEi lowered by a predetermined value Δ, and obtains and updates the latent variable Zi corresponding to the new position in the latent variable space.

[0132] In step S19, the generation unit 73 controls the decoder 92 to decode the latent variable Zi and reconstruct the content X(i)'.

[0133] In step S20, the generation unit 73 stores the reconstructed content X(i)′ in the music DB 82 of the storage unit 53.

[0134] In step S21, the generation unit 73 increments the counter i by one.

[0135] In step S22, the generation unit 73 determines whether or not the counter i is at the maximum value imax, and if the counter i is not at the maximum value imax, the process proceeds to step S23.

[0136] In step S23, the generation unit 73 updates the loss function LLEi by decreasing the loss function LLEi by a predetermined value Δ (LLEi=LLEi−Δ), and the process returns to step S18.

[0137] At this time, in step S18, the generation unit 73 moves and updates the position of the latent variable Zi in the latent variable space so as to correspond to the change in the loss function LLEi that has been updated by descending by a predetermined value Δ, as described with reference to FIG. 5, and obtains and updates a new latent variable Zi.

[0138] That is, until the counter i reaches the maximum value imax, the processes of steps S18 to S23 are repeated, the loss function LLEi is successively decreased by a predetermined value Δ, the latent variables Zi are updated, and the updated latent variables Zi are successively decoded to generate new content Xi, so that the newly reconstructed content Xi changes successively to gradually approach the likelihood desired by the creator while satisfying reality.

[0139] Then, if it is determined in step S22 that the counter i has reached the maximum value imax, the process proceeds to step S24.

[0140] In step S24, the generation unit 73 outputs the content X(i) (i=1, 2, 3, . . . imax) that is stored in the storage unit 53 and serves as output data.

[0141] Through the above processing, it is possible to generate content X(i)' by gradually changing the input data, content X(init), so that it meets the likelihood intended by the creator while still satisfying reality.

[0142] As a result, it becomes possible to automatically generate content while adjusting the naturalness of the content, which is somewhere between ordinary and unusual, as intended by the creator.

[0143] In the process of FIG. 6, in the process of calculating the loss function LLEinit, which is the difference between the likelihood in the input data, content X(init), and the likelihood desired by the creator, the latent variable Zinit is first decoded by the decoder 92 to obtain the content X(init)', and then the calculation is performed. However, in the initial process, since no change is made to the latent variable Zinit, the content Xinit and the content Xinit' are substantially identical.

[0144] For this reason, the initial loss function LLEinit may be obtained directly from the content Xinit, which is the input data.

[0145] In the above, an example has been described in which the loss function LLEinit is gradually decreased by a predetermined value Δ while the latent variable Zi is sequentially changed, and the process of repeatedly reconstructing the content X(i)' is repeated until the counter i reaches imax. However, the process may be terminated when the loss function LLEi cannot be reduced any further.

[0146] Furthermore, in the above, an example has been described in which the loss function LLEinit is gradually lowered to finally output all the contents X(i)' (i = 1, 2, ... imax) that are repeatedly reconstructed. However, it is also possible to output only the final content X(imax)'.

[0147] In addition, the function F1 consisting of terms related to the likelihood that constitute the loss function LLE can be set large in cases where it is sufficient to simply make the likelihood larger and ordinary, and can be set small in cases where it is sufficient to simply make the likelihood smaller and unusual.

[0148] Therefore, when there is no final estimate of likelihood, the following formula (5) may be used.

[0149] F1=EL(X(init))' (When you want to reduce the likelihood and make it odd) =-EL(X(init))' (when you want to increase the likelihood and trivialize it) (5)

[0150] That is, the function F1 may be used differently depending on whether it is desired to make the content unusual or ordinary, as shown in formula (5). Furthermore, formula (5) may be used by multiplying the likelihood EL(X(init)) of the content X(init) by a positive coefficient when it is desired to make the content unusual by reducing the likelihood, and by multiplying the likelihood by a negative coefficient when it is desired to make the content ordinary by increasing the likelihood.

[0151] <<3. Second embodiment>> In the above, an example has been described in which the input data content X(init) is encoded and converted into a latent variable Zinit, the latent variable Zinit is then decoded to obtain the content X(init)', a loss function LLEinit which is the difference between the likelihood of the content X(init)' and a desired likelihood is obtained, the latent variable Zi is gradually updated while updating the loss function LLEi by lowering it by a predetermined value Δ, and the updated latent variable Zi is decoded to repeatedly reconstruct the content X(i)'.

[0152] However, in the case of the above-described process, every time the loss function LLEi is updated, there is a risk that the reconstructed content X(i)' may become completely different from the content X(init) that is the input data.

[0153] For this reason, even if there is a part of content X (init) that the creator likes before changing the naturalness (which is an intermediate level between ordinary and unusual), changing the ordinary or unusualness may change the whole content, including the liked part, making it difficult for the creator to like the whole content.

[0154] Therefore, the parts of the input data content Xinit that the creator likes can be set not to change as context, and only the other parts can be adjusted by likelihood search to a naturalness that is somewhere between ordinary and unusual.

[0155] In the following description, the likelihood exploration performed in a state where an unchanged portion is set as the context is also referred to as a contextual likelihood exploration.

[0156] For example, for content X(init) serving as input data as shown in FIG. 7, the generation unit 73 receives input regarding information on parts that the creator does not want to change, and sets the parts that the creator does not want to change as context.

[0157] In the example of Figure 7, the parts of content X (init) that the creator likes and does not want to change are set as contexts C1 and C2, and the parts to which changes should be made other than contexts C1 and C2 are set as partial data Y (init).

[0158] In addition, in Figure 7, an example is shown in which contexts C1 and C2 are set before and after the partial data Y(init) to which a change is to be made, but the positions at which the contexts are set may be other than this, or may be two or more positions.

[0159] Then, the generating unit 73 controls the encoder 91 to encode only the partial data Y(init), which is the part to be changed, and converts it into a latent variable Zinit.

[0160] The generator 73 controls the decoder 92 to reconstruct the partial data Y(init)' based on the latent variable Zinit.

[0161] The generating unit 73 integrates the reconstructed partial data Y(init) with the contexts C1 and C2 to reconstruct the content X(init)'.

[0162] Thereafter, just as in the case where there is no context, the generation unit 73 calculates a loss function LCLEinit, which is the difference between the likelihood of the content X(init)' and the likelihood desired by the creator, decreases it by a predetermined value Δ, updates the loss function LCLEi and updates the corresponding latent variable Zi, controls the decoder 92 to decode, reconstructs the partial data Y(i)', and further repeats the process of integrating it with contexts C1 and C2 to reconstruct the context X(i)'.

[0163] <Reality evaluator when context is used> When a context is used, as shown in FIG. 7, the reality evaluator 102 learns only partial data V of human-generated content R other than the contexts C11 and C12, using content F reconstructed by adding the contexts C11 and C12 to partial data V' reconstructed by decoding a latent variable Z' obtained by adding noise to a latent variable Z obtained from a prior distribution by a VAE related to the learning of the encoder 91 and the decoder 92, by the decoder 92.

[0164] That is, in the case of using contexts, the reality evaluator 102 learns by regarding content R consisting of contexts C11, C12 and partial data V shown in FIG. 7 as content consisting of a sequence labeled with the real class, and content F reconstructed by adding contexts C11, C12 to the reconstructed partial data V' as content consisting of a sequence labeled with the fake class, and obtains the log-likelihood, which is the logarithm of the likelihood, which is the probability that the reconstructed content (music) X' is content generated by a human, as reality ER(X').

[0165] <Content Generation Process in the Second Embodiment> Next, a content generation process in the second embodiment will be described with reference to the flowchart in FIG.

[0166] In step S51, the generation unit 73 initializes a counter i to one.

[0167] In step S52, the generation unit 73 receives a setting of the reference likelihood.

[0168] In step S53, the generation unit 73 receives an input of content X(init) which is input data.

[0169] In step S54, the generating unit 73 receives information on a part that becomes a context, which is a part of the content X(init) that the creator does not want to change.

[0170] In step S55, the generation unit 73 generates partial data Y(init) by removing the part that becomes the context from the content X(init).

[0171] In step S56, the generation unit 73 controls the encoder 91 to encode the partial data Y(init) and convert it into a latent variable Zinit.

[0172] In step S57, the generation unit 73 controls the decoder 92 to decode the latent variable Zinit and reconstruct the partial data Y(init)'.

[0173] In step S58, the generation unit 73 integrates the partial data Y(init)' and the context to reconstruct the context X(init)'.

[0174] In step S59, the generation unit 73 controls the loss function calculation unit 93 to calculate the loss function LCLEinit of the content X(init)' by using the above-mentioned formula (4). Note that, here, the loss function LCLEinit when a context is set and the loss function LLEinit when a context is not set are given different symbols, but the formulas are both the same, being formula (4).

[0175] In step S60, the generation unit 73 obtains a loss function LCLEi by lowering the loss function LCLEinit by a predetermined value Δ. The loss function LCLEi is also basically the same as the loss function LLEi.

[0176] In step S61, when changing the loss function LCLEinit to a loss function LCLEi lowered by a predetermined value Δ, the generation unit 73 moves the position of the latent variable Zinit in the corresponding latent variable space to obtain and update the latent variable Zi.

[0177] In step S62, the generation unit 73 controls the decoder 92 to decode the latent variable Zi and reconstruct the partial data Y(i)'.

[0178] In step S63, the generation unit 73 integrates the partial data Y(i)' and the context to reconstruct the content X(i)'.

[0179] In step S64, the generation unit 73 stores the reconstructed content X(i)′ in the music DB 82 of the storage unit 53.

[0180] In step S65, the generation unit 73 increments the counter i by one.

[0181] In step S66, the generation unit 73 determines whether or not the counter i is at the maximum value imax, and if the counter i is not at the maximum value imax, the process proceeds to step S67.

[0182] In step S67, the generation unit 73 updates the loss function LLEi by decreasing it by a predetermined value Δ (LLEi=LLEi−Δ), and the process returns to step S61.

[0183] That is, until the counter i reaches the maximum value imax, the processes of steps S61 to S67 are repeated, and the loss function LLEi is successively decreased by a predetermined value Δ while the latent variable Zi is correspondingly changed and decoded, new partial data Y(i)' is generated, and further integrated with the context to repeatedly reconstruct the content X(i)'. As a result, the newly reconstructed content Xi changes successively so as to gradually approach the likelihood desired by the creator while satisfying reality.

[0184] At this time, no changes are made to the parts that have been set as the context, so the newly reconstructed content Xi changes sequentially so as to gradually approach the likelihood desired by the creator while retaining the parts that the creator likes and satisfying realism.

[0185] Then, if it is determined in step S66 that the counter i has reached the maximum value imax, the process proceeds to step S68.

[0186] In step S68, the generation unit 73 outputs the content X(i)′ (i=1, 2, 3, . . . imax) stored in the music DB 82 of the storage unit 53 as output data.

[0187] Through the above processing, it is possible to gradually generate content X(i)' by using contextual likelihood exploration, which changes the input data, content X(i), to the likelihood intended by the creator while maintaining realism, while retaining the parts set as context.

[0188] As a result, it is possible to automatically generate content, as intended by the creator, while retaining the parts of the content that the creator likes, and adjusting the naturalness of only the other parts to fall somewhere between ordinary and unusual.

[0189] <<4. Variation 1>> In the above, we have explained an example in which one likelihood and one reality are found for the entire content and the loss function is calculated. However, since content is a sequence of multiple elements, it is possible to decompose and construct the likelihood for each element, and a sequence made up of the likelihood for each element can be constructed.

[0190] A sequence consisting of the likelihood of each element constituting this content is called an information flow.

[0191] That is, if the content Xinit is, for example, a piece of music, it is composed of elements X1, X2, . . . Xn which are a sequence in the time direction as shown in the left part of FIG.

[0192] Therefore, it becomes possible to find the likelihood EL(Xi) for each element X1, X2, . . . Xn, which becomes the information flow.

[0193] When an information flow is used, a reference likelihood is set for each element X1, X2, ... Xn, and the sum of squares of the difference between the likelihood EL(Xi) of each element and the reference likelihood is used as the function F1 in equation (1) constituting the loss function described above, as shown in, for example, the following equation (6).

[0194] F1 = Σ(EL(X(i)')-βi × ELinit) 2 (i=1,2,...n) (6)

[0195] Here, i (i = 1, 2,..., n) is the identifier of each element, EL(X(i)') is the likelihood of element Xi, βi is the coefficient of each element, and ELinit is the initial value of the likelihood.

[0196] Therefore, in equation (6), the function F1 is expressed as the sum of squares of the difference between the likelihood EL(Xi) of each element and the reference likelihood βi×ELinit.

[0197] As shown in the right part of FIG. 10, the likelihood series for each element can be said to be, in other words, a change in likelihood in the time direction, that is, a change in surprise level in the time direction.

[0198] In FIG. 10, for example, the likelihood peaks or valleys at the timings indicated by the dotted circles, indicating a surprising change.

[0199] It is generally known that surprise levels are such that a small surprise occurs after a large surprise, and a large surprise occurs after a small surprise.

[0200] Therefore, by using a likelihood sequence such as that shown in FIG. 10, it is possible to reflect surprising changes in a piece of music.

[0201] In addition, here, the elements X1, X2, ..., Xn constituting the content Xinit are assumed to be elements with a minimum configuration in the time direction, but a cluster consisting of a plurality of elements may be formed and a likelihood sequence may be set for each cluster. Furthermore, for the function F1 in equation (1) constituting the loss function, instead of the square error as explained with reference to equation (6), a correlation with a reference likelihood may be used, or a statistic such as a variance for the elements of the likelihood sequence (information flow) may be used without using the reference likelihood.

[0202] The content generation process is the same as the process when the context is set as described with reference to the flowchart in FIG. 9, except for the calculation method of the loss function LLE, and therefore will not be described again.

[0203] <<5. Variation 2>> In the above, when a context is set, the likelihood calculated by the likelihood evaluator 101 is the likelihood for the entire reconstructed content X'. However, as shown in Figure 11, instead of the likelihood evaluator 101, a likelihood evaluator 101' may be provided, and the likelihood evaluator 101' may calculate a conditional likelihood CEL from only partial data Y' of the content X' other than contexts C1 and C2, which is to be changed to the likelihood desired by the creator.

[0204] Furthermore, when a context is set in a similar manner, as shown in FIG. 11, instead of the reality evaluator 102, a reality evaluator 102′ that is generated by the same sequence generation model as the likelihood evaluator 101 and is substantially the same as the likelihood evaluator 101 may be provided, and the likelihood EL itself may be used as the reality ER.

[0205] As a result, the loss function is expressed as the following equation (7).

[0206] LLE = F1-α×F2 = (CEL(X(init)')-β×ELinit) 2 -α×EL(X(init)') (7)

[0207] Here, CEL(X(init)') is the conditional likelihood of the content X(init)', β×ELinit is the reference likelihood, and EL(X(init)') is the likelihood of the content X(init)'.

[0208] The conditional likelihood represents "surprise" because it is the likelihood of partial data of the content X(init)' that changes depending on the likelihood. Therefore, by setting a reference likelihood for the conditional likelihood, it is possible to adjust the generated content to make the "surprise" greater or less.

[0209] Furthermore, since the reality evaluator 102′ essentially functions as the likelihood evaluator 101, the same sequence generation model is used for both the likelihood evaluator 101′ and the reality evaluator 102′, so that it is possible to simplify the configuration by consolidating the configuration into only one of them.

[0210] Furthermore, in the second variant example, an example has been described in which the conditional likelihood CEL(X(init)') is used for the function F1, and the likelihood EL(X(init)') is used as the reality ER(X(init)') for the function F2. However, it is also possible to use the conditional likelihood CEL(X(init)') for the function F1, and the reality ER(X(init)') for the function F2.

[0211] Similarly, the likelihood EL(X(init)') may be used in place of the reality ER(X(init)') of the function F2 in the above-mentioned equations (3), (5) and (6).

[0212] The content generation process is the same as the process when the context is set as described with reference to the flowchart in FIG. 9, except for the calculation method of the loss function LLE, and therefore will not be described again.

[0213] <<6. Example of execution by software>> 12 shows an example of the configuration of a general-purpose computer. This personal computer has a built-in CPU (Central Processing Unit) 1001. An input / output interface 1005 is connected to the CPU 1001 via a bus 1004. A ROM (Read Only Memory) 1002 and a RAM (Random Access Memory) 1003 are connected to the bus 1004.

[0214] An input unit 1006 consisting of input devices such as a keyboard and a mouse through which a user inputs operation commands, an output unit 1007 for outputting a processing operation screen and an image of the processing result to a display device, a storage unit 1008 consisting of a hard disk drive for storing programs and various data, and a communication unit 1009 consisting of a LAN (Local Area Network) adapter and the like for executing communication processing via a network such as the Internet are connected to the input / output interface 1005. Also connected is a drive 1010 for reading and writing data from and to a removable storage medium 1011 such as a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor memory.

[0215] The CPU 1001 executes various processes according to a program stored in a ROM 1002 or a program read from a removable storage medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory and installed in a storage unit 1008, and loaded from the storage unit 1008 to a RAM 1003. The RAM 1003 also stores data and the like necessary for the CPU 1001 to execute various processes, as appropriate.

[0216] In a computer configured as described above, the CPU 1001 loads a program stored in the memory unit 1008, for example, into the RAM 1003 via the input / output interface 1005 and the bus 1004, and executes the program, thereby performing the series of processes described above.

[0217] The program executed by the computer (CPU 1001) can be provided by being recorded on a removable storage medium 1011 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0218] In the computer, the program can be installed in the storage unit 1008 via the input / output interface 1005 by mounting the removable storage medium 1011 in the drive 1010. The program can also be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Alternatively, the program can be pre-installed in the ROM 1002 or the storage unit 1008.

[0219] In addition, the program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or it may be a program in which processing is performed in parallel or at the required timing, such as when called.

[0220] 12 realizes the functions of the control unit 52 in FIG. 2, and the storage unit 1008 realizes the functions of the storage unit 53 in FIG.

[0221] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.

[0222] The embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present disclosure.

[0223] For example, the present disclosure can take the form of cloud computing in which a single function is shared and processed collaboratively by multiple devices via a network.

[0224] Furthermore, each step described in the above flow chart can be executed by one device, or can be shared and executed by a plurality of devices.

[0225] Furthermore, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device, or can be shared and executed by multiple devices.

[0226] The present disclosure can also be configured as follows.

[0227] <1> an encoder that encodes input content consisting of a sequence of data into latent variables; a decoder for decoding the latent variables to reconstruct output content; a loss function calculation unit that calculates a loss function based on the likelihood of the input content; a control unit that updates the latent variables by descending a gradient of the loss function, and causes the decoder to decode the updated latent variables and reconstruct output content; An information processing device comprising: <2> The encoder and decoder are trained using a Variational Auto Encoder (VAE) to minimize the reconstruction error while regularizing the posterior distribution by the prior distribution. <1> The information processing device described in <3> The loss function calculation unit is a likelihood evaluator for calculating the likelihood of the input content; a reality evaluator that calculates a reality likelihood that is a likelihood of reality of the input content; Calculating the loss function based on the likelihood as an evaluation result of the likelihood evaluator and the reality likelihood as an evaluation result of the reality evaluator. <1> The information processing device described in <4> When the reality likelihood is smaller than a predetermined value, the control unit, when updating the latent variable by descending the gradient of the loss function, updates the latent variable by descending the gradient of the loss function so that the reality likelihood becomes larger, and causes the decoder to decode the updated latent variable to reconstruct the output content. <3> The information processing device described in <5> The control unit updates the latent variables by moving the position in the latent variable space so as to increase the reality and decreases a gradient of the loss function when the reality likelihood is smaller than a predetermined value based on a distribution of the likelihood for each latent variable in a latent variable space that is a space of the latent variables and a distribution of the reality likelihood, and causes the decoder to decode the updated latent variables to reconstruct the output content. <4> The information processing device described in <6> The likelihood evaluator obtains a logarithm of the likelihood representing the probability that the input content is a content registered in advance as a sample, as the likelihood. <3> The information processing device described in <7> The likelihood evaluator obtains a logarithm of a likelihood obtained as a product of conditional probabilities for each data sequence, which is a probability that the data sequence constituting the input content is a content registered in advance as a sample. <6> The information processing device described in <8> The reality evaluator obtains the reality likelihood by taking a logarithm of the likelihood of the reality, which is the probability that the input content is a human-generated content. <3> The information processing device described in <9> The loss function is composed of a first term based on the likelihood of the input content and a second term based on the reality likelihood. <3> The information processing device described in <10> The first term is a square of the difference between the likelihood of the input content and a predetermined likelihood multiplied by a predetermined constant and a predetermined coefficient. <9> The information processing device described in <11> The predetermined coefficient is set to a value greater than 1 when increasing the likelihood of the output content and making it ordinary, and is set to a value less than 1 when decreasing the likelihood of the output content and making it unusual. <10> The information processing device described in <12> The predetermined constant is an initial value of the likelihood <10> The information processing device described in <13> In the first term, when the likelihood is increased, a positive coefficient is assigned to the likelihood of the input content, and when the likelihood is decreased, a negative coefficient is assigned to the likelihood of the input content. <9> The information processing device described in <14> The first term is a sum of squares of differences between the likelihood of each element constituting the input content and a predetermined likelihood composed of a product of a predetermined coefficient and a constant for each element. <9> The information processing device described in <15> When a portion of the input content that is not to be changed is designated as a context, the encoder encodes a portion of the input content other than the context as input partial data and converts it into a latent variable; The decoder decodes the latent variables to reconstruct output partial data; The control unit updates the latent variables by descending a gradient of the loss function, decodes the updated latent variables by the decoder to reconstruct output partial data, and reconstructs output content by integrating the reconstructed output partial data and the context. <3> The information processing device described in <16> The loss function calculation unit calculates the loss function based on a conditional likelihood that is a likelihood of input partial data other than the context in the input content. <15> The information processing device described in <17> The loss function calculation unit uses the likelihood of the input content as the reality likelihood of the input content. <15> The information processing device described in <18> The control unit gradually decreases a gradient of the loss function by a predetermined magnitude to gradually update the latent variables, and causes the decoder to repeatedly decode the updated latent variables to gradually reconstruct a plurality of output contents. <1> ~ <17> 13. An information processing device according to claim 12, <19> An encoder; A decoder; A loss function calculation unit; An information processing method of an information processing device including a control unit, The encoder encodes input content, which is a sequence of data, into latent variables; the decoder decodes the latent variables to reconstruct output content; The loss function calculation unit calculates a loss function based on the likelihood of the input content; The control unit updates the latent variables by descending a gradient of the loss function, and causes the decoder to decode the updated latent variables and reconstruct the output content. An information processing method comprising the steps of: <20> an encoder that encodes input content consisting of a sequence of data into latent variables; a decoder for decoding the latent variables to reconstruct output content; a loss function calculation unit that calculates a loss function based on the likelihood of the input content; a control unit that updates the latent variables by descending a gradient of the loss function, and causes the decoder to decode the updated latent variables and reconstruct output content; programs that make a computer function. [Explanation of symbols]

[0228] 31 information processing device, 32 input / output device, 51 communication unit, 52 control unit, 53 memory unit, 71 learning unit, 72 optimization unit, 73 generation unit, 81 model memory unit, 82 music DB, 91 encoder, 92 decoder, 93 loss function calculation unit, 101, 101' likelihood evaluator, 102, 102' reality evaluator

Claims

1. an encoder that encodes input content consisting of a sequence of data into latent variables; a decoder for decoding the latent variables to reconstruct output content; a loss function calculation unit that calculates a loss function based on the likelihood of the input content; a control unit that updates the latent variables by descending a gradient of the loss function, and causes the decoder to decode the updated latent variables and reconstruct output content; An information processing device comprising:

2. The encoder and decoder are trained using a Variational Auto Encoder (VAE) to minimize the reconstruction error while regularizing the posterior distribution by the prior distribution. The information processing device according to claim 1 .

3. The loss function calculation unit is a likelihood evaluator for calculating the likelihood of the input content; a reality evaluator that calculates a reality likelihood that is a likelihood of reality of the input content; Calculating the loss function based on the likelihood as an evaluation result of the likelihood evaluator and the reality likelihood as an evaluation result of the reality evaluator The information processing device according to claim 1 .

4. When the reality likelihood is smaller than a predetermined value, the control unit, when updating the latent variable by descending the gradient of the loss function, updates the latent variable by descending the gradient of the loss function so that the reality likelihood becomes larger, and causes the decoder to decode the updated latent variable to reconstruct the output content. The information processing device according to claim 3 .

5. The control unit updates the latent variables by moving the position in the latent variable space so as to increase the reality and decreases a gradient of the loss function when the reality likelihood is smaller than a predetermined value based on a distribution of the likelihood for each latent variable in a latent variable space that is a space of the latent variables and a distribution of the reality likelihood, and causes the decoder to decode the updated latent variables to reconstruct the output content. The information processing device according to claim 4.

6. The likelihood evaluator obtains a logarithm of the likelihood representing the probability that the input content is a content registered in advance as a sample, as the likelihood. The information processing device according to claim 3 .

7. The likelihood evaluator obtains a logarithm of a likelihood obtained as a product of conditional probabilities for each data sequence, which is a probability that the data sequence constituting the input content is a content registered in advance as a sample. The information processing device according to claim 6.

8. The reality evaluator obtains the reality likelihood by taking a logarithm of the likelihood of the reality, which is the probability that the input content is a human-generated content. The information processing device according to claim 3 .

9. The loss function is composed of a first term based on the likelihood of the input content and a second term based on the reality likelihood. The information processing device according to claim 3 .

10. The first term is a square of the difference between the likelihood of the input content and a predetermined likelihood multiplied by a predetermined constant by a predetermined coefficient. The information processing device according to claim 9.

11. The predetermined coefficient is set to a value greater than 1 when increasing the likelihood of the output content and making it ordinary, and is set to a value less than 1 when decreasing the likelihood of the output content and making it unusual. The information processing device according to claim 10.

12. The predetermined constant is an initial value of the likelihood The information processing device according to claim 10.

13. In the first term, when the likelihood is increased, a positive coefficient is assigned to the likelihood of the input content, and when the likelihood is decreased, a negative coefficient is assigned to the likelihood of the input content. The information processing device according to claim 9.

14. The first term is a sum of squares of differences between the likelihood of each element constituting the input content and a predetermined likelihood constituted by the product of a predetermined coefficient and a constant for each element. The information processing device according to claim 9.

15. When a portion of the input content that is not to be changed is designated as a context, the encoder encodes a portion of the input content other than the context as input partial data and converts it into a latent variable; The decoder decodes the latent variables to reconstruct output partial data; The control unit updates the latent variables by descending a gradient of the loss function, decodes the updated latent variables by the decoder to reconstruct output partial data, and reconstructs output content by integrating the reconstructed output partial data and the context. The information processing device according to claim 3 .

16. The loss function calculation unit calculates the loss function based on a conditional likelihood that is a likelihood of input partial data other than the context in the input content. The information processing device according to claim 15.

17. The loss function calculation unit uses the likelihood of the input content as the reality likelihood of the input content. The information processing device according to claim 15.

18. The control unit gradually decreases a gradient of the loss function by a predetermined magnitude to gradually update the latent variables, and causes the decoder to repeatedly decode the updated latent variables to gradually reconstruct a plurality of output contents. The information processing device according to claim 1 .

19. An encoder; A decoder; A loss function calculation unit; An information processing method of an information processing device including a control unit, The encoder encodes input content, which is a sequence of data, into latent variables; the decoder decodes the latent variables to reconstruct output content; The loss function calculation unit calculates a loss function based on the likelihood of the input content; The control unit updates the latent variables by descending a gradient of the loss function, and causes the decoder to decode the updated latent variables and reconstruct the output content. An information processing method comprising the steps of:

20. an encoder that encodes input content consisting of a sequence of data into latent variables; a decoder for decoding the latent variables to reconstruct output content; a loss function calculation unit that calculates a loss function based on the likelihood of the input content; a control unit that updates the latent variables by descending a gradient of the loss function, and causes the decoder to decode the updated latent variables and reconstruct output content; programs that make a computer function.

Images