Neural network learning device, caption generation device

Abstract

A technology for generating image captions that include emotional expressions based on a VAD model is provided. [Solution] The system includes a caption generation unit that generates a caption for an image based on an image and at least one of a VAD score related to emotional valence, a VAD score related to arousal, and a VAD score related to dominance (the VAD score of 1 or more is a VAD score for text containing an emotional expression that is to be generated as a caption for the image), in accordance with the VAD score of 1 or more.

Description

[Technical Field]

[0001] The present invention relates to image caption generation technology. [Background technology]

[0002] Conventionally, image caption generation technology has focused on generating text that objectively describes the content of an image, and little focus has been placed on generating text related to the emotional aspects of images (see Non-Patent Document 1). However, image caption generation technology that takes emotional elements into consideration is extremely important in systems that provide human interaction, for example, by promoting empathy and enabling the provision of customized content based on the user's emotions. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Liming Xu, Quan Tang, Jiancheng Lv, Bochuan Zheng, Xianhua Zeng, Weisheng Li, "Deep image captioning: A review of methods, trends and future challenges," Neurocomputing, Vol.546, 2023. Summary of the Invention [Problem to be solved by the invention]

[0004] The VAD model, which analyzes emotions along three dimensions - valence, arousal, and dominance - is a model that can accurately capture the complexity of human emotions. However, there was no technology based on the VAD model that could generate image captions that include emotional expressions.

[0005] Therefore, an object of the present invention is to provide a technique for generating image captions that include emotional expressions based on a VAD model. [Means for solving the problem]

[0006] One aspect of the present invention is a neural network training device that trains a neural network including an encoder that generates latent variables from an input image and input text that includes an emotional expression for the image, and a decoder that generates an output image and output text from the latent variables, so that the input image and output image, and the input text and output text, are approximately identical, and training is performed using pairs of images, text, and VAD scores as training data, and the neural network outputs the VAD score in addition to the output image and output text.

[0007] One aspect of the present invention includes a caption generation unit that generates a caption for an image based on an image and at least one or more VAD scores from among a VAD score for valence, a VAD score for arousal, and a VAD score for dominance (the one or more VAD scores are VAD scores for text containing an emotional expression that is to be generated as a caption for the image), in accordance with the one or more VAD scores. [Effects of the Invention]

[0008] According to the present invention, it is possible to generate image captions that include emotional expressions based on a VAD model. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 illustrates the architecture of a multimodal variational autoencoder used in embodiments of the present invention. [Figure 2] FIG. 1 illustrates the architecture of a cross-modal reasoner used in an embodiment of the present invention. [Figure 3] FIG. 1 is a block diagram showing the configuration of a neural network learning device 100. [Figure 4]3 is a flowchart showing the operation of the neural network learning device 100. [Figure 5] FIG. 2 is a block diagram showing the configuration of a caption generation device 200. [Figure 6] 10 is a flowchart showing the operation of the caption generation device 200. [Figure 7] FIG. 2 is a diagram illustrating an example of the functional configuration of a computer that realizes each device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described in detail. Note that components having the same functions are given the same numbers and redundant explanations will be omitted.

[0011] Before describing each embodiment, the notation used in this specification will be explained.

[0012] ^ (caret) represents a superscript, e.g., x y^z Yes z is a superscript to x, and x y^z Yes z is a subscript to x. Also, _ (underscore) represents a subscript. For example, x y_z Yes z is a superscript to x, and x y_z Yes z is a subscript to x.

[0013] The superscripts "^" and "~" such as ^x and ~x for a certain letter x should be written directly above the "x", but due to restrictions on the notation in the specification, they are written as ^x and ~x.

[0014] <Technical background> In the embodiment of the present invention, a multimodal variational autoencoder for images and text is used. Figure 1 shows the architecture of the multimodal variational autoencoder. The latent variable z of the multimodal variational autoencoder is a latent variable z related to emotional valence. v , latent variable z related to arousal level a , the latent variable z for dominance d , latent variables z for images and text c (i.e., z=[z v ;z a ;z d ;z c ]. Also, a multimodal variational autoencoder computes the img An image encoder q takes input and outputs latent variable z. img (z|x img ), Text x txt A text encoder q takes input and outputs latent variable z. txt (z|x txt ), the latent variable z is input and the image x img Image decoder p img (x img |z), the latent variable z is input and the text x txt A text decoder p txt (x txt |z), a latent variable z related to emotional valence v The VAD score s for emotional valence is calculated using the input v The valence predictor f v (z v ), latent variable z related to arousal level a The VAD score s for the level of alertness is calculated using the input a The alertness predictor f a (z a ), the latent variable z for dominance d The VAD score for dominance is calculated as s d dominance predictor f d (z d) where the VAD score is a real number between 0 and 1, with 0 representing negative and 1 representing positive in the VAD score for valence, 0 representing calmness and 1 representing excitement in the VAD score for arousal, and 0 representing controllable and 1 representing uncontrollable in the VAD score for dominance.

[0015] Encoder Q img (z|x img ), q txt (z|x txt ), decoder p img (x img |z), p txt (x txt |z) can be configured as a neural network, for example, as follows:

[0016] (1) q img (z|x img )=N(μ(x img ), σ 2 (x img ))(where μ(x img )=MLP1(x img ), σ 2 (x img )= exp(MLP2(x img ))) (2) q txt (z|x txt )=N(μ(x txt ), σ 2 (x txt ))(where μ(x txt )=MLP3(CLIP t-enc (x txt )), σ 2 (x txt )= exp(MLP4(CLIP t-enc (x txt )))) (3) p img (x img |z)=N(μ(z), σ 2 (z)) (where μ(z)=MLP5(z), σ 2(z)=exp(MLP6(z))) (4) p txt (x txt |z)=Π i p txt ((x txt ) i |z, (x txt ) <i ) (where p txt (x txt |z) is based on a pre-trained T5 model. Here, it is advisable to refer to Reference Non-Patent Document 1 and Reference Non-Patent Document 2 for the CLIP model and the T5 model, respectively.

[0017] (Reference non-patent document 1: Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever, "Learning Transferable Visual Models From Natural Language Supervision," https: / / arxiv.org / abs / 2103.00020, 2021) (Reference non-patent document 2: Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu, "Exploring the limits of transfer learning with a unified text-to-text transformer," Journal of Machine Learning, Research 21, pp.1-67, 2020) Also, MLP is an acronym for multi-layer perceptron, and MLP1, MLP2, MLP3, MLP4, MLP5, and MLP6 represent different multi-layer perceptrons.

[0018] Predictor f v (z v ), f a (z a ), f d (z d ) can be implemented as a neural network that calculates, for example:

number

[0019] Then, we train a multimodal variational autoencoder (i.e., two encoders q) using the set of images, text, and VAD scores as training data. img (z|x img ), q txt (z|x txt ), two decoders p img (x img |z), p txt (x txt |z), three predictors f v (z v ), f a (z a ), f d (z d )) is learned by learning the input and output images (encoder q img (z|x img ) input and decoder p img (x img |z) output), input text and output text (encoder q txt (z|x txt ) input and decoder p txt (x txt|z) outputs) are trained so that they are approximately identical. Here, training to be approximately identical means that, ideally, training should be done to make them completely identical, but in reality, training must be done to make them almost identical due to training time constraints and other factors. Therefore, training is done in such a way that when a certain condition is met, the output is deemed to be identical and the processing is terminated.

[0020] Next, we will explain a cross-modal inference system for images and text, which is constructed using a trained multi-modal variational autoencoder. Figure 2 shows the architecture of the cross-modal inference system. The latent variable z of the cross-modal inference system is the same as the latent variable z of the multi-modal variational autoencoder. The cross-modal inference system is also implemented using an image encoder q img (z|x img ), text decoder p txt (x txt |z), VAD scores for emotional valence s v The latent variable z related to emotional valence is input. v The emotional valence manipulator g outputs v (s v ), VAD scores for alertness a The latent variable z related to the level of arousal is input. a The alertness manipulator g outputs a (s a ), VAD scores for dominance d The latent variable z d The dominance manipulator g outputs d (s d ) where the manipulator g v (s v ), g a (s a ), g d (s d ) are the predictors f v (z v ), f a (z a ), f d (z d) and can be configured as a neural network that calculates, for example,

number

[0021] The cross-modal reasoner takes an image and a VAD score as input and outputs a text, where the output text is a caption for the input image containing the emotional expression corresponding to the input VAD score.

[0022] Finally, we will explain an example of the application of a cross-modal reasoner. Consider an art museum that offers experiences using eyeglass-type devices. The system provided by the museum is configured using a cross-modal reasoner, so it can automatically generate captions that express personalized emotions according to the painting the user is viewing. Below is an example of a dialogue between the user and the system.

[0023] (1) The glasses-type device worn by the user recognizes the painting the user is looking at.

[0024] (2) When a user tweets, "That's a little scary," the system responds, "What's scary is that the man's eyes look empty."

[0025] In this way, the system can respond using richer expressions based on what the user says.

[0026] By using a cross-modal reasoner, it is expected that the user's understanding of paintings and their acquisition of ways of expressing paintings will progress through interaction between the system and the user. It is also expected that the user will gain sympathy from the system.

[0027] First Embodiment Neural network learning device 100 The neural network training device 100 uses training data to train the parameters of the neural network to be trained. Here, the training data is a set of an image, text, and a VAD score. The neural network to be trained includes an encoder that generates latent variables from an input image and input text containing an emotional expression for the image, and a decoder that generates an output image and output text from the latent variables, and outputs a VAD score in addition to the output image and output text. An example of a neural network is the multimodal variational autoencoder described in the <Technical Background>. Therefore, the latent variable z of the neural network is a latent variable z related to emotional valence. v , latent variable z related to arousal level a , the latent variable z for dominance d , latent variables z for images and text c Also, the neural network calculates the image x img An image encoder q takes input and outputs latent variable z. img (z|x img ), Text x txt A text encoder q takes input and outputs latent variable z. txt (z|x txt ), the latent variable z is input and the image x img Image decoder p img (x img |z), the latent variable z is input and the text x txt A text decoder p txt (x txt |z), a latent variable z related to emotional valence v The VAD score s for emotional valence is calculated using the input v The valence predictor f v (zv ), latent variable z related to arousal level a The VAD score s for the level of alertness is calculated using the input a The alertness predictor f a (z a ), the latent variable z for dominance d The VAD score for dominance is calculated as s d dominance predictor f d (z d ) and the parameters of the neural network are therefore the two encoders q img (z|x img ), q txt (z|x txt ) weight and bias parameters of the two decoders p img (x img |z), p txt (x txt |z) weight and bias parameters, three predictors f v (z v ), f a (z a ), f d (z d ) weight parameters and bias parameters. Training is performed so that the input image and output image, and the input text and output text are nearly identical.

[0028] Neural network training device 100 will be described below with reference to FIGS. 3 and 4. FIG. 3 is a block diagram showing the configuration of neural network training device 100. FIG. 4 is a flowchart showing the operation of neural network training device 100. As shown in FIG. 3, neural network training device 100 includes an initialization unit 110, a learning unit 120, a termination condition determination unit 130, and a recording unit 190. Recording unit 190 is a component that appropriately records information necessary for the processing of neural network training device 100. Recording unit 190 records, for example, initialization data used to initialize the neural network. Here, initialization data refers to, for example, initial values ​​of parameters of the neural network.

[0029] The operation of the neural network learning device 100 will be described with reference to FIG.

[0030] In S110, the initialization unit 110 performs initialization processing of the neural network using the initialization data. Specifically, the initialization unit 110 sets an initial value for each parameter of the neural network.

[0031] In S120, the learning unit 120 receives training data as input, performs a process of updating each parameter of the neural network using the training data (hereinafter referred to as parameter update process), and outputs the neural network parameters along with information (e.g., the number of times the parameter update process has been performed) necessary for the termination condition determination unit 130 to determine the termination condition. The learning unit 120 trains the neural network by, for example, backpropagation using a loss function. That is, in each parameter update process, the learning unit 120 performs a process of updating each parameter of the neural network so that the loss function becomes smaller.

[0032] In S130, the termination condition determination unit 130 receives as input the neural network parameters output in S120 and information necessary to determine the termination condition, and determines whether the termination condition, which is a condition for terminating learning, is satisfied (for example, whether the number of times the parameter update process has been performed has reached a predetermined number of repetitions).If the termination condition is satisfied, the neural network parameters obtained in the last S120 execution are output as learned parameters and the process is terminated.If the termination condition is not satisfied, the process returns to S120.

[0033] In the above explanation, the neural network to be trained has three predictors f v (z v ), f a (z a ), f d (z d), the neural network to be trained may include two encoders, two decoders, and at least one predictor.

[0034] <> The caption generation device 200 generates a caption for an image according to one or more VAD scores from an image and at least one or more VAD scores out of a VAD score related to valence, a VAD score related to arousal, and a VAD score related to dominance (the one or more VAD scores are VAD scores for text containing an emotional expression desired to be generated as a caption for the image). Here, the caption generation device 200 is configured using a neural network trained using the neural network training device 100. Details will be described later. Note that the image to be captioned is not limited to visual art, and may be a real-life image such as a landscape or everyday scene. It may also be a video.

[0035] The caption generation device 200 will be described below with reference to FIGS. 5 and 6. FIG. 5 is a block diagram showing the configuration of the caption generation device 200. FIG. 6 is a flowchart showing the operation of the caption generation device 200. As shown in FIG. 5, the caption generation device 200 includes a first latent variable generation unit 210, a second latent variable generation unit 220, a latent variable update unit 230, a latent variable decoding unit 240, and a recording unit 290. The recording unit 290 is a component that appropriately records information necessary for the processing of the caption generation device 200. The recording unit 290 records, in advance, parameters of a neural network trained using, for example, the neural network learning device 100. Note that a component including the first latent variable generation unit 210, the second latent variable generation unit 220, the latent variable update unit 230, and the latent variable decoding unit 240 is referred to as a caption generation unit 205.

[0036] The operation of the caption generation device 200 will be described with reference to FIG.

[0037] In S210, the first latent variable generation unit 210 receives an image as input, generates latent variables from the image (wherein the latent variables include latent variables related to valence, latent variables related to arousal, latent variables related to dominance, and latent variables related to the image), and outputs them. When the neural network is a multimodal variational autoencoder described in <Technical Background>, the first latent variable generation unit 210 generates latent variables from the image encoder q img (z|x img ) can be used to construct the

[0038] In S220, the second latent variable generation unit 220 receives one or more VAD scores as input, and if one or more VAD scores include a VAD score related to valence, generates a latent variable related to valence from the VAD scores related to valence; if one or more VAD scores include a VAD score related to arousal, generates a latent variable related to arousal from the VAD scores related to arousal; and if one or more VAD scores include a VAD score related to dominance, generates a latent variable related to dominance from the VAD scores related to dominance, and outputs the latent variable. Hereinafter, the latent variable generated by the second latent variable generation unit 220 is referred to as a VAD latent variable. Therefore, the second latent variable generation unit 220 receives one or more VAD scores as input, generates one or more VAD latent variables from the one or more VAD scores, and outputs the latent variable. When the neural network is a multimodal variational autoencoder described in the <Technical Background> section, the second latent variable generation unit 220 generates and outputs a latent variable related to valence from the manipulator g v (s v ), g a (s a ), g d (s d ) can be used to construct the

[0039] In S230, the latent variable update unit 230 receives the latent variables generated in S210 and the VAD latent variables generated in S220 as input, and updates and outputs the latent variables by replacing the latent variables related to valence, the latent variables related to arousal, and the latent variables related to dominance contained in the latent variables with the latent variables related to valence, the latent variables related to arousal, and the latent variables related to dominance, which are VAD latent variables.

[0040] In S240, the latent variable decoding unit 240 receives the latent variables updated in S230 as input, generates a caption for the image input in S210 from the updated latent variables, and outputs it. If the neural network is a multimodal variational autoencoder described in the <Technical Background> section, the latent variable decoding unit 240 uses the text decoder p txt (x txt |z).

[0041] According to an embodiment of the present invention, it is possible to generate image captions that include emotional expressions based on a VAD model. In particular, by using a VAD model, it is possible to generate text that continuously reflects changes in emotions.

[0042] <Additional Notes> The functions performed by the components described herein may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (Central Processing Units), conventional circuits, and / or combinations thereof, programmed to perform the described functions. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. A processor may also be a programmed processor that executes programs stored in memory.

[0043] In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions.

[0044] If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and / or processor.

[0045] The various processes described above can be implemented by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in Figure 7, and operating the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc.

[0046] The program describing the processing contents can be recorded on a computer-readable recording medium, which may be, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, or any other suitable recording medium.

[0047] The program may be distributed, for example, by selling, transferring, lending, etc. a portable recording medium such as a DVD or CD-ROM on which the program is recorded. Furthermore, the program may be stored in a storage device of a server computer, and then transferred from the server computer to another computer via a network, thereby distributing the program.

[0048] A computer that executes such a program may first temporarily store the program recorded on a portable recording medium or transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored in its storage device and executes the process in accordance with the read program. Alternatively, the computer may directly read the program from a portable recording medium and execute the process in accordance with the program. Furthermore, the computer may execute the process in accordance with the received program each time a program is transferred from a server computer to the computer. The server computer may not transfer the program to the computer, but may instead execute the process through a so-called ASP (Application Service Provider) service, which realizes the processing function by issuing an execution instruction and obtaining the results. Furthermore, the server computer may execute the process on a terminal using a so-called SaaS (Software as a Service) service, which allows users to use part of the server computer along with the program. In this embodiment, the program includes information used for computer processing that is equivalent to a program (such as data that is not a direct instruction to the computer but has properties that define computer processing).

[0049] Furthermore, in this embodiment, the device is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be realized by hardware.

[0050] The present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention. Furthermore, the processes described in the above embodiments may not only be executed in chronological order according to the order described, but may also be executed in parallel or individually depending on the processing capacity of the device that executes the processes or as needed.

Claims

1. A neural network training device that trains a neural network including an encoder that generates latent variables from an input image and input text including an emotional expression for the image, and a decoder that generates an output image and output text from the latent variables, so that the input image and the output image, and the input text and the output text are approximately identical, The training is performed using a set of images, text, and VAD scores as training data. The neural network outputs a VAD score in addition to an output image and output text. Neural network learning device.

2. 2. The neural network learning device according to claim 1, the latent variables of the neural network include a latent variable for emotional valence, a latent variable for arousal, a latent variable for dominance, and a latent variable for image and text; The neural network includes an image encoder that receives an image as an input and outputs a latent variable, a text encoder that receives text as an input and outputs a latent variable, an image decoder that receives a latent variable as an input and outputs an image, a text decoder that receives a latent variable as an input and outputs a text, a valence predictor that receives a latent variable related to valence as an input and outputs a VAD score related to valence, an arousal predictor that receives a latent variable related to arousal as an input and outputs a VAD score related to arousal, and a dominance predictor that receives a latent variable related to dominance as an input and outputs a VAD score related to dominance. A neural network learning device characterized by:

3. a caption generation unit that generates a caption for an image according to one or more VAD scores from an image and at least one or more VAD scores out of a VAD score for valence, a VAD score for arousal, and a VAD score for dominance (the one or more VAD scores are VAD scores for text including an emotional expression to be generated as a caption for the image); and A caption generator including:

4. 4. The caption generating device according to claim 3, The caption generation unit a first latent variable generation unit that generates latent variables from the image (wherein the latent variables include a latent variable related to valence, a latent variable related to arousal, a latent variable related to dominance, and a latent variable related to image); a second latent variable generation unit that generates a latent variable related to valence from the VAD score related to valence if the one or more VAD scores include a VAD score related to valence, generates a latent variable related to arousal from the VAD score related to arousal if the one or more VAD scores include a VAD score related to arousal, and generates a latent variable related to dominance from the VAD score related to dominance if the one or more VAD scores include a VAD score related to dominance; a latent variable updating unit that updates the latent variables by replacing the latent variables related to valence, the latent variables related to arousal, and the latent variables related to dominance included in the latent variables with the latent variables related to valence, the latent variables related to arousal, and the latent variables related to dominance generated by the second latent variable generating unit; and a latent variable decoding unit that generates the caption from the updated latent variables. A caption generating device characterized by:

Images