Learning method and device, recording medium, learning completion model and text generation device
By combining the learning models of the text generation unit and the statistical inference unit, the parameters are updated using image feature quantity and text error, and the metadata inference unit is introduced to adjust the style, which solves the problem of low accuracy in the automatic image generation text and the style does not meet user needs. Problem, achieving high accuracy and user intention matching text generation.
Patent Information
- Application Number
- CN202080030768.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-04-24
- Filing Date
- 2020-04-15
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2040-04-15
AI Technical Summary
During the learning process, the existing automatic image generation text model has problems such as slow parameter convergence and low text generation accuracy, especially the image has a large degree of freedom of text expression, making it difficult to obtain a high-precision text generation model.
Using a learning model combining the text generation unit and the statistical inference unit, the parameters are updated through the error between the image feature quantity and the text, the freedom of text generation is limited, the accuracy of text generation is improved, and the metadata inference unit is introduced to the User attributes adjust the style of the text to generate text close to the user's intention.
It realizes the accuracy of text generation in the automatic image generation text, and can adjust the style according to user attributes to generate text close to the user's intentions, solving the problems of low text generation accuracy and text style not meeting user needs in the prior art.
Smart Images

Figure CN113767420B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a learning method and device, a program, a learning completion model and a text generation device, and in particular to a machine learning technology, an image processing technology and a natural language processing technology for realizing the generation of text related to an image. Background Art
[0002] As a technology for automatically generating an image title from an image, there is a technology described in Non-Patent Document 1. In Non-Patent Document 1, an image title generator is proposed, which is formed by combining a convolutional neural network (CNN) that extracts image features from an image and a recurrent neural network (RNN) that generates a title based on natural language based on the extracted image features. In addition, the title is included in the concept of text.
[0003] Non-patent document 2 generally summarizes conventional techniques for automatically generating image titles.
[0004] Non-patent document 3 describes a study in which a conditional generative adversarial network (ConditionalGAN) is used to output various genres from one model by taking inputs other than images during learning.
[0005] Previous technical literature
[0006] Non-patent literature
[0007] Non-patent literature 1: Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan "Show and Tell: A Neural Image Caption Generator", arXiv: 1411.4555
[0008] Non-patent document 2: Ushihisa Yoshitaka, "Automatic generation of image titles" [searched on December 22, 2018], Internet<URL:https: / / www.slideshare.net / YoshitakaUshiku / ss-57148161>
[0009] Non-patent literature 3: Bo Dai, Sanja Fidler, Raquel Urtasun, Dahua Lin "TowardsDiverse and Natural Image DeScriptions via a Conditional GAN", arXiv: 1703.06029 Summary of the invention
[0010] Technical issues to be solved by the invention
[0011] In order to obtain a text generation model for automatically generating text corresponding to an image, it is considered to perform machine learning using paired data of a learning image and a correct text corresponding to the image. However, the degree of freedom of text expression for an image is large, and in previous learning methods, there are problems such as slow convergence of learning model parameters and / or insufficient accuracy of inference of the text generation model obtained through learning.
[0012] The present invention is completed in view of such a situation, and its purpose is to provide a learning method and device, a program, a learned model and a text generation device that can improve the accuracy of inference of a model used in the process of automatically generating text from an image.
[0013] Means for solving technical problems
[0014] A learning method according to one aspect of the present invention is a learning method comprising the following steps: using a learning model, the learning model comprising: a feature extraction unit for extracting image features from an image; a text generation unit for generating an inferred text related to the image based on the image features; and a statistic inference unit for receiving an input of an inferred text or a correct answer text, inferring statistics of an image corresponding to the input text and generating inferred image statistics; receiving an input of a plurality of learning data consisting of a first image for learning and a first text as a correct answer text for the first image; inputting the first image into the feature extraction unit to obtain the first image features ; Input the first image feature into the text generation unit to obtain the first inferred text; calculate the first error between the first inferred text and the first text; input the first text into the statistics inference unit to obtain the first inferred image statistic; calculate the second error between the first inferred image statistic and the first image statistic calculated based on the first image; input the first inferred text into the statistics inference unit to obtain the second inferred image statistic; calculate the third error between the second inferred image statistic and the first image statistic; update the parameters of the statistics inference unit according to the second error; and update at least the parameters of the text generation unit according to the first error and the third error.
[0015] The learning method of this method uses a learning model composed of a combined text generation unit and a statistic inference unit, and simultaneously performs learning of the text generation unit and the statistic inference unit. According to this method, a mechanism is introduced to update the parameters of the text generation unit using the third error between the first inferred image statistic inferred by the statistic inference unit based on the inference result of the text generation unit and the first image statistic of the correct answer, and learning is performed with restrictions imposed on image statistics for the generation of text with a high degree of freedom. In this way, a model capable of generating highly accurate text based on a provided image can be obtained.
[0016] The update timing of the parameters of the text generation unit based on the first error and the update timing of the parameters of the text generation unit based on the third error may be different timings or simultaneous timings. Furthermore, the timing of updating the parameters of the text generation unit and the timing of updating the parameters of the statistic inference unit may be different timings or simultaneous timings.
[0017] In another aspect of the present invention, a learning method can be configured as follows: a text generation unit is a module for generating inferred text based on image features and metadata about a user associated with the image; a learning model further includes a metadata inference unit, which receives input of an inferred text or a correct text, infers metadata about a user corresponding to the input text, and generates inferred metadata; the learning data further includes first metadata about a first user associated with a first image; the first inferred text is obtained by inputting the first image features and the first metadata into the text generation unit; the learning method further includes the following steps: inputting the first text into the metadata inference unit to obtain the first inferred metadata; calculating a fourth error with the first metadata for the first inferred metadata; updating parameters of the metadata inference unit based on the fourth error; inputting the first inferred text into the metadata inference unit to obtain a second inferred metadata; and calculating a fifth error with the first metadata for the second inferred metadata, and updating at least parameters of the text generation unit based on the fifth error.
[0018] According to this aspect, the style of the generated text can be changed according to the attributes of the user, thereby generating a text close to the user's intention.
[0019] In a learning method according to another aspect of the present invention, the first user may correspond to at least one of a photographer, a creator, a publisher, a provider, and an owner of the image, and the first metadata may include information indicating an attribute of the first user.
[0020] In a learning method according to still another aspect of the present invention, a configuration may be adopted in which the first text includes a text created by a first user.
[0021] In another embodiment of the present invention, the learning method can be configured as follows: the first image is an image posted by the first user to the social networking service, the first text is an article posted by the first user to the social networking service, and the first metadata includes at least a portion of the user information of the first user registered in the social networking service.
[0022] In a learning method according to another aspect of the present invention, the first image may be an image obtained by converting an email received by the first user into image information, and the first text may include a text of an email sent by the first user.
[0023] A learning method according to another aspect of the present invention may be configured such that the learning method further includes a step of updating a parameter of the feature quantity extraction unit based on the first error.
[0024] In a learning method according to another aspect of the present invention, a learning model may be configured using a hierarchical neural network.
[0025] A program according to another aspect of the present invention is a program for causing a computer to execute the processing of the learning method according to any aspect of the present invention.
[0026] A learned model according to another aspect of the present invention is a learned text generation unit that has been learned using the learning method according to any aspect of the present invention.
[0027] A document generation device according to another aspect of the present invention is a document generation device including: a document generation unit that has been learned using the learning method according to any aspect of the present invention; and a feature extraction unit.
[0028] A learning device according to another embodiment of the present invention is a learning device comprising: a learning data acquisition unit, which receives input of a plurality of learning data consisting of a first image for learning and a first text which is a correct answer text for the first image; a learning model, which is trained using the learning data, and the learning model includes: a feature extraction unit, which extracts image features based on the input image; a text generation unit, which generates an inferred text related to the image based on the image features; and a statistic inference unit, which receives input of the inferred text or the first text, infers the statistics of the image corresponding to the input text and generates an inferred image statistic; and a first error calculation unit, which calculates the error generated from the text by inputting the first image features into the text generation unit. a first error calculating unit for calculating a first inferred image statistic obtained from a statistic inference unit by inputting the first text into the statistic inference unit and a first error between the first text, the first image feature being obtained from a feature extraction unit by inputting the first image into the feature extraction unit; a second error calculating unit for calculating a second error between a first inferred image statistic obtained from a statistic inference unit by inputting the first text into the statistic inference unit and a first image statistic calculated based on the first image, and for calculating a third error between a second inferred image statistic obtained from a statistic inference unit by inputting the first inferred text into the statistic inference unit and the first image statistic; and a parameter updating unit for updating parameters of a text generating unit and a statistic inference unit according to the first error and the second error, and for updating at least the parameters of the text generating unit according to the third error.
[0029] The update timing of the parameters of the text generation unit and the statistic inference unit based on the parameter updating unit can be different timings or simultaneous timings. The parameter updating unit can update the parameters of the text generation unit based on the first error and the third error, and update the parameters of the statistic inference unit based on the second error. The parameter updating unit can update the parameters of the statistic inference unit based on the third error.
[0030] In a learning device involved in another embodiment of the present invention, the structure may be as follows: the text generation unit is a module that generates inferred text based on image features and metadata about a user associated with the image, the learning model also includes a metadata inference unit, the metadata inference unit receives an input of the inferred text or the first text, infers the metadata of the user corresponding to the input text and generates the inferred metadata, the learning data also includes the first metadata about the first user associated with the first image, and the first inferred text is obtained by inputting the first image features and the first metadata into the text generation unit. The learning device also has a third error calculation unit: calculates a fourth error between the first inferred metadata obtained from the metadata inference unit by inputting the first text into the metadata inference unit and the first metadata, and calculates a fifth error between the second inferred metadata obtained from the metadata inference unit by inputting the first inferred text into the metadata inference unit and the first image statistic, and the parameter updating unit updates the parameters of the metadata inference unit according to the fourth error, and at least updates the parameters of the text generation unit according to the fifth error.
[0031] A learning device according to another embodiment of the present invention is a learning device comprising at least one processor and a non-temporary computer-readable medium storing instructions for execution by the processor, wherein the processor functions as a learning model by executing the instructions, and the learning model comprises: a feature extraction unit for extracting image features from an image; a text generation unit for generating an inferred text related to the image based on the image features; and a statistic inference unit for receiving an input of an inferred text or a correct answer text, inferring statistics of an image corresponding to the input text, and generating an inferred image statistic, wherein the learning device performs a process including the following steps: receiving an input of a plurality of learning data formed by combining a first image for learning and a first text as a correct answer text for the first image; calculating A first error between a first inferred text obtained from a text generating unit by inputting a first image feature into a text generating unit and the first text, wherein the first image feature is obtained from a feature extracting unit by inputting a first image into a feature extracting unit; calculating a second error between a first inferred image statistic obtained from a statistic inference unit by inputting the first text into a statistic inference unit and a first image statistic calculated based on the first image, and calculating a third error between a second inferred image statistic obtained from a statistic inference unit by inputting the first inferred text into the statistic inference unit and the first image statistic; and updating parameters of the text generating unit and the statistic inference unit according to the first error and the second error, and at least updating parameters of the text generating unit according to the third error.
[0032] Effects of the Invention
[0033] According to the present invention, text corresponding to an image can be accurately generated. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 This is a functional block diagram showing an example of the title generating device according to the first embodiment.
[0035] Figure 2 This is a functional block diagram showing a configuration example of a learning device according to the first embodiment.
[0036] Figure 3 This is a block diagram schematically showing data input to each module of a feature extraction unit, a text generation unit, a statistic estimation unit, and a statistic calculation unit of a learning device, and data output from each module.
[0037] Figure 4 This is a flowchart exemplarily showing the steps of the learning method according to the first embodiment.
[0038] Figure 5 This is a flowchart exemplarily showing the steps of the learning method according to the first embodiment.
[0039] Figure 6 This is a block diagram schematically showing a state of a process of updating a parameter of a text generating unit based on a first error calculated by a first error calculating unit of a learning device.
[0040] Figure 7 This is a block diagram schematically showing a state of processing for updating parameters of a statistic inference unit based on a second error calculated by a second error calculation unit of a learning device.
[0041] Figure 8 This is a block diagram schematically showing a state of processing for updating parameters of a text generating unit and a statistic estimating unit based on a third error calculated by a second error calculating unit of a learning device.
[0042] Fig. 9 This is a functional block diagram showing an example of a title generating device according to the second embodiment.
[0043] Fig.10 This is a functional block diagram showing a configuration example of a learning device according to the second embodiment.
[0044] Fig.11 This is a block diagram schematically showing data input to each module of a feature extraction unit, a text generation unit, a metadata inference unit, a statistic inference unit, and a statistic calculation unit of a learning device, and data output from each module.
[0045] Fig.12 This is a flowchart exemplarily showing the steps of the learning method according to the second embodiment.
[0046] Fig.13This is a flowchart exemplarily showing the steps of the learning method according to the second embodiment.
[0047] Fig.14 This is a block diagram schematically showing a state of a process of updating a parameter of a text generating unit based on a first error calculated by a first error calculating unit of a learning device.
[0048] Fig.15 This is a block diagram schematically showing a state of processing for updating parameters of a metadata estimation unit based on a fourth error calculated by a third error calculation unit of a learning device.
[0049] Fig.16 This is a block diagram schematically showing a state of processing for updating parameters of a statistic inference unit based on a second error calculated by a second error calculation unit of a learning device.
[0050] Fig.17 This is a block diagram schematically showing a state of processing for updating parameters of a text generating unit and a metadata estimating unit based on a fifth error calculated by a third error calculating unit of a learning device.
[0051] Fig.18 This is a block diagram schematically showing a state of processing for updating parameters of a text generating unit and a statistic estimating unit based on a third error calculated by a second error calculating unit of a learning device.
[0052] Fig.19 This is a block diagram showing an example of the overall configuration of a computer system that implements the title generation function involved in Application Example 1.
[0053] Fig. 20 This is a conceptual diagram of the data stored in the data storage department.
[0054] Fig.21 This is a block diagram showing an example of the hardware configuration of a computer. DETAILED DESCRIPTION
[0055] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.
[0056] <<Overview of Document Generating Device According to First Embodiment>>
[0057] Figure 1 1 is a functional block diagram showing an example of a title generation device according to a first embodiment of the present invention. The title generation device 10 takes an image IMG as input, generates and outputs an inferred text PTX related to the input image IMG. The image IMG may be, for example, a digital photo image taken by a camera, a read image read by a scanner, a captured image such as a screenshot, or an illustration created by an image forming tool.
[0058] The inferred text PTX is a title corresponding to the content of the image IMG, and is character information expressed based on natural language. The term "text" includes the concept of "title". The term "text" or "title" mentioned in this specification includes an array of multiple words that convey a meaning, and includes the concepts of sentences, articles, phrases, or paragraphs. The characters used in the text may include symbols, emoticons, and pictorial characters. In this specification, the term "generate" includes the concept of the term "infer".
[0059] The title generation device 10 includes an image acquisition unit 12, a feature extraction unit 16, and a text generation unit 18. The title generation device 10 can be implemented by a computer system including one or more computers. That is, the functions of the image acquisition unit 12, the feature extraction unit 16, and the text generation unit 18 can be implemented by a combination of computer hardware and software. Software has the same meaning as "program".
[0060] The image acquisition unit 12 is an interface for taking in data of the image IMG. The image acquisition unit 12 may be composed of a data input terminal for taking in data of the image IMG from other signal processing units outside or inside the device. The metadata acquisition unit 14 is an interface for taking in data of the metadata MTD. The metadata acquisition unit 14 may be composed of a data input terminal for taking in metadata MTD from other signal processing units outside or inside the device.
[0061] Furthermore, the image acquisition unit 12 may adopt a wired or wireless communication interface unit, a media interface unit that reads and writes a portable external storage medium such as a memory card, or an appropriate combination of these methods.
[0062] The image IMG inputted via the image acquisition unit 12 is sent to the feature extraction unit 16. The feature extraction unit 16 extracts image features from the inputted image IMG and outputs them. The feature extraction unit 16 may be, for example, a learned model using a hierarchical neural network represented by a convolutional neural network (CNN). The feature extraction unit 16 may utilize a network portion equivalent to an encoder of a CNN used for image classification tasks, etc.
[0063] A "neural network" is a mathematical model of information processing that simulates the mechanism of the brain and nervous system. Processing using a neural network can be implemented using a computer. A processing unit including a neural network can be configured as a program module. In this specification, a neural network is sometimes simply labeled as a "network."
[0064] The image feature quantity output from the feature quantity extraction unit 16 is sent to the text generation unit 18 .
[0065] The text generation unit 18 generates an inferred text PTX based on the provided image feature quantity. The text generation unit 18 is composed of a learning model including a hierarchical neural network, and obtains a function of generating text from an image through machine learning using a learning method described later. The title generation device 10 is an example of a "text generation device" in the present invention.
[0066] <<Configuration Example 1 of Learning Device>>
[0067] Next, a learning method for improving the inference function of the text generating unit 18 will be described.
[0068] Figure 2 1 is a functional block diagram showing a configuration example of the learning device 20 according to the first embodiment. The learning device 20 is connected to a learning data storage unit 60. The learning data storage unit 60 includes a storage device that stores learning data LDi required for the learning device 20 to perform machine learning in advance. "Learning data" is training data used for machine learning, and has the same meaning as "learning data" or "training data".
[0069] In the machine learning of this embodiment, a lot of learning data LDi=(Ii, Ti) which is a combination of an image Ii and a text Ti describing the image Ii is used. i is an index number for identifying the learning data, and for example, when the total number of prepared learning data is N, i can be an integer greater than 0 and less than N-1. For example, a set of n+1 sets of learning data from i=0 to i=n is labeled as follows.
[0070] {(I0, T0), (I1, T1),…(Ii, Ti),…(In, Tn)}
[0071] In addition, when performing small batch processing, if the number of learning data in the small batch is set to m, m groups of learning data are selected from N groups.
[0072] The functions of the learning data storage unit 60 and the learning device 20 can be realized by a combination of computer hardware and software. Here, an example in which the learning data storage unit 60 and the learning device 20 are configured as different devices is described, but these functions can be realized by one computer or by two or more computers sharing the processing functions.
[0073] For example, the learning data storage unit 60 and the learning device 20 may be connected to each other via a telecommunication line (not shown). The term "connection" is not limited to wired connection, but also includes the concept of wireless connection. The telecommunication line may be a local area network or a wide area network.
[0074] With such a configuration, generation of learning data and learning of a generation model can be performed without being mutually restricted in terms of both physical and temporal aspects.
[0075] The learning device 20 reads the learning data LDi from the learning data storage unit 60 and performs machine learning. The learning device 20 can read the learning data LDi and update the parameters in small batches that collect a plurality of learning data LDi. Figure 2 In order to simplify the explanation, the process of processing one set of learning data LDi=(Ii, Ti) is shown, but when performing small batch learning, multiple sets (for example, m sets) of learning data included in the small batch are processed together.
[0076] The learning device 20 includes a data acquisition unit 22 , a feature extraction unit 26 , a text generation unit 28 , a statistic inference unit 32 , a statistic calculation unit 34 , a first error calculation unit 41 , a second error calculation unit 42 , and an optimizer 46 .
[0077] The data acquisition unit 22 is an interface for taking in the learning data LDi. Figure 1 The data acquisition unit 22 may have the same structure as the image acquisition unit 12. That is, the data acquisition unit 22 may be composed of a data input terminal for taking in the learning data LDi from other signal processing units outside or inside the device. Furthermore, the data acquisition unit 22 may adopt a wired or wireless communication interface unit, or a media interface unit for reading and writing a portable external storage medium such as a memory card, or an appropriate combination of these methods.
[0078] The feature extraction unit 26 calculates the image feature Fi based on the input image Ii. Figure 1 The image feature quantity Li calculated by the feature quantity extraction unit 26 is represented by a multi-dimensional vector. For example, the image feature quantity Fi can be represented by a 512-dimensional vector. The image feature quantity Fi calculated by the feature quantity extraction unit 26 is sent to the text generation unit 28.
[0079] The text generation unit 28 includes Figure 1 The text generation unit 28 has the same model structure of a hierarchical neural network as the text generation unit 18. The text generation unit 28 generates an inferred text T^i based on the provided image feature quantity Fi. The inferred text T^i output from the text generation unit 28 is sent to the first error calculation unit 41. In addition, the inferred text T^i output from the text generation unit 28 is sent to the statistic inference unit 32.
[0080] The first error calculation unit 41 calculates the error between the inferred text T^i generated by the text generation unit 28 and the text Ti as the teacher data. The first error calculation unit 41 uses a loss function to evaluate the error. The error between the inferred text T^i and the text Ti as the correct answer text is called the first error Er1. The first error Er1 calculated by the first error calculation unit 41 is sent to the optimizer 46.
[0081] The statistic inference unit 32 infers the image statistics of the image corresponding to the text based on the provided text and outputs the inferred image statistics. The image statistics are, for example, statistics that can be obtained through statistical calculation based on the image data, such as the average value of the color components of R (red), G (green), and B (blue), or the distribution of colors, a color histogram, etc. The image statistics can be used as an indicator for evaluating the atmosphere of the entire image. If a text Ti input via the data acquisition unit 22 is provided, the statistic inference unit 32 generates an inferred image statistic IS^i of the image corresponding to the text Ti based on the text Ti. The inferred image statistic IS^i output from the statistic inference unit 32 is sent to the second error calculation unit 42.
[0082] The statistic calculation unit 34 calculates the actual image statistic ISi of the image Ii based on the image Ii inputted via the data acquisition unit 22. The image statistic ISi outputted from the statistic calculation unit 34 is sent to the second error calculation unit 42.
[0083] The second error calculation unit 42 calculates the error between the inferred image statistic IS^i generated by the statistic inference unit 32 and the image statistic ISi corresponding to the teacher data. The second error calculation unit 42 evaluates the error using a loss function. The error between the inferred image statistic IS^i and the image statistic ISi as the correct answer is referred to as the second error Er2. The second error Er2 calculated by the second error calculation unit 42 is sent to the optimizer 46.
[0084] The optimizer 46 performs the following processing: updates the network parameters of the text generation unit 28 and the statistic inference unit 32 according to the calculation results of the first error calculation unit 41 and the second error calculation unit 42. The network parameters include the filter coefficients of the filters used in the processing of each layer (the weights of the connections between nodes) and the deviations of the nodes.
[0085] The optimizer 46 uses the calculation results of the errors obtained from the first error calculation unit 41 and the second error calculation unit 42 to perform parameter calculation processing for calculating the update amount of the network parameters of the text generation unit 28 and the statistic inference unit 32, and parameter update processing for updating the network parameters of the text generation unit 28 and the statistic inference unit 32 according to the calculated parameter update amount. The optimizer 46 updates the parameters according to an algorithm such as the error back propagation method represented by the gradient descent method.
[0086] Then, in the learning device 20, the inferred text T^i generated by the text generation unit 28 is output to the statistic inference unit 32, and the same learning as above is performed. That is, the inferred text T^i generated by the text generation unit 28 is input to the statistic inference unit 32.
[0087] The statistic inference unit 32 generates an inferred image statistic IS^^i of an image corresponding to the inferred text Ti^i based on the provided inferred text Ti. The inferred image statistic IS^^i outputted from the statistic inference unit 32 is sent to the second error calculation unit 42.
[0088] The second error calculation unit 42 calculates the error between the estimated image statistic IS^^i generated by the statistic inference unit 32 and the image statistic ISi as the teacher data. The error between the estimated image statistic IS^^i and the image statistic ISi as the correct answer is referred to as the third error Er3. The third error Er3 calculated by the second error calculation unit 42 is sent to the optimizer 46.
[0089] The optimizer 46 updates the network parameters of the text generation unit 28 and the statistic inference unit 32 using the third error Er3 obtained from the second error calculation unit 42. In addition, the optimizer 46 only needs to calculate at least the update amount of the parameters of the text generation unit 28 based on the third error Er3 and update the parameters of the text generation unit 28, and the parameters of the statistic inference unit 32 may not be updated based on the third error Er3.
[0090] The timing for updating the parameters of the document generation unit 28 and the statistic estimation unit 32 is not particularly limited. The timing for updating the parameters of the document generation unit 28 and the timing for updating the parameters of the statistic estimation unit 32 may be different timings or may be the same timing.
[0091] Furthermore, the timing of updating the parameters of the text generating unit 28 based on the parameter update amount of the text generating unit 28 calculated based on the first error and the timing of updating the parameters of the text generating unit 28 based on the parameter update amount of the text generating unit 28 calculated based on the third error may be different timings or may be simultaneous timings. For example, the optimizer 46 may combine the parameter update amount of the text generating unit 28 calculated based on the first error and the parameter update amount of the text generating unit 28 calculated based on the third error, and perform the parameter update processing of the text generating unit 28 based on the combined parameter update amount.
[0092] Figure 3It is a block diagram schematically showing the data input to each module of the feature extraction unit 26, the text generation unit 28, the statistic inference unit 32 and the statistic calculation unit 34 in the learning device 20 and the data output from each module. The first error Er1 is calculated based on the inferred text Ti output from the text generation unit 28 and the correct text Ti. The text Ti or the inferred text Ti is selectively input to the statistic inference unit 32. The statistic inference unit 32 receives the input of the text Ti and outputs the inferred image statistic ISi. The second error Er2 is calculated based on the inferred image statistic ISi output from the statistic inference unit 32 and the correct image statistic ISi.
[0093] Then, the statistic inference unit 32 receives the input of the inferred text Ti and outputs the inferred image statistic ISi. The third error Er3 is calculated based on the inferred image statistic ISi output from the statistic inference unit 32 and the correct image statistic ISi.
[0094] The data acquisition unit 22 is an example of a "learning data acquisition unit" in the present invention. The image Ii is an example of a "first image for learning" in the present invention. The text Ti is an example of a "first text" in the present invention. The combination of the feature extraction unit 26, the text generation unit 28, and the statistic inference unit 32 is an example of a "learning model" in the present invention. The image feature Fi is an example of a "first image feature" in the present invention. The inferred text T^i is an example of a "first inferred text" in the present invention. The image statistic ISi is an example of a "first image statistic" in the present invention. The inferred image statistic IS^i is an example of a "first inferred image statistic" in the present invention. The inferred image statistic IS^^i inferred based on the inferred text T^i is an example of a "second inferred image statistic" in the present invention. The optimizer 46 is an example of a "parameter updating unit" in the present invention.
[0095] <<Example of a learning method using the learning device 20>>
[0096] Figure 4 and Figure 5 This is a flowchart exemplarily showing the steps of a learning method according to an embodiment of the present invention. Figure 4 and Figure 5 A part or all of the steps of the flowchart shown are executed by the processor functioning as the learning device 20 .
[0097] First, in step S1 , a learning data set is prepared. That is, a plurality of sets of combinations of images Ii and texts Ti are prepared. The function of generating a learning data set may be incorporated into the learning device 20 or may be incorporated into a device different from the learning device 20 .
[0098] Step S1 includes the step of preparing learning data in advance in a state where the learning device 20 can read the learning data. Also, step S1 includes the step of generating learning data as needed. The learning device 20 reads the learning data prepared in step S1. The learning device 20 can acquire learning data in small batches including a plurality of learning data.
[0099] In step S2 , the learning device 20 inputs the image Ii to the feature extraction unit 26 to obtain the image feature Fi.
[0100] In step S3, the learning device 20 inputs the image feature amount Fi to the text generating unit 28 to obtain the inferred text Ti.
[0101] In step S4, the first error calculation unit 41 calculates the error (first error Er1) between the inferred text Ti and the text Ti.
[0102] In step S7, the learning device 20 inputs the text Ti into the statistic inference unit 32 to obtain the inferred image statistic IS^i.
[0103] In step S8, the second error calculation unit 42 calculates the error (second error Er2) between the estimated image statistic IS2i and the image statistic ISi that can be calculated from the image Ii. The statistic calculation unit 34 calculates the image statistic ISi from the image Ii at an appropriate timing before step S8.
[0104] In step S9, the optimizer 46 updates the parameters of the document generating unit 28 and the statistic estimating unit 32 based on the errors calculated in steps S4 and S8. The parameter updating process is performed in units of small batches.
[0105] After step S9, the same processing is performed using the inferred text T^i inferred by the text generating unit 28 as the inputs of the above steps S7 to S9.
[0106] That is, in Figure 5 In step S17, the learning device 20 inputs the inferred text T^i into the statistic inference unit 32 to obtain the inferred image statistic IS^^i.
[0107] In step S18, the second error calculation unit 42 calculates the error (third error Er3) between the estimated image statistic IS^^i and the image statistic ISi which can be calculated from the image Ii.
[0108] In step S19, the optimizer 46 updates the parameters of the text generation unit 28 and the statistic inference unit 32 based on the error calculated in step S18. In addition, the optimizer 46 only needs to update at least the parameters of the text generation unit 28 using the third error Er3 calculated in step S18. For example, in step S19, the optimizer 46 may fix the parameters of the statistic inference unit 32 and exclude them from the update object, and only update the parameters of the text generation unit 28. In addition, in step S19, the optimizer 46 may update the parameters of the text generation unit 28 based on the first error Er1 calculated in step S4 and the third error Er3 calculated in step S18.
[0109] In step S20, the learning device 20 determines whether to end the learning. The learning end condition can be determined based on the error value or the number of parameter updates. As a method based on the error value, for example, the error convergence within a specified range can be used as the learning end condition. As a method based on the number of updates, for example, the number of updates reaching a specified number can be used as the learning end condition.
[0110] If the result of determination in step S20 is "No", the learning device 20 returns to step S1 and repeats the learning process until the learning end condition is satisfied.
[0111] If the result of the determination in step S20 is "yes", the learning device 20 ends Figure 4 and Figure 5 Flowchart shown.
[0112] The part of the text generation unit 28 that has been learned in this way is used as the text generation unit 18 of the title generation device 10. The text generation unit 28 that has been learned is an example of the "learned model" in the present invention. The module of the text generation unit 28 that has been learned can be installed as the text generation unit 18 of the title generation device 10 that is a device different from the learning device 20, and the learning device 20 can also be used directly as the title generation device 10. That is, it is also possible to use the feature extraction unit 26 and the part of the text generation unit 28 that constitute the learning device 20 as the title generation device 10.
[0113] In addition, in the present embodiment, the feature extraction unit 26 is not a learning object, but the parameters of the feature extraction unit 26 can be added to the learning object. In this case, the network parameters of the feature extraction unit 26 and the text generation unit 28 are updated through learning, and the learned feature extraction unit 26 and the text generation unit 28 are used as the feature extraction unit 16 and the text generation unit 18 of the title generation device 10.
[0114] Figure 4 and Figure 5The order of the steps shown in the flowchart can be appropriately changed within the range without technical contradictions, and sometimes multiple steps can be executed simultaneously. In addition, the update timing and update frequency of the parameters of the text generation unit 28 and the statistic inference unit 32 are not limited to Figure 4 and Figure 5 Example.
[0115] <<Example of learning model parameter update process>>
[0116] Figure 6 Schematically shows a state of processing for updating parameters of the text generating section 28 based on the first error Erl calculated by the first error calculating section 41. Figure 6 Shown in Figure 4 When the parameters of the text generating unit 28 are updated by executing steps S2 to S4 and step S9, the flow of the elements and data in the active state becomes.
[0117] Figure 7 It is a block diagram schematically showing a state of processing for updating parameters of the statistic estimation unit 32 based on the second error Er2 calculated by the second error calculation unit 42. Figure 7 Shown in Figure 4 When the parameters of the statistic estimation unit 32 are updated by executing steps D7 to S9, the flow of the elements and data in the active state becomes.
[0118] Figure 8 This is a block diagram schematically showing a state of processing for updating parameters of the text generating unit 28 and the statistic estimating unit 32 based on the third error Er3 calculated by the second error calculating unit 42 . Figure 8 Shown in Figure 5 When the parameters of the text generation unit 28 and the statistic estimation unit 32 are updated by executing steps S17 to S19, the flow of the elements and data in the active state becomes.
[0119] exist Figures 6 to 8 The arrows indicated by dashed lines in each figure indicate the flow of data for the parameter update process based on the error back propagation method. Figures 6 to 8 The boxes filled with gray in the figures represent elements that are not directly involved in the update process of the parameters of the back propagation path represented by the dotted arrows, that is, elements that are not used (hereinafter referred to as inactive elements).
[0120] In execution Figure 6In the case of the processing shown, the statistic calculation unit 34, the statistic inference unit 32, and the second error calculation unit 42 are inactive. The optimizer 46 updates the parameters of the text generation unit 28 according to the update amount of the parameters of the text generation unit 28 calculated based on the first error Er1. In addition, the optimizer 46 can also update the parameters of the feature extraction unit 26 in addition to the text generation unit 28. At this time, the optimizer 46 updates the parameters of the text generation unit 28 and the feature extraction unit 26 according to the update amount of the parameters of the text generation unit 28 and the feature extraction unit 26 respectively calculated based on the first error Er1.
[0121] In execution Figure 7 In the case of the processing shown, the optimizer 46 updates the parameters of the statistic estimation unit 32 according to the update amount of the parameters of the statistic estimation unit 32 calculated based on the second error Er2.
[0122] In execution Figure 8 In the case of the processing shown in FIG. 1 , the optimizer 46 updates the parameters of the statistic estimation unit 32 and the text generation unit 28 according to the update amount of the parameters of the statistic estimation unit 32 and the text generation unit 28 calculated based on the third error Er3. Figure 6 Similarly, the optimizer 46 may update the parameters of the feature extraction unit 26. At this time, the optimizer 46 updates the parameters of the statistic inference unit 32, the text generation unit 28, and the feature extraction unit 26 according to the update amount of the parameters of the statistic inference unit 32, the text generation unit 28, and the feature extraction unit 26 respectively calculated based on the third error Er3.
[0123] Furthermore, the optimizer 46 may not update Figure 8 The method of the parameters of the statistic inference unit 32 is shown.
[0124] The text generation unit 28 is based on Figure 6 The first error Er1 and Figure 8 Since the parameters are updated based on the third error Er3 shown, learning can be performed efficiently compared to the case where the parameters are updated based only on the first error Er1, and more accurate text generation can be achieved.
[0125] The third error Er3 can be a value for evaluating the validity or accuracy of the inferred text generated by the text generating unit 28. In the task of inferring text from an image, the degree of freedom of the text is large, and it is difficult to achieve convergence learning using a learning method using only the first error Er1.
[0126] From this perspective, in the learning method according to the first embodiment, a learning model is adopted that combines the statistic inference unit 32 and the text generation unit 28, and the inferred text generated by the text generation unit 28 is used as an input to the statistic inference unit 32, and the error between the inferred image statistic obtained from the statistic inference unit 32 and the correct image statistic is evaluated and used for learning the text generation unit 28. According to the first embodiment, by performing learning of the statistic inference unit 32 and the text generation unit 28, the text generation unit 28 acquires a function of outputting a more accurate inferred text.
[0127] In addition, there is an idea of using an image generation model that receives text input and outputs an image to replace the statistical inference unit 32, but the task of generating an image from text is a more complex and difficult task than the task of generating text from an image. Therefore, there is no rationality in combining a model that performs such a complex and difficult process with a learning model that focuses on creating a text generation model used in the process of generating text from an image.
[0128] From this perspective, the following mechanism is adopted in the first embodiment: "image statistics" which are simpler information are used instead of inferring "image" from text, and restrictions are imposed in a manner such that the inferred image statistics are close to the correct image statistics, and learning of the learning model including the text generation unit 28 is performed.
[0129] <<Effects of the First Embodiment>>
[0130] According to the learning method involved in the first embodiment, the learning of each network of the text generation unit 28 and the statistic inference unit 32 is performed simultaneously, and the inferred text T^i output from the text generation unit 28 is provided to the input of the statistic inference unit 32 for learning, thereby being able to generate an inferred text with higher accuracy.
[0131] <<Description of Problems to be Solved by Second Embodiment>>
[0132] Next, the second embodiment of the present invention is described. First, the problem to be solved is briefly described. In the previous title generation technology described in non-patent literature 1-2, the title is generated based on the learned data. The style of the automatically generated title depends on the data used in the learning, so in order to output different styles, it is necessary to prepare various models. For example, in an article created by a person, there are expressions and article styles that are unique to the creator, and the characteristics of these unique expressions are manifested as the unique style of the person.
[0133] In order to automatically generate a title close to the style of each person, in the conventional technology, for example, a separate model is prepared for each person, and the separate model is learned for each person using learning data. However, it is difficult to collect a large amount of learning data for each person, which is unrealistic.
[0134] Non-Patent Document 3 proposes a technique for adding information different from an image to one model and outputting various text styles, but does not describe a specific method for reproducing a text style unique to an individual.
[0135] In view of such circumstances, a second embodiment of the present invention provides a model learning method and device, a program, a learned model, and a text generation device that can change the style according to the user's attributes and generate a text close to the user's intention.
[0136] <<Overview of Title Generating Device According to Second Embodiment>>
[0137] Fig. 9 2 is a functional block diagram showing an example of the title generating device 110 according to the second embodiment. Fig. 9 Middle pair and Figure 1 The same or similar elements are marked with the same symbols. Figure 1 The differences are explained below. Fig. 9 The title generating device 110 shown receives an image IMG and metadata MTD including attribute information of a user as input, generates and outputs an inferred text PTX of a style close to that of the user based on these inputs.
[0138] The "user" mentioned here refers to a person who wants to add a title to the image IMG. The user can be, for example, at least one of the photographer, creator, publisher, provider and owner of the image IMG. The user's attribute information includes, for example, at least one of the user's gender, age, occupation, interests, hobbies and social relationships, and preferably includes a combination of multiple information. Part or all of the metadata MTD can be obtained from the user's registration information, etc. For example, in the case of a user of a system using a social networking service (SNS), part or all of the metadata MTD can be read from the registration information of the account. In addition, it is also possible to automatically obtain information such as the user's interests and hobbies by analyzing past posted content on the SNS.
[0139] The inferred text PTX is a title corresponding to the content of the image IMG, and is character information inferred to be a natural language-based expression close to the user's intention determined by the metadata MTD. In other words, "user intention" may be "user's preference", "user's interest, hobby" or "user's style". That is, the inferred text PTX is information inferred to be a language expression close to the expression (style) created by the user determined by the metadata MTD.
[0140] The title generating device 110 includes an image acquiring unit 12, a metadata acquiring unit 14, a feature extracting unit 16, and a text generating unit 18. The title generating device 110 can be implemented by a computer system including one or more computers. The function of the metadata acquiring unit 14 can be implemented by a combination of computer hardware and software.
[0141] The metadata acquisition unit 14 is an interface for taking in data of the metadata MTD. The metadata acquisition unit 14 may be composed of a data input terminal for taking in metadata MTD from the outside or other signal processing units in the device. A common interface that functions as the image acquisition unit 12 and the metadata acquisition unit 14 may be used.
[0142] The metadata MTD input via the metadata acquisition section 14 is sent to the text generation section 18 .
[0143] The text generation unit 18 generates an inferred text PTX based on the provided image feature quantity and metadata MTD associated with the image. The text generation unit 18 acquires a function of generating a text of a style close to the user's unique style through machine learning using a learning method described later. The title generation device 110 is an example of a "text generation device" in the present invention.
[0144] <<Configuration Example 2 of Learning Device>>
[0145] Next, a learning method for improving the inference function of the text generating unit 18 will be described.
[0146] Fig.10 2 is a functional block diagram showing a configuration example of a learning device 120 according to the second embodiment. Fig.10 Middle pair and Figure 3 The same or similar elements are marked with the same symbols. Figure 3 The differences are explained below.
[0147] The learning device 120 is connected to the learning data storage unit 60. The learning data storage unit 60 includes a storage device that stores the learning data LDi required for the learning device 20 to perform machine learning in advance. In the machine learning of the second embodiment, a lot of learning data LDi = (Ii, Mi, Ti) that is a combination of an image Ii, metadata Mi representing the attributes of the user, and text Ti describing the image Ii is used. For example, a set of n+1 sets of learning data from i=0 to i=n is labeled as follows.
[0148] {(I0, M0, T0), (I1, M1, T1), ...(Ii, Mi, Ti), ...(In, Mn, Tn)}
[0149] Such a learning data set can be generated by combining, for example, images posted in the past by users using SNS (Social Networking Service), user information of the users (posters), and titles input by the users for the images.
[0150] The learning device 120 reads the learning data LDi from the learning data storage unit 60 and performs machine learning. Fig.10 In order to simplify the explanation, the process of processing one set of learning data LDi=(Ii, Mi, Ti) is shown, but when performing small batch learning, multiple sets (for example, m sets) of learning data included in the small batch are processed together.
[0151] The learning device 120 includes a data acquisition unit 22, a feature extraction unit 26, a text generation unit 28, a metadata inference unit 30, a statistic inference unit 32, a statistic calculation unit 34, a first error calculation unit 41, a second error calculation unit 42, a third error calculation unit 43 and an optimizer 46.
[0152] The data acquisition unit 22 may be Fig. 9 The image acquisition unit 12 and the metadata acquisition unit 14 have the same structure.
[0153] The text generation unit 28 includes Fig. 9 The text generation unit 28 is configured with the same hierarchical neural network learning model as the text generation unit 18. The text generation unit 28 generates an inferred text T^i based on the provided image feature quantity Fi and metadata Mi. The inferred text T^i output from the text generation unit 28 is sent to the first error calculation unit 41. In addition, the inferred text T^i output from the text generation unit 28 is sent to the metadata inference unit 30 and the statistic inference unit 32.
[0154] The metadata inference unit 30 infers the user's metadata corresponding to the text based on the provided text and outputs the inferred metadata. That is, if the text Ti input via the data acquisition unit 22 is provided, the metadata inference unit 30 generates the inferred metadata M^i based on the text Ti. The inferred metadata M^i output from the metadata inference unit 30 is sent to the third error calculation unit 43.
[0155] The third error calculation unit 43 calculates the error between the inferred metadata Mi generated by the metadata inference unit 30 and the metadata Mi as the teacher data. The third error calculation unit 43 evaluates the error using a loss function. The error between the inferred metadata Mi and the metadata Mi as the correct answer is referred to as a fourth error Er4. The fourth error Er4 calculated by the third error calculation unit 43 is sent to the optimizer 46.
[0156] The optimizer 46 performs the following processing: based on the calculation results of the first error calculation unit 41, the second error calculation unit 42, and the third error calculation unit 43, the network parameters of the text generation unit 28, the metadata inference unit 30, and the statistic inference unit 32 are updated.
[0157] The optimizer 46 uses the calculation results of the errors obtained from the first error calculation unit 41, the second error calculation unit 42 and the third error calculation unit 43 to perform parameter operation processing for calculating the update amount of the network parameters of the text generation unit 28, the metadata inference unit 30 and the statistic inference unit 32 respectively, and parameter update processing for updating the network parameters of the text generation unit 28, the metadata inference unit 30 and the statistic inference unit 32 respectively according to the calculated parameter update amount.
[0158] Then, in the learning device 120, the inferred text T^i generated by the text generation unit 28 is output to the metadata inference unit 30 and the statistic inference unit 32, respectively, and the same learning as above is performed. That is, the inferred text T^i generated by the text generation unit 28 is input to the metadata inference unit 30 and the statistic inference unit 32.
[0159] The metadata inference unit 30 generates the user's inferred metadata M^^i corresponding to the inferred text T^i based on the provided inferred text T^i. The inferred metadata M^^i outputted from the metadata inference unit 30 is sent to the third error calculation unit 43.
[0160] The third error calculation unit 43 calculates the error between the estimated metadata M^^i generated by the metadata inference unit 30 and the metadata Mi as the teacher data. The error between the estimated metadata M^^i and the metadata Mi as the correct answer is referred to as the fifth error Er5. The fifth error Er5 calculated by the third error calculation unit 43 is sent to the optimizer 46.
[0161] The optimizer 46 updates the network parameters of the metadata estimation unit 30 using the fourth error Er4 obtained from the third error calculation unit 43. Furthermore, the optimizer 46 updates the network parameters of the metadata estimation unit 30 and the text generation unit 28 using the fifth error Er5 obtained from the third error calculation unit 43.
[0162] The optimizer 46 only needs to calculate at least the update amount of the parameters of the text generating unit 28 based on the fifth error Er5 and update the parameters of the text generating unit 28. Alternatively, the parameters of the metadata inference unit 30 may not be updated based on the fifth error Er5.
[0163] The timing for updating the parameters of the text generation unit 28, the metadata inference unit 30, and the statistic inference unit 32 is not particularly limited. The updating timing of the parameters of the text generation unit 28, the updating timing of the parameters of the metadata inference unit 30, and the updating timing of the parameters of the statistic inference unit 32 may be different timings or simultaneous timings. Furthermore, the timing for updating the parameters of the text generation unit 28 based on the parameter update amount of the text generation unit 28 calculated based on the first error Er1, the timing for updating the parameters of the text generation unit 28 based on the parameter update amount of the text generation unit 28 calculated based on the third error Er3, and the timing for updating the parameters of the text generation unit 28 based on the parameter update amount of the text generation unit 28 calculated based on the fifth error Er5 may be different timings, respectively, or some or all of them may be simultaneous timings. For example, the optimizer 46 may combine the parameter update amount of the text generation unit 28 calculated based on the first error, the parameter update amount of the text generation unit 28 calculated based on the third error, and the parameter update amount of the text generation unit 28 calculated based on the fifth error, and implement the parameter update processing of the text generation unit 28 based on the combined parameter update amount.
[0164] Fig.11 This is a block diagram schematically showing data input to each module of the feature extraction unit 26, the text generation unit 28, the metadata inference unit 30, the statistic inference unit 32, and the statistic calculation unit 34 of the learning device 120, and data output from each module. Fig.11 Middle pair and Figure 3 The same components as shown are denoted by the same reference numerals, and their descriptions are omitted. Fig.11 , for Figure 3 The differences are explained below.
[0165] Optionally input text Ti or inferred text Ti into Fig.11 The metadata inference unit 30 of the learning device 120 shown in FIG. The metadata inference unit 30 receives the input of the text Ti and outputs the inferred metadata Mi. The fourth error Er4 is calculated based on the inferred metadata Mi outputted from the metadata inference unit 30 and the correct metadata Mi.
[0166] Then, the metadata inference unit 30 receives the input of the inferred text Ti and outputs the inferred metadata Mi. The fifth error Er5 is calculated based on the inferred metadata Mi outputted from the metadata inference unit 30 and the correct metadata Mi.
[0167] The metadata Mi is an example of the "first metadata" in the present invention. The user corresponding to the metadata Mi is an example of the "first user" in the present invention. The combination of the feature extraction unit 26, the text generation unit 28, the metadata inference unit 30, and the statistic inference unit 32 is an example of the "learning model" in the present invention. The inferred metadata Mi is an example of the "first inferred metadata" in the present invention. The inferred metadata Mi inferred from the inferred text Ti is an example of the "second inferred metadata" in the present invention.
[0168] <<Example of a learning method using the learning device 120>>
[0169] Fig.12 and Fig.13 FIG. 1 is a flowchart exemplarily showing the steps of the learning method according to the second embodiment. Fig.12 and Fig.13 In, with Figure 3 and Figure 4 The same or similar steps in the flowcharts shown are marked with the same step numbers. Fig.12 and Fig.13 , for Figure 3 and Figure 4 The differences are explained below.
[0170] Fig.12 The flowchart includes step S1A, step S3A and step S9A instead of Figure 3 Step S1, step D3 and step D9. And, Fig.12 In the flowchart of Figure 3 Steps D5 and D6 are included between step S4 and step S7. Fig.13 In the flowchart of Figure 4 The step S17 shown includes steps S15 and S16. Fig.13 The flowchart includes step S19A instead of Figure 4 Step S19.
[0171] exist Fig.12 In step S1A, a plurality of sets of combinations of images Ii, metadata Mi and text Ti are prepared.
[0172] In step S3A, the learning device 20 inputs the image feature quantity Fi and the metadata Mi to the text generating unit 28 to obtain the inferred text Ti.
[0173] In step S5, the learning device 20 inputs the text Ti to the metadata inference unit 30 to obtain the inferred metadata Mi.
[0174] In step S6, the third error calculation unit 43 calculates an error (fourth error) between the estimated metadata Mi and the metadata Mi.
[0175] In step S9A, the optimizer 46 updates the parameters of the text generating section 28 , the metadata inferring section 30 , and the statistic inferring section 32 based on the errors calculated in steps S4 , S6 , and S8 .
[0176] After step S9A, the same processing is performed using the inferred text T^i inferred by the text generating unit 28 as the input of each of the above steps S5 to S9A.
[0177] That is, in Fig.13 In step S15, the learning device 120 inputs the inferred text T^i into the metadata inference unit 30 to obtain the inferred metadata M^^i.
[0178] In step S16, the third error calculation unit 43 calculates the error (fifth error Er5) between the estimated metadata Mi and the metadata Mi.
[0179] In step S19A, the optimizer 46 updates the parameters of the text generation unit 28, the metadata inference unit 30, and the statistic inference unit 32 based on the errors calculated in steps S4, S16, and S18. In addition, the optimizer 46 only needs to use the fifth error calculated in step S16 and the third error Er3 calculated in step S18 to update at least the parameters of the text generation unit 28. For example, in step S19A, the optimizer 46 may fix the parameters of either or both of the metadata inference unit 30 and the statistic inference unit 32 and exclude them from the update objects, and update the parameters of the text generation unit 28. Furthermore, in step S19A, the optimizer 46 may update the parameters of the text generation unit 28 based on the fifth error Er5 and the third error Er3 instead of using the first error Er1 calculated in step S4. Other processing and Figure 3 and Figure 4 The flowchart shown is the same.
[0180] The portion of the text generating unit 28 that has been learned in this way is used as the text generating unit 18 of the title generating device 110. In addition, the feature amount extracting unit 26 may be included in the learning object, which is the same as the first embodiment.
[0181] Fig.12 and Fig.13The order of the steps shown in the flowchart can be appropriately changed within the range without technical contradictions, and sometimes a plurality of steps can be executed simultaneously. In addition, the update timing and update frequency of the parameters of the text generation unit 28, the metadata inference unit 30, and the statistic inference unit 32 are not limited to Fig.12 and Fig.13 Example.
[0182] <<Example of updating the parameters in the learning model>>
[0183] Fig.14 This is a block diagram schematically showing a state of processing for updating parameters of the text generating unit 28 based on the first error calculated by the first error calculating unit 41 . Fig.14 Shown in Fig.12 The flow of the elements and data in the active state is shown when the parameters of the text generating unit 28 are updated by executing steps S2 to S4 and step S9A. The parameters of the feature amount extracting unit 26 may or may not be included in the update processing target based on learning.
[0184] Fig.15 It is a block diagram schematically showing a state of processing for updating parameters of the metadata estimation unit 30 based on the fourth error calculated by the third error calculation unit 43. Fig.15 Shown in Fig.12 When the parameters of the metadata inference unit 30 are updated by executing steps S5, S6, and S9A, the flow of the requirements and data in the activated state becomes.
[0185] Fig.16 It is a block diagram schematically showing a state of processing for updating parameters of the statistic estimation unit 32 based on the second error calculated by the second error calculation unit 42 . Fig.16 Shown in Fig.12 When the parameters of the metadata inference unit 30 are updated by executing steps S7, S8, and S9A, the flow of the requirements and data in the activated state is obtained.
[0186] Fig.17 It is a block diagram schematically showing a state of processing for updating parameters of each of the metadata estimation unit 30 and the text generation unit 28 based on the fifth error calculated by the third error calculation unit 43 . Fig.17 Shown in Fig.13 When the parameters of the metadata estimation unit 30 and the text generation unit 28 are updated by executing steps S15, S16 and S19A, the flow of the requirements and data in the activated state is obtained.
[0187] Fig.18This is a block diagram schematically showing a state of processing for updating parameters of each of the metadata estimation unit 30 and the text generation unit 28 based on the third error calculated by the second error calculation unit 42 . Fig.18 Shown in Fig.13 When the parameters of the statistic estimation unit 32 and the text generation unit 28 are updated by executing steps S17, S18, and S19A, the flow of the elements and data in the activated state becomes. Figures 14 to 18 The display rules of the various figures are Figures 6 to 8 The display rules are the same.
[0188] In execution Fig.14 In the case of the processing shown, the metadata inference unit 30, the third error calculation unit 43, the statistic inference unit 32, the statistic calculation unit 34, and the second error calculation unit 42 are inactive. The optimizer 46 updates the parameters of the text generation unit 28 according to the update amount of the parameters of the text generation unit 28 calculated based on the first error Er1.
[0189] In execution Fig.15 In the case of the processing shown, the optimizer 46 updates the parameters of the metadata estimation unit 30 according to the update amount of the parameters of the metadata estimation unit 30 calculated based on the fourth error Er4.
[0190] In execution Fig.16 In the case of the processing shown, the optimizer 46 updates the parameters of the statistic estimation unit 32 according to the update amount of the parameters of the statistic estimation unit 32 calculated based on the second error Er2.
[0191] In execution Fig.17 In the case of the processing shown in FIG. 1 , the optimizer 46 updates the parameters of the metadata inference unit 30 and the text generation unit 28 according to the update amount of the parameters of the metadata inference unit 30 and the text generation unit 28 calculated based on the fifth error Er5. Fig.14 Similarly, the optimizer 46 may update the parameters of the feature quantity extraction unit 26. Fig.17 In the illustrated metadata inference unit 30 and the text generation unit 28 , the metadata inference unit 30 updates the parameters, whereas only the text generation unit 28 updates the parameters.
[0192] In execution Fig.18 In the case of the processing shown in FIG. 1 , the optimizer 46 updates the parameters of the statistic estimation unit 32 and the text generation unit 28 according to the update amount of the parameters of the statistic estimation unit 32 and the text generation unit 28 calculated based on the third error Er3. Fig.14 Likewise, the optimizer 46 may update the parameters of the feature quantity extraction unit 26 .
[0193] The text generation unit 28 is based on Fig.14 The first error Er1 shown, Fig.17 The fifth error Er5 and Fig.18 The parameters are updated based on the third error Er3 shown.
[0194] <<Effects of the Second Embodiment>>
[0195] According to the second embodiment, as learning data, in addition to the image Ii and the text Ti, a combination of metadata Mi including the attribute information of the user is used, so that a model that can achieve high-precision text generation reflecting the attributes of the user can be effectively learned. In addition, in the learning method involved in the second embodiment, the learning of each network of the text generation unit 28, the metadata inference unit 30, and the statistic inference unit 32 is performed simultaneously, and the inferred text T^i output from the text generation unit 28 is provided to the input of the metadata inference unit 30 and the statistic inference unit 32 for learning, so that a more accurate inferred text can be generated. According to the second embodiment, a title close to the user's intention can be generated.
[0196] Furthermore, according to the second embodiment, by performing learning using a learning data set including learning data of different users, it is possible to obtain a general-purpose model capable of automatically generating titles corresponding to users of various attributes.
[0197] <<Application Example 1>>
[0198] The technology of the present invention can be used as a service of a title generation function as one of the posting support functions attached to SNS, for example. This title generation function can be realized in the next step.
[0199] [Step 1-1] Upload the image for which the user wants to generate a title to the system. The "system" referred to here is a computer system that provides processing functions for title generation. Hereinafter, the system will be referred to as the "title generation system". The title generation system can be built as an integral part of the SNS system or as an independent system.
[0200] [Step 1-2] The title generation system analyzes the user's past input characters and images based on the user's past published content, etc. The "analysis" mentioned here refers to combining the user's metadata, input characters (text) and images to generate multiple learning data, and performing machine learning using the learning device 20. In addition, the order of the above [Step 1-1] and [Step 1-2] can be reversed.
[0201] [Step 1-3] The target image (uploaded image) is input to the title generating device 10 including the text generating unit 28 that has completed the learning obtained by the analysis (learning) in step 1-2, and a title is generated. The generated title is provided to the user.
[0202] <System Configuration Example of Application Example 1>
[0203] Fig.19 1 is a block diagram showing an example of the overall structure of a computer system that implements the title generation function involved in Application Example 1. The computer system 70 includes a data storage unit 61, a title generation system 72, and a learning device 120. The data storage unit 61, the title generation system 72, and the learning device 120 are respectively connected to a telecommunication line 90 and can exchange data with each other. The telecommunication line 90 can be a wide area network such as the Internet. The data storage unit 61, the title generation system 72, and the learning device 120 are respectively equipped with a communication unit (not shown) for connecting to the telecommunication line 90.
[0204] The data storage unit 61 includes a large-capacity storage that stores user information of each user, past posting data, and the like.
[0205] The title generation system 72 includes an authentication unit 74 for performing user authentication, a learning data generation unit 76 for generating learning data, a learning data storage unit 60, and a title generation device 110. The authentication unit 74 may be provided outside the title generation system 72.
[0206] The user can access the title generation system 72 via the telecommunication line 90 using an information terminal device 80. The information terminal device 80 may be a portable communication terminal such as a smartphone or a personal computer. The information terminal device 80 includes a communication unit (not shown) for connecting to the telecommunication line 90.
[0207] The information terminal device 80 includes an operation unit 84 and a display unit 86 as a user interface. The operation unit 84 is composed of, for example, a keyboard, a mouse, a touch panel or other pointing device or an audio input device, or a suitable combination thereof. The display unit 86 is composed of, for example, a liquid crystal display, an organic EL (organic electro-luminescence: OEL) display or a projector, or a suitable combination thereof.
[0208] The information terminal device 80 may include a camera 88. The camera 88 includes a photographic optical system including a lens (not shown) and an imaging device. The user can upload images captured by the camera 88 and / or images stored in an internal or external memory (not shown) of the information terminal device 80 to the title generation system 72.
[0209] The authentication unit 74 receives the user authentication information sent from the information terminal device 80 and performs user authentication. The authentication unit 74 checks the registration information of the user registered in the data storage unit 61 to determine the user who has accepted the access request. The authentication unit 74 sends the user information related to the determined user to the learning data generation unit 76 and the title generation device 110. The user information sent from the authentication unit 74 to the learning data generation unit 76 and the title generation device 110 includes the user's metadata read from the data storage unit 61. Alternatively, the learning data generation unit 76 and the title generation device 110 can obtain the user's metadata from the data storage unit 61 based on the information that determines the user.
[0210] Fig. 20 is a conceptual diagram of data stored in the data storage unit 61. For each user, user information U(x) is associated with past published data P(x)_j and stored in the data storage unit 61. x is an index to identify the user, and j is an index to identify the published data. The published data P(x)_j is composed of a combination of an image I(x)_j and a text T(x)_j. The user information U(x) includes the user's metadata. Fig. 20 As shown, past posted data P(a)_j posted by user a is stored in association with user information U(a). Similarly, past posted data P(b)_j posted by user b is stored in association with user information U(b).
[0211] The learning data generating unit 76 generates the learning data LDi based on the data stored in the data storage unit 61. Fig.10 As shown, the learning data LDi is a combination of images, metadata, and text. The learning data LDi set generated by the learning data generation unit 76 is stored in the learning data storage unit 60. In addition, the learning data storage unit 60 can be integrated into the data storage unit 61.
[0212] The learning device 120 reads the learning data LDi from the learning data storage unit 60 and performs learning. The learning device 20 sends the learned parameters obtained as the learning result to the title generation device 110, and updates the text generation unit 18 of the title generation device 110 with the learned parameters. Thus, a learned model (text generation unit 18) capable of generating a title close to the user's intention can be installed in the title generation device 110.
[0213] The title generating device 110 generates a title close to the user's intention for the image uploaded by the user and provides it to the user. The title generated by the title generating device 110 is transmitted to the information terminal device 80 via the telecommunication line 90 and displayed on the display unit 86.
[0214] The user can check the title automatically generated by the title generating device 110 and use it for publishing as it is, and can also edit it as needed.
[0215] According to this system, it is possible to reduce the text input work performed by the user when adding a title to an image.
[0216] Furthermore, for a specific user, the learned parameters obtained by executing learning are associated with the user information and stored in the data storage unit 61, so that the learned parameters can be applied to the title generation device 110 at the next visit and the learning process can be omitted. Alternatively, at the next visit, the learned parameters are set as the initial parameters for learning in the learning device 120, and additional learning is performed, thereby improving the accuracy of inference.
[0217] <Modification>
[0218] Fig.19 The learning device 120 can be incorporated into the title generation system 72. The learning data generation unit 76 can be an independent device structure as a learning data generation device. The learned parameters incorporated into the title generation device 10 can be provided to the information terminal device 80, and the title generation device 110 can be constructed in the information terminal device 80.
[0219] <<Application Example 2>>
[0220] The technology of the present invention can be used as a service of an automatic email reply function, for example. The automatic email reply function can be implemented in the next step.
[0221] [Step 2-1] The automatic mail reply system acquires the user's mail as an image. The "user's mail" referred to here is the mail sent to the target user who uses this service and is received by the target user. The user transmits (uploads) the received mail to the automatic mail reply system. The automatic mail reply system acquires the mail image by converting the uploaded mail into image information.
[0222] [Step 2-2] The automatic mail reply system analyzes the user's past input characters and mail images based on the contents of mails sent by the user in the past, etc. In addition, the user's metadata can be extracted during this analysis process.
[0223] [Step 2-3] The target image (here, the image acquired in step 2-1) is input to the title generating device 110 including the learned text generating unit 28 obtained by the analysis (learning) in step 2-2 to generate a title, that is, a reply candidate article. Then, the generated title is provided to the user.
[0224] The system configuration of Application Example 2 is basically the same as that of Application Example 1, and thus description thereof will be omitted. According to Application Example 2, reply candidates in a style close to the user's intention can be automatically generated, thereby reducing the work of creating a reply mail.
[0225] <<Other Application Examples>>
[0226] The technology of the present invention can be applied to various purposes of creating articles for images. For example, the technology of the present invention can be applied to creating a reading report in the medical field or creating a form for infrastructure inspection. The user when creating a reading report is the reader, and the user when creating a form for infrastructure inspection is the staff who performs the inspection work.
[0227] The image to which the title is attached is not limited to a still image, but may be an animation, and the image may be a frame image constituting a part of the animation.
[0228] <<About distillation model and derived models>>
[0229] A derived model and / or a distilled model can be generated based on a learned model obtained by implementing the learning method of the present invention. A derived model is a derived learned model obtained by performing further additional learning on a learned model, and is also called a "reused model". "Additional learning" here means generating new learned parameters by performing further learning by applying different learning data sets to an existing learned model. Additional learning is performed, for example, for the purpose of maintaining or improving the accuracy of the learned model or adapting to a field different from the field initially learned.
[0230] On the other hand, "distillation" refers to the process of performing machine learning using the input to an existing learned model and the output results relative to the input as a learning data set for a new model, thereby generating a new learned model and / or learned parameters. A "distilled model" refers to an inference program (inference model) that incorporates new learned parameters generated by distillation. An "inference program" refers to a program that can output a constant result relative to an input by applying the incorporated learned parameters. A distilled model can be a network structure different from the original learned model. Typically, a distilled model can be a network structure smaller in scale than the original learned model.
[0231] A derivative model and / or a distilled model can be generated based on the learned model obtained by implementing the learning method of the present invention. Therefore, it can be understood that the method of generating these derivative models and / or distilled models, and the derived models and / or distilled models obtained belong to the technical scope of the present invention.
[0232] <<Example of Computer Hardware Configuration>>
[0233] Fig.21 800 is a block diagram showing an example of a hardware structure of a computer. The computer 800 may be a personal computer, a workstation, or a server computer. The computer 800 may be used as any one of the title generation device 10, the learning device 20, the data storage unit 61, the title generation system 72, and the information terminal device 80 described above, or as a device having a plurality of these functions.
[0234] The computer 800 includes a CPU (Central Processing Unit) 802, a RAM (Random Access Memory) 804, a ROM (Read Only Memory) 806, a GPU (Graphics Processing Unit) 808, a storage 810, a communication unit 812, an input device 814, a display device 816, and a bus 818. The GPU (Graphics Processing Unit) 808 can be provided as needed, and can be omitted if the computational burden is not large.
[0235] The CPU 802 reads various programs stored in the ROM 806 or the storage 810, and executes various processes. The RAM 804 can be used as a work area of the CPU 802. In addition, the RAM 804 can be used as a storage unit for temporarily storing the read programs and various data.
[0236] The storage 810 is composed of, for example, a hard disk device, an optical disk, a magneto-optical disk, or a semiconductor memory, or a storage device composed of an appropriate combination thereof. Various programs and data required for image processing such as learning processing and / or image generation processing are stored in the storage 810. The program stored in the storage 810 is loaded into the RAM 804, and the CPU 802 executes the program, whereby the computer functions as a mechanism for performing various processing specified in the program.
[0237] The communication unit 812 is an interface that performs communication processing with an external device via wire or wirelessly and exchanges information with the external device.
[0238] The input device 814 is an input interface for receiving various operation inputs to the computer 800. The input device 814 may be, for example, a keyboard, a mouse, a touch panel or other pointing device or an audio input device, or a suitable combination thereof.
[0239] The display device 816 is an output interface for displaying various information and may be, for example, a liquid crystal display, an organic EL (organic electro-luminescence: OEL) display, a projector, or a suitable combination thereof.
[0240] <<About running computer programs>>
[0241] A program that enables a computer to implement part or all of at least one of the processing functions including the learning function, title generation function, and learning data generation function described in the above-mentioned embodiments is recorded on a CD, a disk, a semiconductor memory, or other tangible non-temporary information storage medium, i.e., a computer-readable medium, and the program is provided through the information storage medium.
[0242] Furthermore, instead of providing the program by storing it in such a tangible non-transitory information storage medium, it is also possible to provide a program signal as a download service using telecommunication lines such as the Internet.
[0243] Furthermore, a service can be provided by providing a part or all of at least one of the learning function, title generation function, and learning data generation function described in the above embodiments as an application server and providing the processing function via a telecommunication line.
[0244] <<About the hardware structure of each processing unit>>
[0245] Figure 1 and Fig. 9 The image acquisition unit 12, the metadata acquisition unit 14, the feature extraction unit 16, the text generation unit 18, Figure 2 and Fig.10 The learning data storage unit 60, the data acquisition unit 22, the feature extraction unit 26, the text generation unit 28, the metadata inference unit 30, the statistic inference unit 32, the statistic calculation unit 34, the first error calculation unit 41, the second error calculation unit 42, the third error calculation unit 43, the optimizer 46, Fig.19 The hardware configuration of the processing unit (processing unit) that executes various processes, such as the data storage unit 61, the authentication unit 74, and the learning data generation unit 76, is, for example, the various processors shown below.
[0246] Various processors include general-purpose processors such as CPUs that execute programs and function as various processing units, processors suitable for image processing such as GPUs, FPGAs (Field Programmable Gate Arrays) whose circuit structures can be changed after manufacturing, programmable logic devices (PLDs), and ASICs (Application Specific Integrated Circuits) that are processors with circuit structures specially designed to perform specific processing, such as dedicated circuits.
[0247] One processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same type or different types. For example, one processing unit may be composed of multiple FPGAs, or a combination of a CPU and an FPGA, or a combination of a CPU and a GPU. Furthermore, multiple processing units may be composed of one processor. As an example of multiple processing units composed of one processor, first, there is a method in which a processor is composed of a combination of one or more CPUs and software, as represented by computers such as clients and servers, and the processor functions as multiple processing units. Secondly, there is a method in which a processor that implements the functions of the entire system including multiple processing units is used by one IC (Integrated Circuit) chip, as represented by a system on chip (SoC). In this way, as a hardware structure, various processing units are composed of one or more of the above-mentioned various processors.
[0248] Furthermore, more specifically, the hardware structure of these various processors is a circuit (circuitry) formed by combining circuit elements such as semiconductor elements.
[0249] <<Others>>
[0250] The configurations described in the above-mentioned embodiments and the matters described in the modified examples can be used in combination as appropriate, and some of the matters can also be replaced.
[0251] The present invention is not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present invention.
[0252] Explanation of symbols
[0253] 10-title generating device, 12-image acquiring unit, 14-metadata acquiring unit, 16-feature extraction unit, 18-text generating unit, 20-learning device, 22-data acquiring unit, 26-feature extraction unit, 28-text generating unit, 30-metadata inference unit, 32-statistic inference unit, 34-statistic calculation unit, 41-first error calculation unit, 42-second error calculation unit, 43-third error calculation unit, 46-optimizer, 60-learning data storage unit, 61-data storage unit, 70-computer system, 72-title generating system, 74-authentication unit, 76-learning data generating unit, 80-information terminal device, 84-operating unit, 86-display unit, 88-camera, 90-telecommunication line, 110-title generating device, 120-learning device, 800-computer, 802-CPU, 804-R AM, 806-ROM, 808-GPU, 810-storage, 812-communication unit, 814-input device, 816-display device, 818-bus, Er1-first error, Er2-second error, Er3-third error, Er4-fourth error, Er5-fifth error, I-image, Ii-image, Fi-image feature, IMG-image, ISi-image statistics, IS^i-inferred image statistics, IS^^i-inferred image statistics, LDi-learning data, MTD-metadata, Mi-metadata, M^i-inferred metadata, M^^i-inferred metadata, Ti-text, PTX-inferred text, T^i-inferred text, U(x)-user information, P(x)_j-published data, S1~S9, S1A, S9A, S15~S20, S19A-learning method steps.
Claims
1. A learning method comprising the following steps: Using a learning model, the learning model includes: A feature extraction unit extracts image features from an image; a text generation unit generates an inferred text related to the image based on the image features; and a statistics inference unit, which receives an input of the inferred text or the correct text, infers statistics of an image corresponding to the input text, and generates inferred image statistics; receiving an input of a plurality of learning data formed by combining a first image for learning and a first text which is the correct answer text for the first image; Inputting the first image into the feature extraction unit to obtain the first image feature; inputting the first image feature into the text generating unit to obtain a first inferred text; Calculating a first error between the first inferred text and the first text; Inputting the first text into the statistic inference unit to obtain a first inferred image statistic; calculating a second error between the first inferred image statistic and a first image statistic calculated based on the first image; inputting the first inferred text into the statistic inference unit to obtain a second inferred image statistic; Calculating a third error between the second inferred image statistic and the first image statistic; updating the parameters of the statistic inference unit according to the second error; and At least a parameter of the text generating unit is updated based on the first error and the third error.
2. The learning method according to claim 1, wherein: The text generation unit is a module that generates the inferred text based on the image feature and metadata about the user associated with the image. The learning model further includes a metadata inference unit that receives an input of the inferred text or the first text, infers metadata of a user corresponding to the input text, and generates inferred metadata. The learning data further includes first metadata related to a first user associated with the first image. The first inferred text is acquired by inputting the first image feature and the first metadata into the text generating unit, The learning method further comprises the following steps: inputting the first text into the metadata inference unit to obtain first inferred metadata; calculating, for the first inferred metadata, a fourth error with respect to the first metadata; updating a parameter of the metadata inference unit according to the fourth error; inputting the first inferred text into the metadata inference unit to obtain second inferred metadata; and For the second inferred metadata, a fifth error is calculated with respect to the first metadata, At least parameters of the text generating unit are updated based on the fifth error.
3. The learning method according to claim 2, wherein: The first user corresponds to at least one of a photographer, a creator, a publisher, a provider, and an owner of the image. The first metadata includes information indicating attributes of the first user.
4. The learning method according to claim 2 or 3, wherein: The first text includes an article created by the first user.
5. The learning method according to claim 2 or 3, wherein: The first image is an image posted by the first user to a social networking service, The first text is an article posted by the first user to the social network service, The first metadata includes at least a portion of user information of the first user registered in the social network service.
6. The learning method according to claim 2 or 3, wherein: The first image is an image obtained by converting the mail received by the first user into image information. The first document includes a text of an email sent by the first user.
7. The learning method according to any one of claims 1 to 3, wherein: The learning method further includes the step of updating a parameter of the feature quantity extraction unit based on the first error.
8. The learning method according to any one of claims 1 to 3, wherein: The learning model is constructed using a hierarchical neural network. 9 . A non-transitory computer-readable recording medium having recorded thereon a program for causing a computer to execute the learning method according to claim 1 .
10. A computer program product, comprising a computer program, wherein when the computer program is executed by a computer, the computer program implements the processing of the learning method according to any one of claims 1 to 8, thereby generating the learned text generation unit as a learned model.
11. A text generation device, comprising: The text generation unit that has completed learning by using the learning method according to any one of claims 1 to 8; and The feature extraction unit.
12. A learning device comprising: A learning data acquisition unit receives an input of a plurality of learning data consisting of a first image for learning and a first text that is a correct answer text for the first image; A learning model is trained by the learning data, and the learning model includes: A feature extraction unit extracts image features based on the input image; a text generating unit that generates an inferred text related to the image based on the image feature; and a statistic inference unit that receives an input of the inferred text or the first text, infers statistics of an image corresponding to the input text, and generates inferred image statistics; a first error calculation unit that calculates a first error between a first inferred text obtained from the text generation unit by inputting a first image feature amount into the text generation unit and the first text, wherein the first image feature amount is obtained from the feature amount extraction unit by inputting the first image into the feature amount extraction unit; a second error calculation unit that calculates a second error between a first inferred image statistic obtained from the statistic inference unit by inputting the first text into the statistic inference unit and a first image statistic calculated based on the first image, and calculates a third error between a second inferred image statistic obtained from the statistic inference unit by inputting the first inferred text into the statistic inference unit and the first image statistic; and The parameter updating unit updates the parameters of the text generating unit and the statistic inferring unit based on the first error and the second error, and updates at least the parameter of the text generating unit based on the third error.
13. The learning device according to claim 12, wherein: The text generation unit is a module that generates the inferred text based on the image feature and metadata about the user associated with the image. The learning model further includes a metadata inference unit that receives an input of the inferred text or the first text, infers metadata of a user corresponding to the input text, and generates inferred metadata. The learning data further includes first metadata related to a first user associated with the first image. The first inferred text is acquired by inputting the first image feature and the first metadata into the text generating unit, The learning device also has: a third error calculation unit that calculates a fourth error between first inferred metadata obtained from the metadata inference unit by inputting the first text into the metadata inference unit and the first metadata, and calculates a fifth error between second inferred metadata obtained from the metadata inference unit by inputting the first inferred text into the metadata inference unit and the first image statistic, The parameter updating unit updates the parameters of the metadata inferring unit according to the fourth error, and updates at least the parameters of the text generating unit according to the fifth error.
