Image processing device, method and program

Abstract

An image processing device according to an embodiment includes an input unit that accepts input of image data, name information indicating a name of a situation in the image data, and semantic role information indicating a semantic role of each word in a caption describing the situation in the image data, and an output unit that outputs a caption describing the situation in the image data on the basis of the image data, the name information, and the semantic role information, that are input by the input unit, and also outputs information indicating a position relating to each word in the caption describing the situation in a display region of the image data, and the semantic role of each word in the caption that is output.

Description

[Technical Field]

[0001] FIELD Embodiments of the present invention relate to an image processing device, method, and program. [Background technology]

[0002] Image description generation technology is a technique for describing the situation captured in image data (sometimes simply referred to as images) in text. This technology is being widely studied, with the expectation that it will be used to reduce the cost of manual administrative work, such as automatically recording work performed using photographs taken in factories, or automatically filling out electronic medical records in medical settings.

[0003] For example, in Non-Patent Document 1, a set of a large number of images and sentences describing the situation depicted in the images is used as training data, and the images are input to an image description generation model. to The model is trained to estimate sentences that describe the situation depicted in the image. This image description generation model is capable of generating simple sentences using common words that are frequently found in the dataset, but it does not allow for purposeful control of the content mentioned in the sentences.

[0004] Controllable image caption generation technology is a technology that has begun to be researched in recent years, in which a control signal is given along with image data as input to an image caption generator, and image captions are generated while controlling the content to be mentioned.

[0005] There is also an object area specification type image description generation technology that specifies a partial area of ​​an image to be mentioned as a control signal. The object area of ​​the control signal in this technology is automatically selected from multiple object areas detected by object detection technology that identifies the position and name of an object in the display area of ​​the image, so it may include areas shown in the image that are not directly related to the event that indicates the situation of the subject to be mentioned. In this case, the use of words unrelated to the event being mentioned may result in an unnatural sentence being generated. For example, if the event being mentioned is blood pressure measurement, but the control information includes a chair area, the generated sentence may include the phrase "with the chair," which is not directly related to the event.

[0006] Non-Patent Document 2 discloses a technique for generating image captions in which the name of an event to be mentioned and semantic role information related to the event are given as control signals. The semantic role is sometimes called the thematic role. The event name in this technique is, for example, "test". in The semantic role information is, for example, "subject," "object," or "location," and is an element necessary when describing the event name indicating the activity to be mentioned in a sentence. There is a possibility that a sentence generated using this technology will contain words that are not related to the event to be mentioned. low stomach. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi and Rita Cucchiara, “Meshed-Memory Transformer for Image Captioning”, in Proc. Of CVPR2020. [Non-patent document 2] Long Chen, Zhihong Jiang, Jun Xiao and Wei Liu, “Human-like Controllable Image Captioning with Verb-specific Semantic Roles”, in Proc. Of CVPR2021. Summary of the Invention [Problem to be solved by the invention]

[0008] However, the image description generation method described in Non-Patent Document 2 is divided into multiple steps, and each step requires a neural network model, which results in high costs associated with learning and inference. The neural network models in the above multiple steps consist of a model that estimates an area in an image for each semantic role given as a control signal, a model that estimates the order of the semantic roles, and a model that estimates words for each of the ordered semantic roles. During learning and inference, each model must be learned and inferred in order, which is expected to require time and effort such as adjusting the model parameters, resulting in high costs.

[0009] Furthermore, in Non-Patent Document 2, the estimation results of the previous step are used to perform the inference in the next step, so an error in the inference in the previous step cannot be corrected in the next step. If an error is made in the inference when estimating the order of semantic roles, there is a possibility that a sentence with an unnatural word order will be generated.

[0010] This invention has been made in light of the above circumstances, and its purpose is to provide an image processing device, method, and program that can appropriately generate image captions that explain the situation depicted in image data. [Means for solving the problem]

[0011] An image processing device according to one aspect of the present invention is Features extracted from name information indicating the name of the situation depicted in the image data; Display area an input unit that receives input of semantic role information indicating the semantic role of each word in an explanatory sentence that explains a situation depicted in the image; Features , name information, and semantic role information, a description that explains the situation depicted in the image data. The word sequence of the output sentence isand a description explaining the situation in a display area of ​​the image data. output statement that is a statement For each word in The corresponding image display area Position, and said output sentence and an output unit that outputs information indicating the semantic role of each word.

[0012] An image processing method according to one aspect of the present invention is performed by an image processing device. circle The image processing device Input section However, image data Features extracted from and the image data Display area receiving input of name information indicating the name of a situation depicted in the image data and semantic role information indicating the semantic role of each word in a description that explains the situation depicted in the image data; Output section of However, the above input Features , name information, and semantic role information, a description that explains the situation depicted in the image data. The word sequence of the output sentence is and an explanatory text explaining the situation in a display area of ​​the image data. The output statement is For each word in The corresponding image display area Position, and said output sentence and outputting information indicating the semantic role of each word. [Effects of the Invention]

[0013] According to the present invention, it is possible to appropriately generate an image caption that explains the situation depicted in the image data. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram showing an application example of an image processing device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing an example of the configuration of the image caption generator. [Figure 3] FIG. 3 is a flow chart showing an example of a learning processing routine executed by the image processing device. [Figure 4]FIG. 4 is a flowchart showing an example of an inference processing routine executed by the image processing device. [Figure 5] FIG. 5 is a block diagram showing an example of the hardware configuration of an image processing device according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0015] An embodiment of the present invention will be described below with reference to the drawings. <Configuration> First, the configuration of an image processing device according to one embodiment of the present invention will be described. This image processing device may also be called an image caption generation device or an image caption device. Figure 1 is a diagram showing an application example of an image processing device 100 according to one embodiment of the present invention. The image processing device 100 is composed of a computer equipped with a CPU (Central Processing Unit), RAM (Random Access Memory), and ROM (Read Only Memory) that stores a program for executing the image description processing routine described below, and is functionally configured as shown below.

[0016] As shown in FIG. 1, the image processing device 100 according to this embodiment is configured to include a fusion information creation unit 1, a storage unit 2, an image description unit 3, a parameter update unit 4, and an output unit 5.

[0017] The fusion information creation unit 1 receives input of image feature x, event information y, and semantic role information z from the memory unit 2, and creates fusion information w by fusing these image feature x, event information y, and semantic role information z.

[0018] The image feature x may be any tensor extracted from certain image data I, such as the tensor output by inputting an image into a VGG (Visual Geometry Group) network in Non-Patent Document 2.

[0019] The event information y is not particularly limited as long as it is a vector indicating the event name, which is the name of the situation shown in the display area of ​​certain image data I. For example, the event information y is a vector of a length corresponding to the number of event types, in which only the index corresponding to the event shown in the image has a value of "1" and the other indexes have values ​​of "0". For example, a vector indicating an event in a certain image may be manually created from pre-given event types, or an event shown in a certain image may be recognized using an event recognition model capable of recognizing pre-given event types, and this recognition result may be used to create a vector indicating an event in the image.

[0020] The semantic role information z is not particularly limited as long as it is a vector that indicates information necessary to explain the content of an event shown by certain image data I, that is, semantic role information that indicates the semantic role of each word in a sentence that explains a situation depicted in the image data. For example, the semantic role information z is a vector of a length according to the number of types of semantic roles, in which only the index value corresponding to the semantic role necessary to explain the content of the event shown in the image is "1", and the other index values ​​are "0". For example, a vector indicating the semantic role of an image may be created manually from among pre-given semantic role types, or a vector indicating the semantic role of the image may be created using the results of analysis by a language analyzer that classifies each word in a sentence describing the content of the same event as the event shown in an image into pre-given semantic role types.

[0021] The fusion information w is not particularly limited as long as it is a tensor created from the image feature amount x, the event information y, and the semantic role information z. Here, for example, assume that the size of image feature x is expressed by width w, height h, and number of channels c, event information y is a vector with length l_y, and semantic role information z is a vector with length l_z. In this case, the fusion information w is a tensor in which each vector of the event information y and the semantic role information z is replicated according to the size of the width w and height h of the image feature x. x When a tensor formed by duplicating the vector of event information y and a tensor formed by duplicating the vector of semantic role information z are superimposed in the channel direction, the size of the tensor is w in width, h in height, and (c+l_y+l_z) in number of channels.

[0022] The memory unit 2 stores one or more sets of the neural network (sometimes referred to as a screen description model) of the image description generator A, image feature x, event information y, semantic role information z, correct sentence C, correct position B, and correct semantic role sequence S. FIG. 2 is a diagram showing an example of the configuration of the image description generator. This FIG. 2 shows the image feature x, event information y, semantic role information z, fusion information w created by the fusion information creation unit 1, and the concept of the configuration of the image description generator A.

[0023] As shown in FIG. 2, the neural network of the image caption generator A is composed of a common neural network, a caption estimation neural network, a position estimation neural network, and a semantic role estimation neural network.

[0024] The neural network of this image description generator A is a common neuralThere are no particular limitations on the neural network as long as it inputs fusion information w into the network to obtain a common feature h, then inputs the common feature h into a description estimation neural network to output a description estimation result c, inputs the common feature h into a position estimation neural network to output a position estimation result b, and inputs the common feature h into a semantic role estimation neural network to output a semantic role estimation result s.

[0025] The common neural network is not particularly limited as long as it is a neural network that inputs fusion information w and outputs common feature amount h. This common feature h is not particularly limited as long as it is a tensor made up of feature vectors indicating each word of the output sentence, and is, for example, a tensor whose size is the output sentence length × l_h.

[0026] The description estimation neural network is not particularly limited as long as it is a network that receives input of a common feature amount h and outputs a description estimation result c. This description estimation result c is not limited to any particular tensor that indicates the word sequence of the output sentence, which is the image description. For example, it is a tensor whose size is "output sentence length l_h × vocabulary size D", and each element of this tensor is the occurrence probability of each word in the output sentence.

[0027] The location estimation neural network is not particularly limited as long as it is a network that receives a common feature amount h as input and outputs a location estimation result b. This position estimation result b is not limited to any particular tensor that indicates the position in the display area of ​​the image corresponding to each word in the output sentence, which is a sentence that describes the situation depicted in the image data, and is, for example, a tensor whose size is "output sentence length l_h x 4", and each element of this tensor is, for example, the coordinates x, y, and width w and height h values ​​based on the top left of the area in the image data that corresponds to each word in the output sentence.

[0028] The semantic role estimation neural network is not particularly limited as long as it is a network that inputs a common feature amount h and outputs a semantic role estimation result s. This semantic role estimation result s is not particularly limited as long as it is a tensor indicating the semantic role of each word in the output sentence, and is, for example, a tensor whose size is "output sentence length l_h × number of semantic role types p", and each element of this tensor is the occurrence probability of the semantic role of each word in the output sentence.

[0029] The correct answer sentence C stored in the memory unit 2 as described above is not particularly limited as long as it is a tensor that indicates the correct answer information for the word sequence of the output sentence that explains the situation depicted in certain image data I, and is, for example, a tensor whose size is "output sentence length l_h × vocabulary size D", and whose elements are a tensor in which only the index value corresponding to each word in the output sentence is "1", and the values ​​of the other indexes are "0".

[0030] The correct position B stored in the memory unit 2 as described above is not particularly limited as long as it is a tensor that indicates the position in the display area of ​​the image data that corresponds to each word in the output sentence, which is a sentence that describes the situation depicted in certain image data I, and is, for example, a tensor whose size is "output sentence length l_h × 4", and each element of this tensor is, for example, the coordinates x, y, width w, and height h based on the top left of the area in the image data that corresponds to each word in the output sentence.

[0031] The correct semantic role sequence S stored in the memory unit 2 as described above is not particularly limited as long as it is a tensor that indicates the semantic role of each word in an output sentence, which is a sentence that explains a situation that is reflected in certain image data I, and is, for example, a tensor whose size is "output sentence length l_h × number of semantic role types", and whose elements are a tensor in which only the index value corresponding to the semantic role of each word in the output sentence is "1", and the values ​​of the other indexes are "0".

[0032] The image description unit 3 receives fusion information w from the fusion information creation unit 1, receives the image description generator A from the memory unit 2, inputs the fusion information w into the neural network of the image description generator A, and outputs the description estimation result c, position estimation result b, and semantic role estimation result s output from this neural network.

[0033] The parameter update unit 4 receives the description estimation result c, the position estimation result b, and the semantic role estimation result s from the image description unit 3, and receives the image description generator A, the correct text C, the correct position B, and the correct semantic role sequence S from the memory unit 2, and updates the parameters of each neural network of the image description generator A (sometimes referred to as the parameters of the image description generator A) so as to satisfy the following three constraints.

[0034] The first constraint is to update the parameters of the image caption generator A so that the content of the caption estimation result c and the correct answer sentence C become closer to or the same, and there are no particular limitations on the learning method that is set to satisfy this constraint. For example, the parameter update unit 4 calculates the cross-entropy loss between the caption estimation result c and the correct answer sentence C as in the following equation (1), and updates the parameters of the caption estimation neural network of the image caption generator A so that this error becomes smaller, for example, to a certain value or less, or becomes zero.

[0035]

number

[0036] Here, k in Equation (1) is the index of the explanatory sentence estimation result c and the correct sentence C, and y k is the value in the description estimation result c output from the neural network of image description generator A, and t k is the value in the correct sentence C. k is a value where only the index value of the correct class is "1" and the values ​​of the other indexes are "0".

[0037] The second constraint is to update the parameters of the image description generator A so that the position estimation result b and the correct position B become closer to or the same as each other, and there is no particular limitation on the learning method as long as it is set to satisfy this constraint. For example, the parameter update unit 4 updates the parameters of the image description generator A so that the position estimation result b becomes closer to or the correct position B. b , y b , wb , h b} and the correct position B is {x B , y B , w B , h B}, {x b , y b , w b , h b} and {x B , y B , w B , h B}, and updates the parameters of the position estimation neural network of the image caption detector A so that this distance becomes small, for example, below a certain value, or becomes zero. The L1 distance is expressed as follows: |x b -x B |+|y b -y B |+|w b -w B |+|h b -h B |

[0038] The third constraint is to update the parameters of image description generator A so that the content of semantic role estimation result s and correct semantic role sequence S become closer to or the same, and there are no particular limitations on the learning method that can be used as long as it is set to satisfy this constraint. Parameter update unit 4 calculates the cross-entropy error between semantic role estimation result s and correct semantic role sequence S, for example, as in equation (2) below, and updates the parameters of the semantic role neural network of image description generator A so that this error becomes smaller, for example, to a certain value or less, or becomes zero.

[0039]

number

[0040] Here, m in Equation (2) is the index of the semantic role estimation result s and the correct semantic role sequence S, and y m is the value in the semantic role estimation result s output from the neural network of image description generator A, and t mis the value in the correct semantic role sequence S. m is a value where only the index that is the correct class has a value of "1" and the other indexes have values ​​of "0".

[0041] The output unit 5 receives the description estimation result c, the position estimation result b, and the semantic role estimation result s from the image description unit 3, and outputs these estimation results. This estimation result may be only an output sentence c' in which the description estimation result c is converted into a word string, or in addition to this output sentence c', position output information b' based on the position estimation result b may be further output as an estimation result, or an output semantic role s' in which each word of the output sentence is estimated from the semantic role estimation result s may be further output as an estimation result.

[0042] The output sentence c' is not particularly limited as long as it is a word string found based on the explanatory sentence estimation result c. For example, when the explanatory sentence estimation result c is a tensor whose size is "output sentence length I_h × vocabulary size D" and each element of this tensor is the occurrence probability of each word in the output sentence, it may be a word string found by searching for the sentence with the highest probability using a beam search with a beam width of "5" from the beginning of the output sentence, or it may be a word string with the highest probability found by calculating the occurrence probability for all possible word strings using a grid search.

[0043] The above-mentioned position output information b' is not particularly limited as long as it is data based on the position estimation result b, and may be, for example, a visualized image in which a rectangle is superimposed at the position indicated by the position estimation result b on the display area of ​​the image data, or a file in which the position estimation result b is output as text data.

[0044] The above-mentioned output semantic role s' is not particularly limited as long as it is data based on the semantic role estimation result s, and may be, for example, a visualized image in which a rectangle is superimposed at the position indicated by the position estimation result b on the display area of ​​the image data, and the index of the maximum value of the semantic role estimation result s is superimposed near this rectangle, for example in the upper left, or it may be a file in which the index of the maximum value of the semantic role estimation result s is output as text data.

[0045] <Action by image processing device> Next, the operation of the image processing device 100 according to this embodiment will be described. The image processing device 100 executes a learning processing routine and an inference processing routine, which will be described below. <<Learning processing routine>> First, the learning process routine will be described with reference to a flowchart of FIG. In this learning processing routine, first, image feature x, event information y, and semantic role information z are input, and fusion information w, which combines these pieces of information, is input to the neural network of image caption generator A. Then, caption estimation result c, position estimation result b, and semantic role estimation result s are output from the neural network of image caption generator A.

[0046] Then, the correct sentence C, the correct position B, and the correct semantic role sequence S are received from the memory unit 2, and the parameters of the various neural networks of the image description generator A are updated so that the above three constraints are met: (1) the content of the output description estimation result c and the correct sentence C are close to or the same, (2) the content of the output position estimation result b and the correct position B are close to or the same, and (3) the content of the output semantic role estimation result s and the correct semantic role sequence S are close to or the same.

[0047] First, in step S101, the fusion information creation unit 1 accepts input of image feature x, event information y, and semantic role information z from the memory unit 2, creates fusion information w by fusing these image feature x, event information y, and semantic role information z, and outputs this created fusion information w to the image description unit 3.

[0048] In step S102, the image description unit 3 receives the fusion information w output in step S101 and the image description generator A stored in the memory unit 2, and inputs this received fusion information w to the neural network of the image description estimator A. The image description unit 3 outputs a description estimation result c, a position estimation result b, and a semantic role estimation result s from the neural network of the image description estimator A, and outputs these description estimation result c, position estimation result b, and semantic role estimation result s to the parameter update unit 4.

[0049] In step S103, the parameter update unit 4 receives the description estimation result c, position estimation result b, and semantic role estimation result s output in step S102, and receives the image description generator A, correct sentence C, correct position B, and correct semantic role sequence S stored in the memory unit 2, and calculates the error between the description estimation result c and the correct sentence C (sometimes referred to as image description loss), the error between the position estimation result b and the correct position B (sometimes referred to as position estimation loss), and the error between the semantic role estimation result s and the correct semantic role sequence S (sometimes referred to as semantic role estimation loss).

[0050] Then, the parameter update unit 4 updates the parameters of various neural networks (parameters of the image description model) of the image description generator A so that the following three constraints are satisfied: (1) the content of the description estimation result c and the correct answer sentence C become closer to or the same, (2) the position estimation result b and the correct answer position B become closer to or the same, and (3) the content of the semantic role estimation result s and the correct answer semantic role sequence S become closer to or the same. The parameter update unit 4 stores the image description generator A with the updated parameters in the storage unit 2.

[0051] <<Inference processing routine>> Next, the inference processing routine will be described with reference to a flowchart of FIG. In this inference processing routine, first, image feature x, event information y, and semantic role information z are received as input, and fusion information w obtained by fusing these information is input to the neural network of image caption generator A. Then, caption estimation result c, position estimation result b, and semantic role estimation result s are output from the neural network of image caption generator A.

[0052] Then, the output sentence c' is obtained by converting the explanatory sentence estimation result c into a word string, the position output information b' is obtained by visualizing the position estimation result b, and the output semantic role s' is obtained by estimating each word in the output sentence from the semantic role estimation result s.

[0053] First, in step S201, the fusion information creation unit 1 accepts input of image feature x, event information y, and semantic role information z from the memory unit 2, creates fusion information w by fusing these image feature x, event information y, and semantic role information z, and outputs this fusion information w to the image description unit 3.

[0054] In step S202, the image description unit 3 1 The image description unit 3 receives the fusion information w output by the image description generator A stored in the memory unit 2 and inputs the received fusion information w into the neural network of the image description generator A. The image description unit 3 outputs the description estimation result c, the position estimation result b, and the semantic role estimation result s from the neural network of the image description estimator A, and outputs these description estimation result c, position estimation result b, and semantic role estimation result s to the output unit 5.

[0055] In step S203, based on the description estimation result c, position estimation result b, and semantic role estimation result s output in step S202, the output unit 5 outputs an output sentence c' obtained by converting the description estimation result c into a word string, position output information b' obtained by visualizing the position estimation result b, and an output semantic role s' obtained by inferring each word of the output sentence from the semantic role estimation result s.

[0056] According to one embodiment of the present invention, by inputting image data and an event name and semantic role information as control signals, it is possible to use a neural network to generate an image description with the event name and semantic role information given as control signals at low cost and with high accuracy.

[0057] Furthermore, according to one embodiment of the present invention, by inputting image data, an event name as a control signal, and semantic role information, and using a technique such as that disclosed in Non-Patent Document 2, it is possible to obtain the effect of training a neural network that simultaneously estimates semantic role domains, semantic role order, and words from semantic roles using a single neural network.

[0058] FIG. 5 is a block diagram showing an example of the hardware configuration of an image processing apparatus according to one embodiment of the present invention. 5, the image processing device 100 according to the embodiment is configured, for example, by a server computer or a personal computer, and has a hardware processor 111A such as a CPU. A program memory 111B, a data memory 112, an input / output interface 113, and a communication interface 114 are connected to the hardware processor 111A via a bus 115.

[0059] The communication interface 114 includes, for example, one or more wireless communication interface units, and enables transmission and reception of information to and from a communication network NW. As the wireless interface, for example, an interface that adopts a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.

[0060] The input / output interface 113 is connected to an input device 200 and an output device 300 that are attached to the image processing device 100 and used by a user or the like. The input / output interface 113 takes in operation data input by a user or the like via an input device 200 such as a keyboard, a touch panel, a touchpad, or a mouse, and outputs output data to an output device 300 including a display device using a liquid crystal or an organic electroluminescence (EL) display, etc. Note that the input device 200 and the output device 300 may be devices built into the image processing device 100, or may be input devices and output devices of other information terminals that can communicate with the image processing device 100 via a network NW.

[0061] The program memory 111B is a non-transitory tangible storage medium that is a combination of a non-volatile memory that can be written to and read from at any time, such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and a non-volatile memory such as a ROM, and stores programs necessary to execute various control processes, etc., according to one embodiment.

[0062] The data memory 112 is a tangible storage medium, for example, a combination of the above-mentioned nonvolatile memory and a volatile memory such as RAM, and is used to store various data acquired and created during various processing steps.

[0063] An image processing device 100 according to one embodiment of the present invention can be configured as a data processing device having a fusion information creation unit 1, an image description unit 3, a parameter update unit 4, and an output unit 5 shown in FIG. 1 as software processing function units.

[0064] Each information storage unit and storage unit 2 used as a working memory by each unit of image processing device 100 can be configured using data memory 112 shown in Fig. 5. However, these configured storage areas are not essential components within image processing device 100, and may be areas provided in an external storage medium such as a USB (Universal Serial Bus) memory, or a storage device such as a database server located in the cloud.

[0065] The processing function units in the fusion information creation unit 1, image description unit 3, parameter update unit 4, and output unit 5 can all be realized by having the hardware processor 111A read and execute a program stored in the program memory 111B. Note that some or all of these processing function units may be realized in various other forms, including integrated circuits such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0066] The methods described in each embodiment may be stored as a program (software means) that can be executed by a computer on a recording medium such as a magnetic disk (e.g., a floppy disk, a hard disk, etc.), an optical disk (e.g., a CD-ROM, a DVD, an MO, etc.), or a semiconductor memory (e.g., a ROM, a RAM, a flash memory, etc.), or may be transmitted and distributed via a communication medium. The program stored on the medium also includes a configuration program that configures the software means (including not only execution programs but also tables and data structures) that the computer executes. The computer that realizes this device reads the program stored on the recording medium and, in some cases, configures the software means using the configuration program, and executes the above-described processing by having the operation controlled by this software means. The term "recording medium" as used herein is not limited to a storage medium for distribution, but also includes a storage medium such as a magnetic disk or semiconductor memory installed inside the computer or in a device connected via a network.

[0067] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention. [Explanation of symbols]

[0068] 100...Image processing device 1. Fusion Information Creation Department 2...Storage section 3...Image description section 4...Parameter update section 5...Output section

Claims

1. an input unit that receives input of feature amounts extracted from image data, name information indicating the name of a situation depicted in the image data, and semantic role information indicating the semantic role of each word in an explanatory sentence that explains the situation depicted in a display area of ​​the image data; an output unit that outputs a word string of an output sentence that is an explanatory sentence that explains a situation depicted in the image data based on the feature amount, name information, and semantic role information input by the input unit, and outputs information indicating a position in a display area of ​​the image that corresponds to each word of the output sentence that is an explanatory sentence that explains the situation, and a semantic role of each word of the output sentence in a display area of ​​the image data; An image processing device comprising:

2. The output unit The feature amount, name information, and semantic role information input by the input unit are input to a neural network, and based on the result of this input, a word string of an output sentence which is an explanatory sentence that explains the situation depicted in the image data is output, and information indicating the position in the display area of ​​the image corresponding to each word of the output sentence that is an explanatory sentence that explains the situation in the display area of ​​the image data, and the semantic role of each word of the output sentence is output. The image processing device according to claim 1 .

3. a fusion information creation unit that creates fusion information by combining the feature amount, name information, and semantic role information input by the input unit, The output unit inputting the fusion information created by the fusion information creation unit into a neural network, and outputting feature information indicating the features of each word in the explanatory text that explains the situation based on the result of this input; based on the feature information, outputting an estimation result of a word string of an output sentence which is an explanatory sentence explaining the situation, an estimation result of a position in a display area of ​​the image data which corresponds to each word of the output sentence which is an explanatory sentence explaining the situation, and an estimation result of a semantic role of each word of the output sentence; The image processing device according to claim 1 .

4. Based on correct answer information for the word sequence of the output sentence, which is a description explaining the situation depicted in the image data, correct answer information for the position in the display area of ​​the image corresponding to each word of the output sentence, which is a description explaining the situation, in the display area of ​​the image data, and correct answer information for the semantic role of each word of the output sentence, and information indicating the word sequence of the output sentence, which is a description explaining the situation depicted in the image data and output using the neural network, the position in the display area of ​​the image corresponding to each word of the output sentence, which is a description explaining the situation, in the display area of ​​the image data and output using the neural network, and the semantic role of each word of the output sentence, which is output using the neural network, updating parameters of the neural network so that a word string of an output sentence, which is an explanatory sentence explaining the situation depicted in the image data and is output using the neural network, approaches correct answer information of the word string of the output sentence, which is the explanatory sentence; updating parameters of the neural network so that positions in the display area of ​​the image corresponding to each word of an output sentence, which is an explanatory sentence explaining the situation in the display area of ​​the image data and is output using the neural network, approach correct answer information of the positions; The parameters of the neural network are updated so that information indicating the semantic role of each word in the output sentence, which is output using the neural network, approaches correct information of the semantic role. further comprising an update unit; The image processing device according to claim 2 .

5. A method performed by an image processing device, comprising: an input unit of the image processing device receives input of a feature extracted from image data, name information indicating a name of a situation depicted in the image data, and semantic role information indicating a semantic role of each word in an explanatory sentence explaining the situation depicted in a display area of ​​the image data; an output unit of the image processing device outputs a word string of an output sentence, which is an explanatory sentence that explains the situation depicted in the image data, based on the input feature amount, name information, and semantic role information, and outputs information indicating a position in a display area of ​​the image corresponding to each word of the output sentence that is an explanatory sentence that explains the situation, and a semantic role of each word of the output sentence, in a display area of ​​the image data; An image processing method comprising:

6. The output by the output unit inputting the input feature amount, name information, and semantic role information into a neural network, and based on the result of this input, outputting a word string of an output sentence which is an explanatory sentence that explains the situation depicted in the image data, and outputting information indicating the position in the display area of ​​the image corresponding to each word of the output sentence that is an explanatory sentence that explains the situation in the display area of ​​the image data, and the semantic role of each word of the output sentence. The image processing method according to claim 5 .

7. The image processing device further comprises a fusion information creation unit that creates fusion information by fusing the input feature amount, name information, and semantic role information, The output by the output unit inputting the created fusion information into a neural network, and outputting feature information indicating the features of each word in the explanatory text that explains the situation based on the result of this input; outputting, based on the feature information, an estimation result of a word string of an output sentence which is an explanatory sentence explaining the situation, an estimation result of a position in a display area of ​​the image data which corresponds to each word of the output sentence which is an explanatory sentence explaining the situation, and an estimation result of a semantic role of each word of the output sentence, The image processing method according to claim 5 .

8. An image processing program that causes a processor to function as each unit of the image processing device according to any one of claims 1 to 4.

Images