Image language processing device, image language processing method, image language processing program, learning device, learning method, and learning program

Abstract

An image language processing device (10) generates text (Tx) for explaining a target image using an image language model, and comprises: an image encoder (12) that generates an image feature amount (Fi) from image data (I) of a target image; a knowledge encoder (13) that generates a knowledge feature amount (Fk) from the image feature amount (Fi); a feature amount synthesis unit (14) that generates a synthesis feature amount (Fc) from the image feature amount (Fi) and the knowledge feature amount (Fk); an alignment unit (15) that generates an alignment feature amount (Fa) from the synthesis feature amount (Fc); an output text decoder (16, 26, 36) that generates an output text token (Tt) from the alignment feature amount (Fa); and a text output unit (17) that generates text (Tx) from the output text token (Tt). The image language model is optimized through learning.

Description

Image language processing device, image language processing method, image language processing program, learning device, learning method, and learning program

[0001] The present disclosure relates to an image language processing device, an image language processing method, and an image language processing program that generate text that describes a target image using an image language model, as well as a learning device, a learning method, and a learning program for optimizing the image language model.

[0002] A vision language model (VLM), which is a learning model that explains images in natural language (text), can perform image captioning, which generates text that explains a target image, or visual question answering (VQA), which generates answers to questions about the target image, using a VLM optimized by learning using a dataset of a large number of images and text pairs on the web as learning data (see, for example, Patent Document 1). Furthermore, in order to improve the accuracy of the VLM output, a method has been proposed in which a scene graph that indicates the relationships or attributes of objects in the target image is combined with the VLM (see, for example, Patent Document 2).

[0003] Patent Publication No. 2017-534956 Chinese Patent Application Publication No. 116431793

[0004] However, in conventional VLM, image features created by an image encoder are input into a language model to execute a task, but because the texture (image features) and context (knowledge) contained in the image features are utilized without internal awareness, the knowledge contained in the image features is not handled efficiently, resulting in low output accuracy.

[0005] The method described in Patent Literature 1 attempts to improve answer accuracy by fusing and aligning language vectors and visual representations (image features), but does not use the context contained in the visual representation, resulting in insufficient improvement in output accuracy. Furthermore, the method described in Patent Literature 2 extracts context from an image by creating a scene graph from the image, and extracts image regions that are highly correlated with the question and uses them to answer the question, but does not use information outside the highly correlated image regions, resulting in insufficient improvement in output accuracy.

[0006] The present disclosure aims to provide an image language processing device, an image language processing method, and an image language processing program that can improve the accuracy of text that explains a target image, as well as a learning device, a learning method, and a learning program that can optimize an image language model so as to improve the accuracy of text that explains a target image.

[0007] The image language processing device disclosed herein is a device that generates text that explains a target image using an image language model, and is characterized by having an image encoder that generates image features from image data of the target image, a knowledge encoder that generates knowledge features from the image features, a feature synthesis unit that generates synthesized features from the image features and the knowledge features, an alignment unit that generates alignment features from the synthesized features, an output text decoder that generates output text tokens from the alignment features, and a text output unit that generates the text from the output text tokens.

[0008] The image language processing method disclosed herein is a method implemented by an image language processing device that generates text that explains a target image using an image language model, and is characterized by having the steps of generating image features from image data of the target image, generating knowledge features from the image features, generating composite features from the image features and the knowledge features, generating alignment features from the composite features, generating output text tokens from the alignment features, and generating the text from the output text tokens.

[0009] The learning device disclosed herein is a device that acquires learning data including image data of a target image and correct answer data of the target image, and generates an image language model for inferring text from the image data of the target image using the learning data, and is characterized by having an image encoder that generates image features from the image data of the target image, a knowledge encoder that generates knowledge features from the image features, a feature synthesis unit that generates synthesized features from the image features and the knowledge features, an alignment unit that generates alignment features from the synthesized features, an output text decoder that generates output text tokens from the alignment features, and an optimization unit that optimizes the image language model based on the output text tokens and the correct answer data.

[0010] The learning method disclosed herein is a method implemented by a learning device that acquires learning data including image data of a target image and correct answer data for the target image, and generates an image language model for inferring text from the image data of the target image using the learning data, and is characterized by having the steps of generating image features from the image data of the target image, generating knowledge features from the image features, generating composite features from the image features and the knowledge features, generating alignment features from the composite features, generating output text tokens from the alignment features, and optimizing the image language model based on the output text tokens and the correct answer data.

[0011] By using the image language processing device, image language processing method, or image language processing program of the present disclosure, it is possible to improve the accuracy of text that explains a target image using an image language model.

[0012] By using the learning device, learning method, or learning program of the present disclosure, it is possible to optimize an image language model so as to improve the accuracy of text that describes a target image.

[0013] FIG. 1 is a functional block diagram schematically showing the configuration of an image language processing device according to embodiment 1. FIG. 2 is a diagram showing an example of the hardware configuration of the image language processing device according to embodiment 1. FIG. 3 is a flowchart showing the inference operation of the image language processing device according to embodiment 1. FIG. 4 is a functional block diagram schematically showing the configuration of a learning device #1 according to embodiment 1. FIG. 5 is a flowchart showing the learning operation (operation using text error) of the learning device #1 according to embodiment 1. FIG. 6 is a flowchart showing the learning operation (operation using knowledge error) of the learning device #1 according to embodiment 1. FIG. 7 is a diagram showing an example of the hardware configuration of the learning device #1 according to embodiment 1. FIG. 8 is a functional block diagram schematically showing the configuration of a learning device #2 according to embodiment 1. FIG. 9 is a flowchart showing the learning operation (operation using text / knowledge error) of the learning device #2 according to embodiment 1. FIG. 10 is a functional block diagram schematically showing the configuration of an image language processing device according to embodiment 2. FIG. 11 is a flowchart showing the inference operation of the image language processing device according to embodiment 2. FIG. 12 is a functional block diagram schematically showing the configuration of a learning device according to embodiment 2. FIG. 13 is a functional block diagram schematically showing the configuration of an image language processing device according to embodiment 3. FIG. 14 is a flowchart showing the inference operation of the image language processing device according to embodiment 3. FIG. 15 is a functional block diagram schematically showing the configuration of a learning device according to embodiment 3.

[0014] Hereinafter, an image language processing device, an image language processing method, an image language processing program, a learning device, a learning method, and a learning program according to embodiments will be described with reference to the drawings. In the drawings, elements having the same or similar functions are given the same reference numerals. The following embodiments are merely examples, and the embodiments can be combined as appropriate and each embodiment can be modified as appropriate.

[0015] The image language processing device according to the embodiment is an inference device that generates text (also referred to as natural language or sentence) that explains a target image using an image language model (VLM) as a learning model, and, for example, describes an object shown in the image in natural language or generates an answer to a question about the target image. "Text that explains the target image" is natural language related to the whole or part of the target image, such as a sentence that explains the whole or part of the target image, or a sentence that answers a question about the whole or part of the target image. The image language processing device according to the embodiment is, for example, a computer. Computers include personal computers (PCs), server computers, and mobile terminals such as smartphones.

[0016] The image language processing method according to the embodiment is a method for generating text that explains a target image using a VLM, and is implemented by, for example, an image language processing device.

[0017] The image language processing program according to the embodiment is a software program that generates text that explains a target image using VLM, and is executed by, for example, a computer that functions as an image language processing device.

[0018] The learning device according to the embodiment is a device that generates (optimizes) a VLM, which is a learning model for inferring text (e.g., a sentence explaining the target image) from the image data of the target image, using learning data including image data of the target image and correct answer data. The learning device is, for example, a computer.

[0019] The learning method according to the embodiment is a method implemented by a learning device, and uses learning data to generate (optimize) a VLM, which is a learning model for inferring text from image data of a target image.

[0020] The learning program according to the embodiment is a software program that uses learning data to generate a learning model for inferring text from image data of a target image, and is executed, for example, by a computer as a learning device.

[0021] The learning device and the image language processing device may be configured as a common computer, which may be a computer system configured by a plurality of information processing devices connected to each other so as to be able to communicate via a network or the like.

[0022] The learning device and the image language processing device may be configured as separate computers, and each computer may be a computer system configured by multiple information processing devices connected to each other so as to be able to communicate via a network or the like.

[0023] <<1>> Embodiment 1 <<1-1>> Image and language processing device 10 <Configuration overview>> Fig. 1 is a functional block diagram showing a schematic configuration of an image and language processing device 10 according to embodiment 1. The image and language processing device 10 has an image input unit 11 as a data acquisition unit that generates image data I of a target image from electronic data D of the target image captured by a camera (e.g., a video camera), and an inference unit 10a that generates text Tx that explains the target image using a VLM as a learning model.

[0024] The inference unit 10a includes an image encoder 12 that generates image features Fi from image data I of a target image, a knowledge encoder 13 that generates knowledge features Fk from the image features Fi, a feature synthesis unit 14 that generates composite features Fc from the image features Fi and the knowledge features Fk, an alignment unit 15 that generates alignment features Fa from the composite features Fc, an output text decoder 16 that generates output text tokens Tt from the alignment features Fa, and a text output unit 17 that generates text Tx from the output text tokens Tt.

[0025] <Image Input Unit 11> The image input unit 11 receives electronic data D of a target image as input, and generates image data I that can be expressed as a vector from the electronic data D. The image data I is data whose image size, contrast, etc. have been adjusted to make it easier to handle in subsequent processing.

[0026] <Image Encoder 12> The image encoder 12 receives image data I as input and generates image features Fi that can be expressed as vectors based on the image data I. In the image encoder 12 using deep learning, the image features Fi are treated as features that combine image features (e.g., edges, colors, patterns, etc.) and context (e.g., object relationships, situations, etc.) without distinguishing between them. When generating the image features Fi, a neural network (NN) such as a convolutional neural network (CNN) or a transformer can be used. When a transformer is used, the architecture is less susceptible to local bias in the image and correlations between distant pixels in the image are also calculated, so it is considered possible to include context information in the image features Fi. The context information includes, for example, context information.

[0027] <Knowledge Encoder 13> The knowledge encoder 13 receives image features Fi as input, and generates knowledge features Fk that can be expressed as vectors based on the image features Fi. The knowledge features Fk are obtained based on one or both of the image features Fi and context extracted from the image. In this embodiment, knowledge information in the form of clear scalar values ​​such as a scene graph is not used, and for example, a scene graph feature immediately before being decoded into scene graph information is used as the knowledge feature Fk.

[0028] The knowledge feature Fk includes one or more of object information indicating an object in the target image and its position and size, object relationship information indicating the relationship between an object and other objects (i.e., the relationship between objects), pixel attribute information indicating the attributes of pixels in the target image, image attribute information indicating the attributes of the target image, distance information indicating the distance in the target image, anomaly information indicating an abnormality in the target image (e.g., the presence or absence of an abnormality, the location where the abnormality occurs, the type of abnormality, etc.), and image description information which is information describing the whole or part of the target image in natural language.

[0029] For example, the knowledge encoder 13 can generate a knowledge feature Fk including information (object information) indicating the position and size of one or more predefined objects. The knowledge encoder 13 can also generate a knowledge feature Fk including information (object relation information) indicating the relationship between one or more predefined objects and other objects. The knowledge encoder 13 can also generate a knowledge feature Fk including one or more predefined pixel attribute information. The knowledge encoder 13 can also generate a knowledge feature Fk including one or more predefined image attribute information. The knowledge encoder 13 can also generate a knowledge feature Fk including at least distance information. The knowledge encoder 13 can also generate a knowledge feature Fk including at least anomaly information. The knowledge encoder 13 can also generate a knowledge feature Fk including at least information describing the image in natural language. The type of knowledge feature Fk generated by the knowledge encoder 13 can also be configured to be automatically set depending on the target image. The type of knowledge feature Fk generated by the knowledge encoder 13 can also be configured to be user-settable.

[0030] <Feature Synthesis Unit 14> The feature synthesis unit 14 receives the image feature Fi and the knowledge feature Fk as input, and generates a synthesized feature Fc that can be expressed as a vector based on the image feature Fi and the knowledge feature Fk. The synthesized feature Fc is a feature obtained by combining the image feature Fi and the knowledge feature Fk. Combination methods include a method of concatenating the image feature Fi and the knowledge feature Fk to generate a synthesized feature, and a method of projecting the image feature Fi and the knowledge feature Fk onto the synthesized feature using a neural network.

[0031] <Alignment Unit 15> The alignment unit 15 receives the composite feature Fc as input and generates an alignment feature Fa that can be expressed as a vector based on the composite feature Fc. The alignment feature Fa is a feature that has been aligned (adjusted) so that the image feature Fi can be treated as a text feature in order to input the image feature Fi to the output text decoder 16. For example, the alignment unit 15 performs projective transformation of the composite feature Fc using a neural network so that the alignment feature Fa can be treated as a text feature so that the image-derived composite feature Fc can be input to the output text decoder 16.

[0032] <Output Text Decoder 16> The output text decoder 16 receives alignment features Fa as input and generates output text tokens Tt that can be expressed as vectors based on the alignment features Fa. The output text decoder 16 generates the output text tokens Tt by projectively transforming the alignment features Fa using a large-scale neural network called a Large Language Model (LLM).

[0033] <Text Output Unit 17> The text output unit 17 receives an output text token Tt as input and generates text Tx based on the output text token Tt. Here, the correspondence between the word-by-word vector pattern of the output text token Tt and the word is prepared in advance and stored in a storage device.

[0034] <Hardware Configuration> Fig. 2 is a diagram showing an example of the hardware configuration of the image language processing device 10 according to embodiment 1. The image language processing device 10 has at least one processor 51 such as a CPU (Central Processing Unit), a memory 52 as a storage device such as a RAM (Random Access Memory), a storage device 53 as a non-volatile storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and an interface 54 to which a camera, a display device, etc. are connected. These components may be configured using dedicated processing circuits.

[0035] The processor 51 can execute the image language processing method according to embodiment 1. An image language processing program for executing the image language processing method is recorded on a recording medium such as an SD memory card (Secure Digital memory card) or a USB (Universal Serial Bus) memory card, or is provided by downloading via a network. The hardware configuration shown in FIG. 2 is an example, and various modifications to the hardware configuration are possible.

[0036] <Inference Operation> Figure 3 is a flowchart showing the inference operation of the image language processing device 10 according to the first embodiment. As shown in Figure 3, in inference using VLM, the image input unit 11 first generates image data I from electronic data D of the target image (step S11), and the image encoder 12 generates image features Fi from the image data I (step S12). Next, the knowledge encoder 13 generates knowledge features Fk from the image features Fi (step S13), and the feature synthesis unit 14 generates synthesized features Fc from the image features Fi and the knowledge features Fk (step S14). The alignment unit 15 generates alignment features Fa from the synthesized features Fc (step S15), and the output text decoder 16 generates output text tokens Tt from the alignment features Fa (step S16). Next, the text output unit 17 generates text Tx (natural language) explaining the target image from the output text tokens Tt (step S17).

[0037] <1-2> Learning Device #1 <Configuration> Figure 4 is a functional block diagram showing a schematic configuration of the learning device #1 (110) according to embodiment 1. The learning device #1 (110) has a model generation unit 110a that acquires learning data including image data I of a target image and supervised data of the target image, and generates (optimizes) a VLM for inferring text from the image data I of the target image using the learning data. The model generation unit 110a uses, as supervised data, text supervised data Lt stored in a text supervised data storage unit 111 and knowledge supervised data Lk stored in a knowledge supervised data storage unit 113.

[0038] The model generation unit 110a includes an image encoder 12 that generates image features Fi from image data I of a target image, a knowledge encoder 13 that generates knowledge features Fk from the image features Fi, a feature synthesis unit 14 that generates composite features Fc from the image features Fi and the knowledge features Fk, an alignment unit 15 that generates alignment features Fa from the composite features Fc, an output text decoder 16 that generates output text tokens Tt from the alignment features Fa, and an optimization unit that optimizes the VLM as a learning model based on the output text tokens Tt and the supervised data. In Figure 4, the optimization unit includes one or both of an output text loss function calculation unit 112 that calculates a loss function of the output text tokens based on the output text tokens Tt and the supervised text data Lt, and a knowledge loss function calculation unit 114 that calculates a loss function of the knowledge features based on the knowledge features Fk and the supervised knowledge data Lk.

[0039] 5 is a flowchart showing the learning operation (operation using the text error Et) of the learning device #1 (110) according to embodiment 1. During training, the model generation unit 110a of the learning device #1 (110) repeatedly performs the same processing as during inference.

[0040] First, the model generation unit 110a sets the number of epochs N1, the total number of correct answer data M1, and the number of batches B1 (step S101), sets count values ​​i, j, and k to initial values ​​(=0) (steps S102 to S104), and generates image data I from electronic learning data D captured by a camera (step S105). Next, the model generation unit 110a generates image features Fi from the image data I (step S106), generates knowledge features Fk from the image features Fi (step S107), generates composite features Fc from the image features Fi and the knowledge features Fk (step S108), generates alignment features Fa from the composite features Fc (step S109), and generates output text tokens Tt from the alignment features Fa (step S110).

[0041] When training the weights of the models of the feature synthesis unit 14 and the alignment unit 15, the model generation unit 110a inputs the text supervised answer data Lt stored in the text supervised answer data storage unit 111 and the generated output text tokens Tt to the output text loss function calculation unit 112. The output text loss function calculation unit 112 calculates the text error Et between the text supervised answer data Lt and the output text tokens Tt (step S111). Here, the text supervised answer data Lt is the text supervised answer data corresponding to the image data I. The output text loss function calculation unit 112 performs contrastive learning to calculate similarity and cross-entropy error for pairs of image and text supervised answer data for batch number B1 (steps S105 to S112, S116). However, the loss functions used are not limited to these.

[0042] The model generation unit 110a repeats the process of adjusting the weights of the learning models of the feature synthesis unit 14 and the alignment unit 15 using the text error Et a number of times expressed by the following formula (1) (steps S105 to S118): Number of times=(N1×M1) / B1 (1)

[0043] 6 is a flowchart showing the learning operation (operation using the knowledge error Ek) of the learning device #1 (110) according to embodiment 1. During training, the model generation unit 110a of the learning device #1 (110) repeatedly performs the same processing as during inference.

[0044] First, the model generation unit 110a sets the number of epochs N2, the total number of correct answer data M2, and the number of batches B2 (step S201), sets the count values ​​i, j, and k to initial values ​​(=0) (steps S202 to S204), and generates image data I from electronic learning data D captured by a camera (step S205). Next, the model generation unit 110a generates image features Fi from the image data I (step S206), and generates knowledge features Fk from the image features Fi (step S207).

[0045] When training the weights of the knowledge model, the model generation unit 110a inputs the knowledge correct answer data Lk stored in the knowledge correct answer data storage unit 113 and the generated knowledge feature Fk to the knowledge loss function calculation unit 114. The knowledge loss function calculation unit 114 calculates the knowledge error Ek between the knowledge correct answer data Lk and the generated knowledge feature Fk (step S208). Here, the knowledge correct answer data Lk is knowledge correct answer data corresponding to the input image. The knowledge correct answer data is one or more of information regarding the object, position, and size corresponding to the input image, information indicating the relationship between the object and other objects, pixel attribute information, image attribute information, distance information, anomaly information, and image description information. In addition, the knowledge loss function calculation unit 114 uses a loss function corresponding to each piece of knowledge, and the knowledge error Ek can be a mean squared error, a mean absolute error, a mean squared logarithmic error, a Hinge error, a cross-entropy error, or the like. However, the errors used are not limited to these. The knowledge loss function calculation unit 114 performs contrastive learning to calculate errors for pairs of image and correct knowledge data for the batch number B2 (steps S205 to S209, S216).

[0046] The model generation unit 110a repeats the process of adjusting the weight of the learning model of the knowledge encoder 13 using the knowledge error Ek the number of times expressed by the following equation (2) (steps S205 to S218): Number of times=(N2×M2) / B2 (2)

[0047] 7 is a diagram showing an example of the hardware configuration of the learning device #1 (110) according to embodiment 1. The learning device #1 (110) has at least one processor 61 such as a CPU, a memory 62 as a storage device such as RAM, a storage device 63 as a non-volatile storage device such as an HDD or SSD, and an interface 64 to which a camera, a display device, etc. are connected. These components may be configured using dedicated processing circuits.

[0048] The processor 61 can implement the learning method according to the first embodiment. A learning program for implementing the learning method is provided, for example, by being recorded on a recording medium such as a USB memory card, or by being downloaded via a network. The hardware configuration shown in Fig. 5 is an example, and various modifications to the hardware configuration are possible.

[0049] <1-3> Learning Device #2 <Configuration> Fig. 8 is a functional block diagram showing an outline of the configuration of learning device #2 (120) according to embodiment 1. Learning device #2 (120) has a model generation unit 120a that acquires learning data including image data I of a target image and supervised data of the target image, and generates (optimizes) a VLM for inferring text from the image data I of the target image using the learning data. The model generation unit 120a uses text / knowledge supervised data Ltk stored in a text / knowledge supervised data storage unit 121 as supervised data.

[0050] Similar to the model generation unit 110a, the model generation unit 120a includes an image encoder 12, a knowledge encoder 13, a feature synthesis unit 14, an alignment unit 15, an output text decoder 16, and an optimization unit. In Fig. 8, the optimization unit includes a text-knowledge loss function calculation unit 122 that calculates a loss function of the output text token Tt based on the output text token Tt and the text-knowledge correct answer data Ltk.

[0051] <Hardware Configuration> The hardware configuration of the learning device #2 (120) is the same as that shown in FIG.

[0052] 9 is a flowchart showing the learning operation (operation using the text-knowledge error Etk) of the learning device #2 (120) according to embodiment 1. During training, the model generation unit 120a of the learning device #2 (120) repeatedly performs the same processing as during inference.

[0053] First, the model generation unit 120a sets the number of epochs N3, the total number of correct answer data M3, and the number of batches B3 (step S301), sets count values ​​i, j, and k to initial values ​​(=0) (steps S302 to S304), and generates image data I from electronic learning data D captured by a camera (step S305). Next, the model generation unit 120a generates image features Fi from the image data I (step S306), generates knowledge features Fk from the image features Fi (step S307), generates composite features Fc from the image features Fi and the knowledge features Fk (step S308), generates alignment features Fa from the composite features Fc (step S309), and generates output text tokens Tt from the alignment features Fa (step S310).

[0054] When training the weights of the models of the feature synthesis unit 14 and the alignment unit 15, the model generation unit 120a inputs the text / knowledge correct answer data Ltk stored in the text / knowledge correct answer data storage unit 121, the output text token Tt generated by the output text decoder 16, and the knowledge feature Fk generated by the knowledge encoder 13 to the text / knowledge loss function calculation unit 122. The text / knowledge loss function calculation unit 122 calculates the text / knowledge error Etk between the text / knowledge correct answer data Ltk, the output text token Tt, and the knowledge feature Fk (step S311). Here, the text / knowledge correct answer data Ltk is text and knowledge correct answer data corresponding to the image data I. The text / knowledge correct answer data Ltk is composed of the text correct answer data Lt and the knowledge correct answer data Lk for the same input image. The text / knowledge loss function calculation unit 122 calculates a loss function for the output text token Tt based on the input text token Tt and the text supervised answer data Lt, and calculates a loss function for the knowledge feature Fk based on the knowledge feature Fk and the knowledge supervised answer data Lk. The text / knowledge loss function calculation unit 122 simultaneously performs contrastive learning on pairs of image-text supervised answer data for the batch number B3, and calculates the mean squared error, mean absolute error, mean squared logarithmic error, Hinge error, and cross-entropy error for the supervised answer knowledge and knowledge feature. The text / knowledge loss function calculation unit 122 performs contrastive learning on pairs of image and text / knowledge supervised answer data for the batch number B3, calculating similarity and cross-entropy error (steps S305 to S312, S316). However, the loss functions used are not limited to these.

[0055] The model generation unit 120a repeats the process of adjusting the weights of the learning models of the feature synthesis unit 14 and the alignment unit 15 using the text-knowledge error Etk the number of times expressed by the following equation (3) (steps S305 to S318): Number of times=(N3×M3) / B3 (3)

[0056] <<1-4>> Effects As described above, by using the image language processing device 10, image language processing method, and image language processing program according to embodiment 1, knowledge about the target image (e.g., background, situation, relationships) can be extracted and utilized as knowledge features Fk, thereby improving the performance of the VLM task (i.e., improving the accuracy of the output text Tx).

[0057] Furthermore, by using the learning device, learning method, and learning program according to embodiment 1, knowledge about the target image (e.g., background, situation, relationship) can be extracted and utilized as knowledge features Fk, thereby improving the accuracy of the learning model used in the image language processing device 10.

[0058] <<2>> Embodiment 2 <<2-1>> Image language processing device 20 <Configuration>> Fig. 10 is a functional block diagram showing an outline of the configuration of an image language processing device 20 according to embodiment 2. The image language processing device 20 has an image input unit 11, a text input unit 21 that converts input text Txi into input text tokens Tti, and an inference unit 20a that generates text Tx that explains a target image using a VLM as a learning model.

[0059] The inference unit 20a includes an image encoder 12 that generates image features Fi from image data I of a target image, a knowledge encoder 13 that generates knowledge features Fk from the image features Fi, a feature synthesis unit 14 that generates composite features Fc from the image features Fi and the knowledge features Fk, an alignment unit 15 that generates alignment features Fa from the composite features Fc, an output text decoder 26 that generates output text tokens Tt from the alignment features Fa and input text tokens Tti provided from the text input unit 21, and a text output unit 17 that generates text Tx from the output text tokens Tt.

[0060] The text input unit 21 receives an input text Txi, which is text information, and generates an input text token Tti, which can be expressed as a vector, from the input text Txi. Here, the correspondence between the word-by-word vector patterns of the input text token and the words is prepared in advance and stored in a storage device.

[0061] The inference unit 20a of the image language processing device 20 receives image data I and input text tokens Tti as input, and outputs text Tx. According to the image language processing device 20, when a question is asked in natural language from the text input unit 21 about the image (image data I) (for example, when a user inputs a question as input text Txi), an answer can be obtained in natural language as text Tx.

[0062] The output text decoder 26 receives the alignment feature Fa and the input text token Tti and generates the output text token Tt. For example, the output text decoder 26 concatenates the alignment feature Fa and the input text token Tti as a vector and performs a projective transformation using a neural network to generate the output text token Tt.

[0063] <Hardware Configuration> The hardware configuration of the image language processing device 20 is the same as that shown in FIG.

[0064] <Inference Operation> Figure 11 is a flowchart showing the operation of the image language processing device 20 according to the second embodiment. As shown in Figure 11, the image input unit 11 first generates image data I from electronic data D of a target image (step S11), and the image encoder 12 generates image features Fi from the image data I (step S12). Next, the knowledge encoder 13 generates knowledge features Fk from the image features Fi (step S13), the feature synthesis unit 14 generates synthesized features Fc from the image features Fi and the knowledge features Fk (step S14), and the alignment unit 15 generates alignment features Fa from the synthesized features Fc (step S15). Next, the output text decoder 16 generates output text tokens Tt from the alignment features Fa and the input text tokens Tti (step S26). The text output unit 17 generates text Tx (natural language) explaining the target image from the output text tokens Tt (step S17).

[0065] 12 is a functional block diagram showing a schematic configuration of a learning device 210 according to the second embodiment. The learning device 210 has a model generation unit 210a that acquires learning data including image data I of a target image and supervised data for the target image, and generates (optimizes) a VLM using the learning data to infer text from the image data I of the target image. The model generation unit 210a uses the text token supervised data Ltt stored in the text token supervised data storage unit 211 as the supervised data. However, the model generation unit 210a may use the knowledge supervised data Lk stored in the knowledge supervised data storage unit 113 as the supervised data, in addition to the text token supervised data Ltt, as in the first embodiment.

[0066] The model generation unit 210a includes an image encoder 12 that generates image features Fi from image data I of a target image, a knowledge encoder 13 that generates knowledge features Fk from the image features Fi, a feature synthesis unit 14 that generates a composite feature Fc from the image features Fi and the knowledge features Fk, an alignment unit 15 that generates alignment features Fa from the composite feature Fc, an output text decoder 16 that generates an output text token Tt from the alignment features Fa, and an optimization unit that optimizes the VLM as a learning model based on the output text token Tt and the supervised data. In Figure 12, the optimization unit includes one or both of a text loss function calculation unit 212 that calculates a loss function of the output text token based on the output text token Tt and the text token supervised data Ltt, and a knowledge loss function calculation unit 214 that calculates a loss function of the knowledge feature based on the knowledge features Fk and the knowledge supervised data Lk.

[0067] <Learning Operation> During learning, the model generation unit 210a sets the number of epochs, the total number of correct answer data, and the number of batches, generates image data I from electronic data D for learning captured by a camera, generates image features Fi from the image data I, generates knowledge features Fk from the image features Fi, generates composite features Fc from the image features Fi and the knowledge features Fk, generates alignment features Fa from the composite features Fc, and generates output text tokens Tt from the alignment features Fa.

[0068] When training the weights of the models of the feature synthesis unit 14 and the alignment unit 15, the model generation unit 210a inputs the text token supervised data Ltt stored in the text supervised data storage unit 111 and the generated output text token Tt to the text loss function calculation unit 212. The text loss function calculation unit 212 calculates the text token error Ett between the text token supervised data Ltt and the output text token Tt. Here, the text token supervised data Ltt is the supervised data of the text token corresponding to the image data I. The text loss function calculation unit 212 performs contrastive learning, for example, by calculating similarity and cross-entropy error for pairs of text token supervised data for a set number of batches. However, the loss function used is not limited to these.

[0069] When training the weights of the knowledge model, the model generation unit 210a inputs the knowledge correct answer data Lk stored in the knowledge correct answer data storage unit 213 and the generated knowledge feature Fk to the knowledge loss function calculation unit 214. The knowledge loss function calculation unit 214 calculates the knowledge error Ek between the knowledge correct answer data Lk and the generated knowledge feature Fk. Here, the knowledge correct answer data Lk is knowledge correct answer data corresponding to the input image. The knowledge correct answer data is one or more of information regarding the object, position, and size corresponding to the input image, information indicating the relationship between the object and other objects, pixel attribute information, image attribute information, distance information, anomaly information, and image description information. In addition, the knowledge loss function calculation unit 214 uses a loss function corresponding to each piece of knowledge, and the knowledge error Ek can be a mean squared error, a mean absolute error, a mean squared logarithmic error, a Hinge error, a cross-entropy error, or the like. However, the errors used are not limited to these.

[0070] The model generation unit 210a repeatedly performs a process of adjusting the weights of the learning models of the feature synthesis unit 14 and the alignment unit 15 using the text token error Ett. The model generation unit 210a also repeatedly performs a process of adjusting the weights of the learning model of the knowledge encoder 13 using the knowledge error Ek.

[0071] <Hardware Configuration> The hardware configuration of the learning device 210 is the same as that shown in FIG.

[0072] {2-3} Effects As described above, by using the image language processing device 20, image language processing method, and image language processing program according to embodiment 2, knowledge about the target image can be extracted and utilized as knowledge features Fk, thereby improving the accuracy of the output text Tx. Furthermore, when a question is asked in natural language about the image (image data I) from the text input unit 21, an answer can be obtained in natural language as text Tx in response to the question about the image (input text).

[0073] Furthermore, by using the learning device 210, learning method, and learning program according to embodiment 2, knowledge about the target image can be extracted and utilized as knowledge feature Fk, thereby improving the accuracy of the learning model used in the image language processing device 20.

[0074] <3> Embodiment 3 <3-1> Image language processing device 30 <Configuration> Fig. 13 is a functional block diagram showing an outline of the configuration of an image language processing device 30 according to embodiment 3. The image language processing device 30 has an image input unit 11, an inference unit 30a, a text input unit 41 that generates input text tokens Tti from input text Txi, and a knowledge encoder information storage unit 43 that stores knowledge information Sk.

[0075] The inference unit 30a includes an image encoder 12 that generates image features Fi from image data I of a target image, a knowledge encoder 33 that generates knowledge features Fk from the image features Fi, a feature synthesis unit 14 that generates composite features Fc from the image features Fi and the knowledge features Fk, an alignment unit 15 that generates alignment features Fa from the composite features Fc, an output text decoder 36 that generates output text tokens Tt from the alignment features Fa and input text features Fti provided from an input text encoder 42, a text output unit 17 that generates text Tx from the output text tokens Tt, an input text encoder 42 that generates input text features Fti from the input text tokens Tti, and a knowledge encoder selection unit 44 that generates knowledge encoder selection information Se from the input text features Ft and knowledge information Sk.

[0076] The knowledge encoder 33 generates knowledge features Fk from the image features Fi and the knowledge encoder selection information Se. The output text decoder 36 generates output text tokens Tt from the alignment features Fa and the input text tokens Tti.

[0077] The knowledge encoder information storage unit 43 stores knowledge feature information that can be expressed as a vector and that is generated by encoding an explanatory sentence that represents each piece of knowledge feature information.

[0078] The input text encoder 42 receives the input text token Tti as an input and generates input text features Fti that can be represented as vectors based on the input text token Tti. A neural network such as a transformer can be used to generate the input text features Fti.

[0079] The knowledge encoder selection unit 44 receives the input text features Fti and the knowledge information Sk as inputs, and creates knowledge encoder selection information Se based on the input text features Fti and the knowledge information Sk. The knowledge encoder selection unit 44 calculates the similarity between the input text features Fti and each piece of knowledge information Sk, and creates knowledge encoder selection information Se including knowledge information to be used in the knowledge encoder 33 according to predefined conditions. The predefined conditions include the knowledge information with the highest similarity, knowledge information that is within a predefined rank after sorting by similarity, knowledge information that is equal to or higher than a predefined similarity, etc.

[0080] The knowledge encoder 33 receives the image feature Fi and the knowledge encoder selection information Se as inputs, and generates a knowledge feature Fk that can be expressed as a vector. The knowledge feature Fk can be created by using the knowledge encoder information to acquire one or more pieces of information from predefined object, position, and size information, predefined object-to-object relationship information, predefined pixel attribute information, predefined image attribute information, distance information, anomaly information, image description information, etc., and concatenating them as a vector. Except for the above, the inference unit 30a is the same as the inference unit 10a in embodiment 1.

[0081] <Hardware Configuration> The hardware configuration of the image language processing device 30 is the same as that shown in FIG.

[0082] <Inference Operation> Fig. 14 is a flowchart showing the operation of the image language processing device 30 according to embodiment 3. As shown in Fig. 14, first, the image input unit 11 generates image data I from electronic data D of a target image (step S11), and the image encoder 12 generates image features Fi from the image data I (step S12).

[0083] When an input text Txi is input from the text input unit 41, an input text token Tti is generated from the input text, and the input text encoder 42 generates input text features Fti from the input text token Tti (step S31). Next, the knowledge encoder selection unit 44 generates knowledge encoder selection information Se from the input text features Ft and knowledge information Sk (step S32).

[0084] Next, the knowledge encoder 33 generates a knowledge feature Fk from the image feature Fi and the knowledge encoder selection information Se (step S33). The feature synthesis unit 14 generates a synthesized feature Fc from the image feature Fi and the knowledge feature Fk (step S14), and the alignment unit 15 generates an alignment feature Fa from the synthesized feature Fc (step S15). Next, the output text decoder 36 generates an output text token Tt from the alignment feature Fa and the input text token Tti (step S34). The text output unit 17 generates text Tx (natural language) explaining the target image from the output text token Tt (step S17). The output text decoder 36 generates an output text token Tt from the alignment feature Fa and the input text token Tti.

[0085] 15 is a functional block diagram showing a schematic configuration of a learning device 310 according to embodiment 3. The learning device 310 has a model generation unit 310a that acquires learning data including image data I of a target image and supervised data for the target image, and generates (optimizes) a VLM using the learning data to infer text from the image data I of the target image. The model generation unit 310a uses, as supervised data, the knowledge supervised data Lsk stored in the knowledge information supervised data storage unit 311 and the text supervised data Lxi stored in the text supervised data storage unit 312. However, the model generation unit 310a may use either the knowledge supervised data Lsk or the text supervised data Lxi as the supervised data.

[0086] <Learning Operation> During learning, the model generation unit 310a sets the number of epochs, the total number of correct answer data, and the number of batches, generates image data I from electronic data D for learning captured by a camera, receives as input the text correct answer data Lxi stored in the text correct answer data storage unit 312 and the knowledge information correct answer data stored in the knowledge information correct answer data storage unit 311, calculates a text token error Ett between the text token correct answer data and the output text token Tt, and calculates a knowledge error Ek between the knowledge correct answer data Lsk and the generated knowledge feature Fk.

[0087] The model generation unit 310a repeatedly adjusts the weights of the learning model using the text token error Ett. The model generation unit 310a also repeatedly adjusts the weights of the learning model of the knowledge encoder 13 using the knowledge error Ek.

[0088] <Hardware Configuration> The hardware configuration of the learning device 310 is the same as that shown in FIG.

[0089] {3-3} Effects As described above, by using the image language processing device 30, image language processing method, and image language processing program according to embodiment 3, knowledge about the target image can be extracted and utilized as knowledge features Fk, thereby improving the accuracy of the output text Tx. Furthermore, when a question is asked in natural language from the text input unit 41, an answer can be obtained in natural language as text Tx in response to the question about the image (input text).

[0090] Furthermore, by using the learning device 310, learning method, and learning program according to embodiment 3, knowledge about the target image can be extracted and utilized as knowledge feature Fk, thereby improving the accuracy of the learning model used in the image language processing device 30.

[0091] 10, 20, 30 Image language processing device, 10a, 20a, 30a Inference unit, 11 Image input unit, 12 Image encoder, 13, 33 Knowledge encoder, 14 Feature synthesis unit, 15 Alignment unit, 16, 26, 36 Output text decoder, 17 Text output unit, 21, 41 Text input unit, 42 Input text encoder, 43 Knowledge encoder information storage unit, 44 Knowledge encoder selection unit, 110, 120, 210, 310 Learning device, 110a, 120a, 210a, 310a Model generation unit, 111 Text correct answer data storage unit, 112 Output text loss function calculation unit (optimization unit), 113 Knowledge correct answer data storage unit, 114 Knowledge loss function calculation unit (optimization unit), 121 Text / knowledge correct answer data storage unit, 122 Text / knowledge loss function calculation unit (optimization unit), 211 text token correct answer data storage unit, 212 text loss function calculation unit (optimization unit), 311 knowledge information correct answer data storage unit, 312 text correct answer data storage unit, 313 output text loss function calculation unit (optimization unit), 314 knowledge loss function calculation unit (optimization unit), D electronic data, I image data, Fi image feature, Fk knowledge feature, Fc composite feature, Fa alignment feature, Tt output text token, Tx text, Tti input text token, Sk knowledge information, Ft input text feature, Se knowledge encoder selection information, D electronic data for learning, Lt, Lti text correct answer data, Lk knowledge correct answer data, Ltk text / knowledge correct answer data, Ltt text token correct answer data, Et text error, Ett Text token error, Ek knowledge error, Etk text-knowledge error.

Claims

1. An image language processing device that generates text that explains a target image using an image language model, comprising: an image encoder that generates image features from image data of the target image; a knowledge encoder that generates knowledge features from the image features; a feature synthesis unit that generates synthesized features from the image features and the knowledge features; an alignment unit that generates alignment features from the synthesized features; an output text decoder that generates output text tokens from the alignment features; and a text output unit that generates the text from the output text tokens.

2. The image language processing device according to claim 1, characterized in that the knowledge encoder generates the knowledge feature quantity containing information indicating the position and size of one or more predefined objects.

3. The image language processing device according to claim 1 or 2, characterized in that the knowledge encoder generates the knowledge feature amount including information indicating the relationship between one or more predefined objects and other objects.

4. The image language processing device according to any one of claims 1 to 3, characterized in that the knowledge encoder generates the knowledge feature including one or more pieces of predefined pixel attribute information.

5. The image language processing device according to any one of claims 1 to 4, characterized in that the knowledge encoder generates the knowledge feature including one or more pieces of predefined image attribute information.

6. The image language processing device according to any one of claims 1 to 5, characterized in that the knowledge encoder generates the knowledge feature quantity including at least distance information.

7. The image language processing device according to any one of claims 1 to 6, characterized in that the knowledge encoder generates the knowledge feature quantity including at least abnormality information.

8. An image language processing device according to any one of claims 1 to 7, characterized in that the knowledge encoder generates the knowledge feature amount including at least information describing the image in natural language.

9. An image language processing device according to any one of claims 1 to 8, further comprising a text input unit that generates input text tokens from input text, wherein the output text decoder generates the output text tokens from the alignment features and the input text tokens.

10. An image language processing device as claimed in any one of claims 1 to 8, comprising: a text input unit that generates input text tokens from input text; an input text encoder that generates input text features from the input text tokens; a knowledge encoder information storage unit that stores knowledge information; and a knowledge encoder selection unit that generates knowledge encoder selection information from the input text features and the knowledge information, wherein the knowledge encoder generates the knowledge features from the image features and the knowledge encoder selection information, and the output text decoder generates the output text tokens from the alignment features and the input text tokens.

11. An image language processing method implemented by an image language processing device that generates text explaining a target image using an image language model, comprising the steps of: generating image features from image data of the target image; generating knowledge features from the image features; generating composite features from the image features and the knowledge features; generating alignment features from the composite features; generating output text tokens from the alignment features; and generating the text from the output text tokens.

12. An image language processing program that causes a computer that generates text explaining a target image using an image language model to execute the following steps: generating image features from image data of the target image; generating knowledge features from the image features; generating composite features from the image features and the knowledge features; generating alignment features from the composite features; generating output text tokens from the alignment features; and generating the text from the output text tokens.

13. A learning device that acquires training data including image data of a target image and correct answer data for the target image, and generates an image language model for inferring text from the image data of the target image using the training data, comprising: an image encoder that generates image features from the image data of the target image; a knowledge encoder that generates knowledge features from the image features; a feature synthesis unit that generates synthesized features from the image features and the knowledge features; an alignment unit that generates alignment features from the synthesized features; an output text decoder that generates output text tokens from the alignment features; and an optimization unit that optimizes the image language model based on the output text tokens and the correct answer data.

14. The learning device according to claim 13, characterized in that the correct answer data includes text correct answer data and knowledge correct answer data, and the optimization unit includes one or both of an output text loss function calculation unit that calculates a loss function of the output text token based on the output text token and the text correct answer data, and a knowledge loss function calculation unit that calculates a loss function of the knowledge feature based on the knowledge feature and the knowledge correct answer data.

15. The learning device according to claim 13, wherein the correct answer data includes text / knowledge correct answer data, and the optimization unit includes a text / knowledge loss function calculation unit that calculates a loss function of the output text token based on the output text token and the text / knowledge correct answer data.

16. A learning method implemented by a learning device that acquires training data including image data of a target image and correct answer data for the target image, and generates an image language model for inferring text from the image data of the target image using the training data, the learning method comprising: a step of generating image features from the image data of the target image; a step of generating knowledge features from the image features; a step of generating composite features from the image features and the knowledge features; a step of generating alignment features from the composite features; a step of generating output text tokens from the alignment features; and a step of optimizing the image language model based on the output text tokens and the correct answer data.

17. A learning program that causes a computer to acquire training data including image data of a target image and correct answer data for the target image, and generate an image language model for inferring text from the image data of the target image using the training data, to execute the following steps: generating image features from the image data of the target image; generating knowledge features from the image features; generating composite features from the image features and the knowledge features; generating alignment features from the composite features; generating output text tokens from the alignment features; and optimizing the image language model based on the output text tokens and the correct answer data.

Images