Information processing device, information processing method, and program

Abstract

It is desirable that technologies be provided to reduce hallucination. [Solution] An information processing device is provided, comprising: a feature fusion unit that calculates a first fused feature based on the features of a first image data and the features and weight parameters of a masked second text data, and a second fused feature based on the features of the masked second image data and the features and weight parameters of the first text data; a loss calculation unit that calculates a loss based on the first fused feature and the second fused feature; and an update unit that updates the weight parameters based on the loss.

Description

[Technical Field]

[0001] This invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Neural networks (hereinafter also referred to as "NN") exhibit high performance in problems such as natural language processing and image recognition. In fields that utilize NNs, self-supervised learning, a type of deep learning, is a learning method that does not require correct answer data for the training data used to train the model, and in recent years has attracted widespread attention, mainly in the fields of image recognition and natural language processing.

[0003] Furthermore, the type of data used by a model (e.g., images or language) is called a modality or modal, and research into models that use multiple modalities is progressing. Models that use multiple modalities are also called multimodal models. The technique proposed in Non-Patent Document 1 is one of the representative techniques for training multimodal models.

[0004] BLIP-2, a technology proposed in Non-Patent Document 1, is a method that aims to improve feature extraction and semantic understanding by jointly learning image data and text data. Furthermore, BLIP-2 aims to improve the integrated understanding of both modalities by promoting the interaction between image data and text data.

[0005] Specifically, BLIP-2 integrates features extracted from image data and features extracted from text data using a fusion module (Q-former), and learns a multimodal representation using the integrated features. Having learned a multimodal representation in this way, BLIP-2 can generate descriptive text for image data and generate answers to image data and corresponding questions. [Prior art documents]

Non-Patent Literature

[0006]

Non-Patent Literature 1

Summary of the Invention

Problems to be Solved by the Invention

[0007] However, the data generated by a model that has learned such multimodal representations may contain plausible lies. This phenomenon where the model outputs plausible lies is also called hallucination. For example, when trying to search for image data paired with text data that matches a search keyword, such hallucination can be a problem.

[0008] For example, assume that the model generates text data saying "wearing a black coat" as the text data paired with image data showing a person wearing a blue coat. In such a case, even if a search is conducted using the search keyword "blue coat", it is conceivable that the image data showing a person in a blue coat will not be retrieved. The possibility that the model may generate a description different from the fact can be a problem.

[0009] Therefore, the present invention has been proposed to solve these problems, and it is desired to provide a technique for reducing hallucination.

Means for Solving the Problems

[0010] To solve the above problems, according to one aspect of the present invention, an image mask determination unit that determines the position of at least a portion of the data in the first image data as an image mask position based on the first image data; a text mask determination unit that determines the position of at least a portion of the data in the first text data as a text mask position based on the first text data paired with the first image data; an image feature extraction unit that extracts features of the first image data based on the first image data and extracts features of the second image data based on the second image data obtained by masking the image mask position of the first image data; and an image feature extraction unit that extracts features of the first text data based on the first text data An information processing device is provided, comprising: a text feature extraction unit that extracts features of text data and extracts features of the second text data based on the second text data obtained by masking the text mask positions of the first text data; a feature fusion unit that calculates a first fusion feature based on the features of the first image data, the features of the second text data, and weight parameters, and calculates a second fusion feature based on the features of the second image data, the features of the first text data, and weight parameters; a loss calculation unit that calculates a loss based on the first fusion feature and the second fusion feature; and an update unit that updates the weight parameters based on the loss.

[0011] The feature fusion unit may calculate a first fused feature based on a first fusion process applied to the features of the first image data and the features of the second text data using the weight parameters, and calculate a second fused feature based on a second fusion process applied to the features of the second image data and the features of the first text data using the weight parameters.

[0012] The first fusion process may include a process for calculating a first autocorrelation of concatenated data obtained by linking the features of the second text data with a portion of the weight parameters, and the second fusion process may include a process for calculating a second autocorrelation of concatenated data obtained by linking the features of the first text data with a portion of the weight parameters.

[0013] The first fusion process may include a process for calculating a first cross-correlation between a portion of the first autocorrelation corresponding to a part of the weight parameters and the features of the first image data, and the second fusion process may include a process for calculating a second cross-correlation between a portion of the second autocorrelation corresponding to a part of the weight parameters and the features of the second image data.

[0014] The loss calculation unit may calculate a first loss based on the predicted data corresponding to the first fusion feature and the ground truth data corresponding to the first text data, calculate a second loss based on the predicted data corresponding to the second fusion feature and the ground truth data corresponding to the first image data, and calculate the loss based on the first loss and the second loss.

[0015] The image feature extraction unit extracts features of the third image data based on the third image data which is not paired with the first text data; the text feature extraction unit extracts features of the third text data based on the third text data which is not paired with the first image data; the loss calculation unit calculates a third loss based on the features of the first image data, the features of the third image data, the features of the first text data, and the features of the third text data; and the loss may be calculated based on the first loss, the second loss, and the third loss.

[0016] The loss calculation unit may calculate the third loss based on a first similarity between the features of the first image data and the features of the first text data, a second similarity between the features of the first image data and the features of the third text data, and a third similarity between the features of the third image data and the features of the first text data.

[0017] The loss calculation unit may calculate the third loss such that the higher the first similarity, the smaller the third loss; the lower the second similarity, the smaller the third loss; and the lower the third similarity, the smaller the third loss.

[0018] The image mask determination unit may, based on its determination that the first text data contains a name that matches or is similar to the class name to which the object associated with the object detection frame attached to the first image data belongs, set a mask probability for the first image data such that the mask probability inside the object detection frame is greater than the mask probability outside the object detection frame, and then determine the image mask position based on the mask probability.

[0019] The text mask determination unit may, based on its determination that the first text data contains a name that matches or is similar to the class name to which the object associated with the object detection frame attached to the first image data belongs, set a mask probability for the first text data such that the mask probability of the name is greater than the mask probability of data other than the name, and determine the text mask position based on the mask probability.

[0020] The image feature extraction unit may be configured using a neural network.

[0021] Furthermore, in order to solve the above problems, according to another aspect of the present invention, the position of at least a portion of the data in the first image data is determined as an image mask position based on the first image data; the position of at least a portion of the data in the first text data is determined as a text mask position based on the first text data paired with the first image data; features of the first image data are extracted based on the first image data, and features of the second image data are extracted based on the second image data obtained by masking the image mask positions of the first image data; and features of the first text data are extracted based on the first text data. A computer-based information processing method is provided, which includes: extracting features of the second text data based on the second text data obtained by masking the text mask positions of the first text data; calculating a first fused feature based on the features of the first image data, the features of the second text data, and weight parameters; calculating a second fused feature based on the features of the second image data, the features of the first text data, and weight parameters; calculating a loss based on the first fused feature and the second fused feature; and updating the weight parameters based on the loss.

[0022] Furthermore, in order to solve the above problems, according to another aspect of the present invention, the computer comprises: an image mask determination unit that determines the position of at least a portion of the data in the first image data as an image mask position based on the first image data; a text mask determination unit that determines the position of at least a portion of the data in the first text data as a text mask position based on the first text data paired with the first image data; an image feature extraction unit that extracts features of the first image data based on the first image data and extracts features of the second image data based on the second image data obtained by masking the image mask positions of the first image data; and the first... A program is provided that functions as follows: a text feature extraction unit that extracts features from text data and extracts features from the second text data based on the second text data obtained by masking the text mask positions of the first text data; a feature fusion unit that calculates a first fused feature based on the features of the first image data, the features of the second text data, and weight parameters, and calculates a second fused feature based on the features of the second image data, the features of the first text data, and weight parameters; a loss calculation unit that calculates a loss based on the first fused feature and the second fused feature; and an update unit that updates the weight parameters based on the loss. [Effects of the Invention]

[0023] As described above, the present invention provides a technology for reducing hallucination. [Brief explanation of the drawing]

[0024] [Figure 1] This figure shows an example of the functional configuration of a learning device 10 according to the first embodiment of the present invention. [Figure 2] This is a diagram to illustrate the general image of MLM. [Figure 3] This is a diagram to illustrate a general concept of MIM (Mechanism of Inspection and Manufacturing). [Figure 4]This is a diagram to explain the concept of mask text loss 123. [Figure 5] This is a diagram to illustrate the image of mask image loss 122. [Figure 6] This flowchart shows an example of the operation of the learning stage performed by the learning device 10 according to the first embodiment of the present invention. [Figure 7] This figure shows an example of a bounding box applied to image data without a mask. [Figure 8] This figure shows examples of multiple captions associated with unmasked image data. [Figure 9] This figure shows an example of image data with a mask. [Figure 10] This figure shows an example of masked text data. [Figure 11] This flowchart shows an example of the operation of the learning stage performed by the learning device 10 according to the second embodiment of the present invention. [Figure 12] This figure shows the hardware configuration of an information processing device 900 as an example of a learning device 10 according to the first embodiment of the present invention. [Modes for carrying out the invention]

[0025] Preferred embodiments of the present invention will be described in detail below with reference to the attached drawings. In this specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0026] (0. Overview of the Embodiment) An overview of embodiments of the present invention will be described.

[0027] As mentioned above, data generated by models that have learned multimodal representations may contain plausible lies. This phenomenon, where a model outputs plausible lies, is also called hallucination. As mentioned above, BLIP-2 is a method that aims to learn from both image data and text data.

[0028] However, BLIP-2 is characterized by not actively utilizing the mechanism by which each modality provides hints to one another. This characteristic may be the reason why BLIP-2 is prone to hallucination.

[0029] The learning method according to the embodiment of the present invention is designed to use image data as supplementary information for text data, and text data as supplementary information for image data. The model generated by this learning method can solve tasks by relating the image data and text data to each other. This allows for correspondence between the parts contained in the image data and text data, and is expected to improve the final task performance.

[0030] More specifically, this specification proposes improvements to the three losses in BLIP-2's pre-training Step 1: ITC (Image Text Contrastive), ITM (Image Text Matching), and ITG (Image Text Generation).

[0031] Here, ITC is designed so that the similarity between features of unpaired image data and text data is low, while the similarity between features of paired image data and text data is high. ITM is the result of predicting whether the image data and text data are paired or not, and is represented as 0 or 1. ITG is the result of predicting the next word in the text data based on the image data and text data.

[0032] In other words, while ITC and ITM reflect whether or not image data and text data are paired, they do not reflect what kind of information each image data and text data contains. Therefore, it is thought that simply using ITC and ITM as losses will not allow the model to learn the correspondence between the parts contained in each image data and text data.

[0033] Furthermore, it is known that text data contributes significantly to the prediction of the next word, and it is thought that ITG does not adequately reflect the information contained in image data.

[0034] Therefore, in this specification, we propose using a loss that reflects the information contained in each image data and text data for training.

[0035] In this embodiment of the present invention, an information processing device for training a neural network (hereinafter also referred to as the "training device") will be described. In the training device, the neural network is trained based on training data (training stage). A trained neural network is generated through the training of the neural network. The trained neural network may correspond to a trained model. Subsequently, the discrimination device outputs test results based on the trained neural network and test data (testing stage).

[0036] In embodiments of the present invention, the case where the learning device and the identification device are implemented by the same computer is primarily assumed. However, the learning device and the identification device may be implemented by different computers. In such cases, the trained neural network (NN) generated by the learning device is provided to the identification device. For example, the trained NN may be provided from the learning device to the identification device via a recording medium or via communication.

[0037] (1. First Embodiment) First, a first embodiment of the present invention will be described.

[0038] (Configuration of the learning device) Referring to Figure 1, an example of the configuration of a learning device according to the first embodiment of the present invention will be described. Figure 1 is a diagram showing an example of the functional configuration of a learning device 10 according to the first embodiment of the present invention. As shown in Figure 1, the learning device 10 according to the first embodiment of the present invention comprises a dataset 100, an input unit 101, a neural network 110, a loss calculation unit 120, and an update unit 131.

[0039] The neural network 110 is composed of neurons and includes an image mask determination unit 113, an image feature extraction unit 114, a text mask determination unit 115, a text feature extraction unit 116, a feature fusion unit 117, a text generation unit 118, weight parameters 111 and 112. In other words, the image mask determination unit 113, the image feature extraction unit 114, the text mask determination unit 115, the text feature extraction unit 116, the feature fusion unit 117 and the text generation unit 118, weight parameters 111 and 112 can constitute the neural network 110.

[0040] In the following explanation, the propagation of data in the forward direction of the neural network 110 will also be simply referred to as "forward propagation." Conversely, the propagation of data in the reverse direction of the neural network 110 (hereinafter also simply referred to as "backward direction") will also be simply referred to as "backpropagation."

[0041] The input unit 101, image mask determination unit 113, image feature extraction unit 114, text mask determination unit 115, text feature extraction unit 116, feature fusion unit 117, text generation unit 118, loss calculation unit 120, and update unit 131, etc., include a computing device such as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit), and their functions can be realized by the computing device loading a program stored in ROM (Read Only Memory) into RAM (Random Access Memory) and executing it.

[0042] At this time, a computer-readable recording medium on which the program is recorded may also be provided. Alternatively, these blocks may be composed of dedicated hardware or a combination of multiple hardware components. Data necessary for calculations by the arithmetic unit is appropriately stored in a storage unit (not shown).

[0043] The dataset 100, weight parameter 111, and weight parameter 112 are stored in a storage unit (not shown). Such a storage unit may consist of memory such as RAM, a hard disk drive, or flash memory.

[0044] (Weight parameter 111) The weight parameter 111 includes weight parameters corresponding to the image mask determination unit 113, the image feature extraction unit 114, the text mask determination unit 115, the text feature extraction unit 116, and the text generation unit 118, respectively. This weight parameter does not need to be updated by the update unit 131. Furthermore, the weight parameter 111 includes weight parameters corresponding to the feature fusion unit 117 that are not updated by the update unit 131. Note that, as representative examples of arrows extending from the weight parameter 111, only the arrows extending from the weight parameter 111 to the image feature extraction unit 114, the arrows extending from the weight parameter 111 to the text feature extraction unit 116, and the arrows extending from the weight parameter 111 to the text generation unit 118 are shown.

[0045] (Weight parameter 112) The weight parameter 112 includes weight parameters that correspond to the feature fusion unit 117 and are updated by the update unit 131. In the initial state, the weight parameter 112 is set to an initial value. For example, the initial value set for the weight parameter 112 may be a random value, but it may be any value. For example, the initial value set for the weight parameter 112 may be a learned value obtained in advance through learning.

[0046] (Dataset 100) Dataset 100 consists of multiple training data sets (hereinafter also referred to as "input data"). The input data are unlabeled data that do not have assigned teacher labels. Each of the input data sets contains paired image data and text data.

[0047] Image data may be still image data. Still image data may be frames extracted from moving image data. Text data may be a descriptive text attached to the image data (hereinafter also referred to as "caption"). Text data may contain multiple captions. Image data and text data belonging to the same pair are the paired image data and text data. On the other hand, image data and text data belonging to different pairs are the unpaired image data and text data.

[0048] (Input section 101) The input unit 101 sequentially acquires input data from the dataset 100. The input unit 101 then outputs the acquired input data to the neural network 110. At this time, the input unit 101 may generate input data in the format required by the neural network 110 based on the acquired input data, and output the input data in the format required by the neural network 110 to the neural network 110.

[0049] For example, if the input unit 101 has input data that includes multiple captions in the text data paired with the image data, it may randomly select one caption from the multiple captions so that only the selected caption is included in the text data paired with the image data.

[0050] The input unit 101 may output a single input data to the neural network 110. However, in the following, we will assume that the input unit 101 outputs N input data (where N is an integer greater than or equal to 1) to the neural network 110 together. In this case, the N input data may include two or more input data where the image data is the same but the text data is different.

[0051] N input data sets include a pair of first image data and first text data. N input data sets may also include a pair of third image data and third text data. The first image data and third image data are not paired. Similarly, the first text data and third text data are not paired.

[0052] Since the image data included in the input data is not masked, the image data included in the input data is also called "unmasked image data." Similarly, since the text data included in the input data is not masked, the text data included in the input data is also called "unmasked text data."

[0053] For example, when mini-batch learning is performed by the learning device 10, the input unit 101 may create a mini-batch of a predetermined size of input data combinations and output the created mini-batch to the neural network 110. In this case, the size of the mini-batch is N, and generally N is an integer of 2 or more. When online learning is performed by the learning device 10, mini-batches do not need to be created, and N is 1.

[0054] (Image mask detection unit 113) The image mask determination unit 113 obtains N unmasked image data from the N input data output from the input unit 101. Based on the obtained N unmasked image data, the image mask determination unit 113 performs a process to determine the position of at least a portion of the data in each of the N unmasked image data as the image mask position. Various methods can be considered for determining the image mask position.

[0055] As an example, the image mask determination unit 113 designates some of the multiple regions (hereinafter, each of these multiple regions is also referred to as a "patch") contained in the unmasked image data as image masks (hereinafter, also referred to as "mask patches"). Mask patches may be designated based on a pre-set mask designation probability.

[0056] For example, if the unmasked image data is 224 x 224 pixels in size and the patch size is 14 x 14 pixels, then the unmasked image data will contain 16 x 16 = 256 patches.

[0057] If a 75% probability of designating each of the 256 patches as a mask is set, then each of the 256 patches will be designated as a mask patch with a 75% probability. In this case, on average, approximately 192 out of the 256 patches will be designated as mask patches.

[0058] Alternatively, a 75% probability of masking all 256 patches may be set. In this case, 192 of the 256 patches will always be designated as mask patches.

[0059] The image mask determination unit 113 outputs N unmasked image data to the image feature extraction unit 114, and for each of the N unmasked image data, it outputs the position of the mask patch in the unmasked image data to the image feature extraction unit 114.

[0060] (Image feature extraction unit 114) The image feature extraction unit 114 acquires N unmasked image data output from the image mask determination unit 113. Then, for each of the N unmasked image data, the image feature extraction unit 114 extracts features of the unmasked image data based on the unmasked image data itself. More specifically, the image feature extraction unit 114 obtains the data output from the image feature extraction NN as features of the unmasked image data by inputting the unmasked image data into the image feature extraction NN and performing forward propagation.

[0061] Features of unmasked image data can be represented by vectors. The process of extracting features from unmasked image data can also be described as the process of encoding (or encoding) the unmasked image data into features.

[0062] The image feature extraction unit 114 may divide the unmasked image data into multiple patches and add position encodings indicating the position of each patch in the unmasked image data before inputting the unmasked image data to the image feature extraction NN. Generally, the shape of the patches may be square, but the shape of the patches does not have to be square. Also, generally, the image data is divided so that adjacent patches do not overlap with each other, but the unmasked image data may be divided so that adjacent patches have overlapping portions.

[0063] Furthermore, the image feature extraction unit 114 obtains the position of the mask patch for each of the N unmasked image data output from the image mask determination unit 113. Then, the image feature extraction unit 114 applies a mask to each of the N unmasked image data at the position of the mask patch. This generates N masked image data. Note that the masked image data generated from the first image data may correspond to the second image data.

[0064] In the following explanation, masked image data will also be referred to as "masked image data."

[0065] The process of masking the location of a mask patch in unmasked image data may be a process of hiding the data at the location of the mask patch in the unmasked image data. The process of hiding the data at the location of a mask patch in unmasked image data may be achieved by removing the patch at the location of the mask patch from multiple patches to which position encoding has been added.

[0066] The image feature extraction unit 114 extracts features from the masked image data based on the masked image data. More specifically, the image feature extraction unit 114 inputs the masked image data into the image feature extraction neural network (NN) and performs forward propagation, thereby obtaining the data output from the NN as features of the masked image data.

[0067] An image feature extraction neural network (NN) may be constructed by including convolutional layers, pooling layers, and activation functions. For example, it is desirable to use a model that is designed to accept masked image data as input for the image feature extraction NN. This model may be a model trained using MIM (Masked Image Modeling). As an example, an existing model such as Transformer's EVA-02 may be used as the image feature extraction NN. Alternatively, a modified model may be generated by adding a linear layer or the like to the end of such an existing model, and this modified model may be used as the image feature extraction NN.

[0068] The image feature extraction unit 114 outputs N combinations of features from unmasked image data and features from masked image data to the feature fusion unit 117.

[0069] (Text mask determination unit 115) The text mask determination unit 115 obtains N unmasked text data from the N input data output from the input unit 101. Based on the obtained N unmasked text data, the text mask determination unit 115 determines the position of at least a portion of the data in each of the N unmasked text data as the text mask position. Various methods can be considered for determining the text mask position.

[0070] As an example, the text mask determination unit 115 designates some of the multiple words (hereinafter, each of these multiple words is also referred to as a "token") contained in the unmasked text data as a text mask (hereinafter, also referred to as a "mask token"). The mask tokens may be designated based on a pre-set mask designation probability.

[0071] For example, if each token is assigned a 50% probability of being designated as a mask token, then each token will be designated as a mask token with a 50% probability. In this case, on average, about half of the tokens in the unmasked text data will be designated as mask tokens. Alternatively, according to the BERT (Bidirectional Encoder Representations from Transformers) model, 80% of the tokens initially designated as mask tokens may ultimately remain mask tokens, 10% may be replaced with arbitrary tokens, and the remaining 10% may be reverted to their original tokens.

[0072] The text mask determination unit 115 outputs N unmasked text data to the text feature extraction unit 116, and for each of the N unmasked text data, it outputs the position of the mask token in the unmasked text data to the text feature extraction unit 116.

[0073] (Text feature extraction unit 116) The text feature extraction unit 116 acquires N unmasked text data output from the text mask determination unit 115. Then, for each of the N unmasked text data, the text feature extraction unit 116 extracts features of the unmasked text data based on the unmasked text data itself. More specifically, the text feature extraction unit 116 obtains the data output from the text feature extraction NN as features of the unmasked text data by inputting the unmasked text data into the text feature extraction NN and performing forward propagation.

[0074] Features of unmasked text data can be represented by vectors. The process of extracting features from unmasked text data can also be described as the process of encoding (or encoding) the unmasked text data into features.

[0075] The text feature extraction unit 116 may, before inputting the unmasked text data to the text feature extraction NN, divide the unmasked text data into multiple tokens and add a position encoding indicating the position of each of the multiple tokens in the unmasked text data. While it is preferable to use the same tokenizer as the one used by the feature fusion unit 117 for dividing the unmasked text data into multiple tokens, any tokenizer may be used. For example, BPE (Byte Pair Encoding) may be used as the tokenizer for dividing the unmasked text data into multiple tokens.

[0076] Furthermore, the text feature extraction unit 116 obtains the positions of the mask tokens in each of the N unmasked text data output from the text mask determination unit 115. Then, the text feature extraction unit 116 applies a mask to each of the N unmasked text data, specifically to the positions of the mask tokens in the unmasked text data. This generates N masked text data. Note that the masked text data generated from the first text data may correspond to the second text data.

[0077] In the following explanation, masked text data will also be referred to as "masked text data."

[0078] The process of masking the position of a mask token in unmasked text data may be the process of hiding the data at the position of the mask token in the unmasked text data. The process of hiding the data at the position of a mask token in unmasked text data may be achieved by replacing the token at the position of the mask token with the mask token among a group of tokens to which positional encoding has been added.

[0079] The text feature extraction unit 116 extracts features of the masked text data based on the masked text data. More specifically, the text feature extraction unit 116 inputs the masked text data into the text feature extraction neural network (NN) and performs forward propagation, thereby obtaining the data output from the NN as features of the masked text data.

[0080] A tokenizer contains a dictionary that associates tokens with their corresponding features. Therefore, a text feature extraction neural network (NN) can extract features from both unmasked and masked text data by using this dictionary to extract the features corresponding to each token. For example, if the dictionary also contains information that associates masked tokens with their features, the NN can use this information to extract the features of masked tokens.

[0081] The text feature extraction unit 116 outputs N combinations of features from the unmasked text data and features from the masked text data to the feature fusion unit 117.

[0082] (Feature Fusion Section 117) The feature fusion unit 117 acquires N combinations of features from unmasked image data and features from masked image data output by the image feature extraction unit 114. The feature fusion unit 117 also acquires N combinations of features from unmasked text data and features from masked text data output by the text feature extraction unit 116.

[0083] The feature fusion unit 117 calculates a first fusion feature (hereinafter also referred to as the "unmasked image-masked text fusion feature") for each of the N unmasked image data, based on the features of the unmasked image data, the features of the masked text data generated from the unmasked text data paired with the unmasked image data, and the weight parameter 112.

[0084] Furthermore, for each of the N unmasked text data, the feature fusion unit 117 calculates a second fusion feature (hereinafter also referred to as the "unmasked text-masked image fusion feature") based on the features of the unmasked text data, the features of the masked image data generated from the unmasked image data paired with the unmasked text data, and the weight parameter 112.

[0085] More specifically, the feature fusion unit 117 can calculate unmasked image-masked text fused features based on a first fusion process of the features of unmasked image data and the features of masked text data generated from unmasked text data paired with the unmasked image data, using weight parameters 112.

[0086] The feature fusion unit 117 can calculate unmasked text-masked image fusion features based on a second fusion process of the features of unmasked text data and the features of masked image data generated from unmasked image data paired with the unmasked text data, using weight parameters 112.

[0087] Here, the first fusion process may include a process to calculate a first autocorrelation of concatenated data obtained by linking the features of the masked text data with a portion of the weight parameter 112. The first fusion process may also include a process to calculate a first cross-correlation between the portion of the first autocorrelation corresponding to a portion of the weight parameter 112 and the features of the unmasked image data. In this way, the features of the unmasked image data and the features of the masked text data can be fused.

[0088] The second fusion process may include a process to calculate a second autocorrelation of concatenated data obtained by linking the features of the unmasked text data with a portion of the weight parameter 112. The second fusion process may also include a process to calculate a second cross-correlation between the portion of the second autocorrelation corresponding to a portion of the weight parameter 112 and the features of the masked image data. In this way, the features of the unmasked text data and the features of the masked image data can be fused.

[0089] The first and second fusion processes can be performed by a common fusion processing module. That is, based on the input of features of unmasked image data, features of masked text data, and a portion of the weight parameters 112 into the fusion processing module, the fusion processing module outputs an unmasked image-masked text fused feature. Similarly, based on the input of features of unmasked text data, features of masked image data, and a portion of the weight parameters 112 into the fusion processing module, the fusion processing module outputs an unmasked text-masked image fused feature.

[0090] The following describes the functions of the fusion processing module. When describing the functions of the fusion processing module, we will not distinguish between the features of unmasked image data and masked image data, and will instead focus on the features of the image data z. i This notation does not distinguish between the features of unmasked text data and the features of masked text data, and the features of the text data z t This is written as, and a portion of the weight parameter 112 is queryed z qIt is denoted as such. Also, without distinguishing between the maskless image - masked text fusion feature and the maskless text - masked image fusion feature output from the fusion processing module, it is also denoted as the fusion feature.

[0091] In the following description, mainly the case where the fusion processing module has one module including a self - attention layer, a cross - attention layer, and a feed - forward layer will be described. However, the fusion processing module may have a plurality of such modules. At this time, for each of the plurality of modules, the feature z of the image data i , the feature z of the text data t and the query z q may be input, and the output from the plurality of modules may be treated as the fusion feature. Among the plurality of modules, there may be a module that does not have a cross - attention layer.

[0092] The self - attention layer concatenates the feature z of the text data t and the query z q to generate concatenated data. The concatenated data can be denoted as concat(z q ,z t ). The self - attention layer has three linear layers, and the three linear layers respectively have weight parameters W Q ,W K ,W V . Also, although each of the three linear layers has a bias, the bias is omitted here.

[0093] The self - attention layer calculates the following formula (1) based on the input x = concat(z q ,z t ) and the weight parameters W Q ,W K ,W V .

[0094]

Equation

[0095] Here, xW Q xW K xW V This is the matrix obtained by passing the input x through each of the three linear layers. It is the numerator of the argument of the softmax function (xW). Q )(xW K ) T This corresponds to autocorrelation. By multiplying the matrices after linear transformation in this way, the problem of the autocorrelation result becoming an identity diagonal matrix is ​​avoided.

[0096] Autocorrelation (xW Q )(xW K ) T By normalizing and applying softmax, attention (= importance or level of attention) can be calculated. Furthermore, this attention can be expressed as xW. V By multiplying by this, an attention is obtained in which the more important elements have larger values. The self-attention layer then uses the attention obtained in this way for query z q The part corresponding to (hereinafter referred to as "query z") q It is also written as ''. ) outputs to the cross-attention layer.

[0097] The cross-attention layer receives queries from the self-attention layer. q ' is obtained, and the image feature extraction unit 114 extracts the image data features z i The cross-attention layer has three linear layers, and each of these three linear layers has a weight parameter W. Q ',W K ',W V It has '. Furthermore, each of these three linear layers has a bias, but the bias is omitted here.

[0098] The cross-attention layer is query z q 'and the features of the image data z i And the weight parameter W Q ',W K ',W V Based on this, calculate equation (2) below.

[0099]

number

[0100] Here, z q 'W Q ' is, z q This is the matrix obtained by passing ' through the first of three linear layers, z i W K ' is, z i This is the matrix obtained by passing through the second of three linear layers, z i W V ' is, z i This is the matrix obtained by passing it through the third of three linear sheaves. (z is the numerator of the argument of the softmax function) q 'W Q ')(z i W K ') T However, the characteristics of the image data and query z q This corresponds to a cross-correlation with '.

[0101] Cross-correlation (z q 'W Q ')(z i W K ') T By normalizing and applying softmax, attention (= importance of the query from the perspective of image data features) can be calculated. Furthermore, if this attention is z i W V By multiplying by ', attention is obtained in which the query elements that are more important to the image data features have larger values. In this way, the image data features z i and query z q A fused attention is obtained. The cross-attention layer outputs the attention obtained in this way to the feedforward layer.

[0102] The feedforward layer is a MultiLayer Perceptron (MLP) and includes two linear layers. Each of these two linear layers has weight parameters and biases. The feedforward layer performs a transformation on the output from the cross-attention layer using the two linear layers to obtain the transformed data. The feedforward layer outputs the transformed data.

[0103] The self-attention layer contains features from unmasked image data, features from masked text data, and a portion of the weight parameter 112 (= query z). q When the input is ) the output from the feedforward layer may be an unmasked image-masked text fusion feature. In this case, the feature fusion unit 117 outputs N unmasked image-masked text fusion features to the loss calculation unit 120.

[0104] Furthermore, the self-attention layer includes features of unmasked text data, features of masked image data, and a portion of the weight parameter 112 (=query z). q When the input is ) the output from the feedforward layer may be unmasked text-masked image fusion features. In this case, the feature fusion unit 117 outputs N unmasked image-masked text fusion features to the loss calculation unit 120. Furthermore, the feature fusion unit 117 outputs N unmasked image data features and N unmasked text data features to the loss calculation unit 120.

[0105] (Loss calculation unit 120) The loss calculation unit 120 obtains N unmasked text-masked image fusion features, N unmasked image-masked text fusion features, N unmasked image data features, and N unmasked text data features from the feature fusion unit 117.

[0106] The loss calculation unit 120 calculates a loss to output to the update unit 131 based on N combinations of unmasked text-to-masked image fusion features corresponding to unmasked text data and unmasked image-to-masked text fusion features corresponding to unmasked image data paired with the unmasked text data.

[0107] More specifically, the loss calculation unit 120 may calculate a masked text loss 123 (first loss) based on the unmasked image-masked text fusion features, and a masked image loss 122 (second loss) based on the unmasked text-masked image fusion features. The loss calculation unit 120 may then calculate the loss to output to the update unit 131 based on the masked text loss 123 and the masked image loss 122. In this case, the loss calculation unit 120 may calculate the loss to output to the update unit 131 by adding the masked text loss 123 and the masked image loss 122.

[0108] Before explaining the images of masked text loss 123 and masked image loss 122, we will first explain the general image of MLM (Masked Language Modeling) with reference to Figure 2, and then explain the general image of MIM (Masked Image Modeling) with reference to Figure 3.

[0109] Figure 2 is a diagram illustrating a typical MLM (Multi-Level Marketing) model. In the example shown in Figure 2, User 31 is the input entity for masked text data T11 to the model, and Robot 32 is the output entity for unmasked text data T12 from the model. Referring to Figure 2, the masked text data T11 is shown as "I am a [MASK]." where [MASK] is the mask token. The unmasked text data T12 is shown as "I am a cat."

[0110] In MLM, the model is trained to reconstruct the features of the unmasked text data T12 from the features of the masked text data T11.

[0111] Figure 3 is a diagram illustrating a typical MIM (Model Inspection and Modeling) concept. In the example shown in Figure 3, user 31 is the input source for masked image data G11 to the model, and robot 32 is the output source for unmasked image data G12 from the model. Referring to Figure 3, the masked image data G11 is shown as image data with hatching applied to a portion of it. The hatched portion is the masked portion. The unmasked image data G12 is shown as image data without any masking applied.

[0112] In MIM, the model is trained to reconstruct the features of the unmasked image data G12 from the features of the masked image data G11.

[0113] Figure 4 is a diagram illustrating the concept of masked text loss 123. In the example shown in Figure 4, user 31 is the input entity for masked text data T11 and unmasked image data G12 to the model, and robot 32 is the output entity for unmasked text data T12 from the model.

[0114] In an embodiment of the present invention, a masked text loss 123 is calculated to reconstruct the features of the unmasked text data T12 from the features of the masked text data T11 and the features of the unmasked image data G12, and the model is trained based on the calculated masked text loss 123.

[0115] As shown in Figure 4, the loss calculation unit 120 may calculate the masked text loss 123 (first loss) based on the predicted data corresponding to the unmasked image-masked text fusion features and the ground truth data corresponding to the unmasked text data. For example, the masked image loss 122 may be calculated by the cross-entropy error between the predicted data corresponding to the unmasked image-masked text fusion features and the ground truth data corresponding to the unmasked text data, but it may also be calculated by other methods.

[0116] The prediction data corresponding to the unmasked image-masked text fusion feature may be the unmasked image-masked text fusion feature itself, or it may be a feature obtained by performing a transformation (e.g., scaling) on ​​the unmasked image-masked text fusion feature.

[0117] The correct data corresponding to unmasked text data may be tokens representing the text mask positions in the unmasked text data. In this case, the predicted data may be data output from a neural network (NN) based on the input of unmasked image-masked text fusion features into a pre-prepared NN.

[0118] Figure 5 is a diagram illustrating the concept of mask image loss 122. In the example shown in Figure 5, user 31 is the input entity for masked image data G11 and unmasked text data T12 to the model, and robot 32 is the output entity for unmasked image data G12 from the model.

[0119] In an embodiment of the present invention, a mask image loss 122 is calculated to reconstruct the features of unmasked image data G12 from the features of masked image data G11 and the features of unmasked text data T12, and the model is trained based on the calculated mask image loss 122.

[0120] As shown in Figure 5, the loss calculation unit 120 may calculate the masked image loss 122 (second loss) based on the predicted data corresponding to the unmasked text-masked image fusion features and the ground truth data corresponding to the unmasked image data. For example, the masked image loss 122 may be calculated by the cosine similarity between the predicted data corresponding to the unmasked text-masked image fusion features and the ground truth data corresponding to the unmasked image data, but it may also be calculated by other methods.

[0121] The prediction data corresponding to the unmasked text-masked image fusion feature may be the unmasked text-masked image fusion feature itself, or it may be a feature obtained by performing a transformation (e.g., scaling) on ​​the unmasked text-masked image fusion feature.

[0122] The ground truth data corresponding to the unmasked image data may be features of the unmasked image data extracted from the unmasked image data by the image feature extraction unit 114. Alternatively, the ground truth data corresponding to the unmasked image data may be features of the unmasked image data extracted from the unmasked image data by an existing model different from the image feature extraction unit 114. In this case, if a large-scale and high-performance existing model is used, distillation learning incorporating the knowledge of the high-performance existing model can be performed.

[0123] Alternatively, the ground truth data corresponding to unmasked image data may be the unmasked image data itself. In this case, the prediction data corresponding to the unmasked text-masked image fusion features may be features that have been expanded to match the ground truth data corresponding to the unmasked image data.

[0124] Alternatively, if there is dictionary data that returns IDs corresponding to features of unmasked image data, the correct data corresponding to unmasked image data may be the IDs returned from the dictionary data corresponding to the features of the unmasked image data. In this case, the prediction data corresponding to the unmasked text-masked image fusion features may be the IDs returned from the dictionary data corresponding to the unmasked text-masked image fusion features.

[0125] The loss calculation unit 120 may calculate a control loss 121 (third loss) based on N × N combinations of features of unmasked image data and features of unmasked text data. The loss calculation unit 120 may then calculate the loss to output to the update unit 131 based on the control loss 121, the masked image loss 122, and the masked text loss 123. In this case, the loss calculation unit 120 may calculate the loss to output to the update unit 131 by adding the control loss 121, the masked image loss 122, and the masked text loss 123.

[0126] The method for calculating the control loss 121 is not limited to a specific method. For example, the loss calculation unit 120 may calculate the similarity between the features of the unmasked image data and the features of the unmasked text data paired with the unmasked image data (hereinafter also referred to as "similarity between the features of the data belonging to the pair"). The similarity between the features of the data belonging to the pair corresponds to the first similarity and there may be N such similarities.

[0127] If N is 2 or greater, the loss calculation unit 120 may calculate the similarity between the features of the unmasked image data and the features of the unmasked text data that are not paired with the unmasked image data (hereinafter also referred to as "similarity between features of data that do not belong to a pair"). The similarity between features of data that do not belong to a pair corresponds to the second or third similarity, and there may be N × (N-1) such similarities.

[0128] The loss calculation unit 120 may calculate the counter loss 121 based on the similarity between the features of the data belonging to the pair and the similarity between the features of the data not belonging to the pair.

[0129] Here, the loss calculation unit 120 may calculate the control loss 121 such that the control loss 121 becomes smaller as the similarity between the features of the data belonging to the pair increases. This is expected to update the weight parameters 112 so that the features of the data belonging to the pair become closer together.

[0130] Furthermore, the loss calculation unit 120 may calculate the control loss 121 such that the control loss 121 becomes smaller the lower the similarity between the features of the data that do not belong to a pair. This is expected to update the weight parameters 112 so that the features of the data that do not belong to a pair are further apart.

[0131] As an example, info NCE (Noise Contrastive Estimation) may be used to calculate the contrast loss 121 based on similarity. The loss calculation unit 120 calculates the loss in the manner described above. When mini-batch learning is performed by the learning device 10, the loss calculation unit 120 calculates the loss in mini-batch units. The loss calculation unit 120 outputs the calculated loss to the update unit 131.

[0132] (Updated part 131) The update unit 131 updates the weight parameters 112 based on the loss calculated by the loss calculation unit 120. For example, the update of the weight parameters 112 is performed using backpropagation.

[0133] More specifically, the update unit 131 updates the weight parameters 112 using backpropagation based on the loss calculated by the loss calculation unit 120. For example, when mini-batch learning is performed by the learning device 10, the update unit 131 updates the weight parameters 112 in mini-batch units.

[0134] The update unit 131 determines whether the learning termination condition has been met each time it finishes updating the weight parameters 112. If it is determined that the learning termination condition has not been met, the input unit 101 acquires the next input data, and the image mask determination unit 113, image feature extraction unit 114, text mask determination unit 115, text feature extraction unit 116, feature fusion unit 117, loss calculation unit 120, and update unit 131 perform processing based on the next input data again. On the other hand, if it is determined that the learning termination condition has been met, learning is terminated.

[0135] The learning termination conditions are not particularly limited and can be any conditions that indicate that learning has been performed to a certain extent. Specifically, the learning termination conditions may include the condition that the loss is less than a threshold. Alternatively, the learning termination conditions may include the condition that the change in loss is less than a threshold (the condition that the loss has converged). Alternatively, the learning termination conditions may include the condition that the weight parameter 112 has been updated a predetermined number of times.

[0136] (Text generation unit 118) The text generation unit 118 generates text data. Once the update unit 131 has finished updating the weight parameters 112, the first stage of model training is completed, and the second stage of model training is performed. The first stage of model training may correspond to the first stage of pre-training of BLIP-2. The second stage of model training may correspond to the second stage of pre-training of the BLIP-2 model. It is mainly assumed that the weight parameters 111 corresponding to the text generation unit 118 are not updated during the first stage of model training. However, the weight parameters corresponding to the text generation unit 118 may be updated during the first stage of model training. In such cases, the weight parameters corresponding to the text generation unit 118 may be included in the weight parameters 112 updated by the update unit 131.

[0137] In the second stage of model training, the feature fusion unit 117 and the text generation unit 118 are connected. The text generation unit 118 generates text data based on the output from the feature fusion unit 117, and the model is trained based on the loss corresponding to the generated text data.

[0138] A trained model can generate highly accurate text data. For example, a trained model can generate descriptive text for image data. Such descriptive text can be used to search for image data that corresponds to a descriptive text containing desired keywords. Furthermore, a trained model can generate answers to both image data and corresponding questions.

[0139] The above describes an example of the configuration of the learning device 10 according to the first embodiment of the present invention.

[0140] (Learning phase actions) Next, with reference to Figure 6, the flow of operations in the learning stage performed by the learning device 10 according to the first embodiment of the present invention will be described. Figure 6 is a flowchart showing an example of operations in the learning stage performed by the learning device 10 according to the first embodiment of the present invention.

[0141] First, the input unit 101 obtains N input data from the dataset 100 (S101). Each of the N input data contains a pair of image data and text data.

[0142] The image mask determination unit 113 performs a process to determine the position of at least some of the data in each of the N unmasked image data as the image mask position (S113). On the other hand, the text mask determination unit 115 performs a process to determine the position of at least some of the data in each of the N unmasked text data as the text mask position (S115).

[0143] The image feature extraction unit 114 extracts features from each of the N unmasked image data (S114-1). The image feature extraction unit 114 generates masked image data by applying a mask to each of the N unmasked image data at the image mask positions. The image feature extraction unit 114 extracts features from each of the N masked image data (S114-2).

[0144] The text feature extraction unit 116 extracts features of unmasked text data for each of the N unmasked image data (S116-1). The text feature extraction unit 116 generates masked text data by applying a mask to the text mask positions for each of the N unmasked text data. The text feature extraction unit 116 extracts features of masked text data for each of the N masked text data (S116-2).

[0145] The feature fusion unit 117 fuses the features of the unmasked image data with the features of the masked text data generated from the unmasked text data paired with the unmasked image data for each of the N unmasked image data, and calculates an unmasked image-masked text fusion feature.

[0146] Furthermore, the feature fusion unit 117 fuses the features of the unmasked text data with the features of the masked image data generated from the unmasked image data paired with the unmasked text data, for each of the N unmasked text data, and calculates an unmasked text-masked image fusion feature (S117).

[0147] The loss calculation unit 120 calculates the masked text loss 123 based on the unmasked image-masked text fusion features, and calculates the masked image loss 122 based on the unmasked text-masked image fusion features.

[0148] Furthermore, the loss calculation unit 120 calculates a comparison loss 121 based on N × N combinations of features of unmasked image data and features of unmasked text data. Then, the loss calculation unit 120 calculates a loss based on the comparison loss 121, the masked image loss 122, and the masked text loss 123 (S120).

[0149] The update unit 131 updates the weight parameters 112 based on the loss calculated by the loss calculation unit 120 (S131). If the learning termination condition is not met (NO in S132), the operation proceeds to S101. On the other hand, if the learning termination condition is met (YES in S132), learning by the neural network 110 is terminated.

[0150] The above describes the flow of operations in the "learning stage" performed by the learning device 10 according to the first embodiment of the present invention.

[0151] (Summary of the first embodiment) As described above, according to the first embodiment of the present invention, a masked image loss 122 and a masked text loss 123 are used in learning based on image data and text data (multimodal pre-training). This is expected to promote cross-referencing of image data and text data (both modalities) and reduce hallucination.

[0152] The first embodiment of the present invention has been described above.

[0153] (2. Second Embodiment) Next, a second embodiment of the present invention will be described.

[0154] (Configuration of the learning device) Referring to Figure 1, an example of the configuration of the learning device 10 according to the second embodiment of the present invention will be described. The learning device 10 according to the second embodiment of the present invention and the learning device 10 according to the first embodiment of the present invention differ in the functions of the image mask determination unit 113 and the text mask determination unit 115. Therefore, in the second embodiment of the present invention, the functions of the image mask determination unit 113 and the text mask determination unit 115 will be mainly described, and a detailed description of the functions of the other components will be omitted.

[0155] In a second embodiment of the present invention, each of the N unmasked image data is provided with a bounding box. The bounding box is merely one example of an object detection frame. Therefore, other object detection frames may be used instead of bounding boxes. For example, each of the N unmasked image data may be provided with an object detection frame based on a segmentation mask.

[0156] Figure 7 shows an example of bounding boxes applied to unmasked image data. Referring to Figure 7, unmasked image data G21 is shown. Unmasked image data G21 shows a cat and a goldfish bowl as examples of objects, and bounding boxes B11 for detecting the cat and B12 for detecting the goldfish bowl are applied.

[0157] Furthermore, in a second embodiment of the present invention, the unmasked image data is associated with one or more captions. The captions may be descriptive texts of the unmasked image data. In a second embodiment of the present invention, if multiple captions are associated with the unmasked image data, one caption is selected from the multiple captions to be paired with the unmasked image data as unmasked text data.

[0158] Figure 8 shows an example of multiple captions associated with unmasked image data. Referring to Figure 8, a caption group C20 containing multiple captions is shown. Specifically, caption group C20 contains five captions (captions C21-C25). The number of captions included in caption group C20 is not limited.

[0159] (Image mask detection unit 113) If there is unmasked image data with multiple bounding boxes, the image mask determination unit 113 selects one bounding box from among the multiple bounding boxes. For example, the image mask determination unit 113 may randomly select one bounding box.

[0160] For example, in the example shown in Figure 7, the image mask determination unit 113 may select bounding box B11, which is the detection frame for a cat, from bounding box B11, which is the detection frame for a goldfish bowl, and bounding box B12, which is the detection frame for a goldfish bowl. A bounding box is associated with the class name to which the object belongs. For example, bounding box B11, which is the detection frame for a cat, is associated with the class name "cat".

[0161] The image mask determination unit 113 obtains the class name to which the object associated with the bounding box belongs. For example, the image mask determination unit 113 obtains the class name "cat" associated with bounding box B11, which is the detection frame for cats. In addition to the class name to which the object belongs, the image mask determination unit 113 may also obtain similar names if similar names are associated with the bounding box.

[0162] The image mask determination unit 113 selects one caption from the multiple captions if there is unmasked image data to which multiple captions are associated. For example, the image mask determination unit 113 may randomly select one caption. For example, in the example shown in Figure 8, the image mask determination unit 113 may select caption C21 from captions C21 to C25. The caption thus narrowed down to one is used as unmasked text data to be paired with the unmasked image data.

[0163] The image mask determination unit 113 determines whether the unmasked text data contains a name that matches or is similar to the class name to which the object associated with the bounding box belongs.

[0164] It is assumed that the image mask determination unit 113 determines that the unmasked text data contains a name that matches or is similar to the class name to which the object associated with the bounding box belongs. In such a case, it is expected that the inside of the bounding box can be easily reconstructed without referring to the inside of the bounding box. Therefore, in such a case, the image mask determination unit 113 sets the mask probability for the unmasked image data such that the mask probability inside the bounding box is greater than the mask probability outside the bounding box.

[0165] In the examples shown in Figures 7 and 8, the class name "cat," which is associated with the bounding box B11 that is the detection frame for cats, is included in the caption C21. Therefore, the image mask determination unit 113 may set the mask probability for the unmasked image data G21 such that the mask probability of patches inside the bounding box B11 is greater than the mask probability of patches outside the bounding box B11.

[0166] The image mask determination unit 113 may then determine the image mask position based on the set mask probability. Once the image mask determination unit 113 determines the image mask position, the image feature extraction unit 114 can apply a mask to the image mask position of the unmasked image data to generate masked image data. Note that a patch that spans from inside to outside the bounding box B11 may be considered as either an inside or outside patch of the bounding box B11.

[0167] Figure 9 shows an example of masked image data. Referring to Figure 9, masked image data G22 is shown. Masked image data G22 is image data obtained by masking unmasked image data G21 when a mask probability of 90% is set for the patches inside the bounding box B11 of unmasked image data G21 and a mask probability of 75% is set for the patches outside the bounding box B11 of unmasked image data G21.

[0168] On the other hand, it is conceivable that the image mask determination unit 113 determines that no name matching or similar to the class name to which the object associated with the bounding box belongs is included in the unmasked text data. In such a case, the image mask determination unit 113 may set the same mask probability (e.g., 75%) for the entire unmasked image data. Then, the image mask determination unit 113 may determine the image mask position based on the set mask probability. Alternatively, in such a case, the image mask determination unit 113 may re-select the bounding box.

[0169] (Text mask determination unit 115) It is assumed that the image mask determination unit 113 determines that the unmasked text data contains a name that matches or is similar to the class name to which the object associated with the bounding box belongs. In such a case, it is expected that the name that matches or is similar to the class name can be easily reconstructed without referring to the name that matches or is similar to the class name. Therefore, in such a case, the text mask determination unit 115 sets the mask probability for the unmasked text data such that the mask probability of the name that matches or is similar to the class name is greater than the mask probability of data other than the name that matches or is similar to the class name.

[0170] In the examples shown in Figures 7 and 8, the class name "cat," which is associated with the bounding box B11 that is the detection frame for cats, is included in the caption C21. Therefore, the text mask determination unit 115 may set the mask probability for the unmasked text data (caption C21) such that the mask probability of tokens matching the class name "cat" is greater than the mask probability of tokens other than those matching the class name "cat."

[0171] The text mask determination unit 115 may then determine the text mask position based on the set mask probability. Once the text mask determination unit 115 determines the text mask position, the text feature extraction unit 116 can generate masked text data by applying a mask to the text mask position in the unmasked text data.

[0172] Figure 10 shows an example of masked text data. Referring to Figure 10, the upper part of the figure shows the unmasked text data "a cat lying on the floor next to a fish bowl". In addition, tokens that match the class name "cat" are set to a mask probability of 100%, and all other tokens are set to a mask probability of 50%.

[0173] The middle section of Figure 10 shows masked text data, where the mask is applied based on the mask probabilities set in this way. [M] indicates a mask token. The bottom section of Figure 10 shows masked text data where, of the tokens set to a mask probability of 50%, 80% are set as mask tokens, 10% are replaced with arbitrary tokens, and 10% are set without a mask.

[0174] On the other hand, it is conceivable that the image mask determination unit 113 determines that no name matching or similar to the class name to which the object associated with the bounding box belongs is included in the unmasked text data. In such a case, the text mask determination unit 115 may set the same mask probability (for example, 50%) for the entire unmasked text data. Then, the text mask determination unit 115 may determine the text mask position based on the set mask probability.

[0175] The above describes an example of the configuration of the learning device 10 according to the second embodiment of the present invention.

[0176] (Learning phase actions) Next, with reference to Figure 11, the flow of operations in the "learning stage" performed by the learning device 10 according to the second embodiment of the present invention will be described. Figure 11 is a flowchart showing an example of operations in the learning stage performed by the learning device 10 according to the second embodiment of the present invention.

[0177] Comparing the example of the learning stage operation performed by the learning device 10 according to the second embodiment of the present invention (Figure 11) with the example of the learning stage operation performed by the learning device 10 according to the first embodiment of the present invention (Figure 6), we find that S101, S114 (S114-1, S114-2), S116 (S116-1, S116-2) to S132 are common, while S113 and S115 are replaced by S202 to S215. Therefore, in the following, we will mainly explain S202 to S215, and omit a detailed explanation of the common operations.

[0178] The image mask determination unit 113 selects one bounding box from the multiple bounding boxes if there is unmasked image data with multiple bounding boxes (S202). The image mask determination unit 113 determines whether the unmasked text data contains a name that matches or is similar to the class name to which the object associated with the bounding box belongs (S203).

[0179] It is assumed that the image mask determination unit 113 determines that no name matching or similar to the class name to which the object associated with the bounding box belongs is included in the unmasked text data (NO in S204). In such a case, the image mask determination unit 113 may return to S202.

[0180] Alternatively, although not shown in the diagram, the image mask determination unit 113 may set the same mask probability for the entire unmasked image data and determine the image mask position based on the set mask probability. Furthermore, the text mask determination unit 115 may set the same mask probability for the entire unmasked text data and determine the text mask position based on the set mask probability.

[0181] On the other hand, it is conceivable that the image mask determination unit 113 determines that the name in the unmasked text data matches or is similar to the class name to which the object associated with the bounding box belongs (YES in S204).

[0182] In such cases, the image mask determination unit 113 sets the mask probability for the unmasked image data such that the mask probability inside the bounding box is greater than the mask probability outside the bounding box. Then, the image mask determination unit 113 determines the image mask position based on the set mask probability (S213). Then, the operation proceeds to S114 (S114-1, S114-2).

[0183] Furthermore, in such cases, the text mask determination unit 115 sets mask probabilities for the unmasked text data such that the mask probability for names that match or are similar to the class name is greater than the mask probability for data other than that name. Then, the text mask determination unit 115 determines the text mask position based on the set mask probabilities (S215). Then, the operation proceeds to S116 (S116-1, S116-2).

[0184] The above describes the flow of operations in the "learning stage" performed by the learning device 10 according to the second embodiment of the present invention.

[0185] (Summary of the second embodiment) As described above, according to the learning device 10 of the second embodiment of the present invention, the mask probability of portions that can be easily reconstructed by referring to the other data is set to a high value in both the image data and the text data. As a result, it is expected that cross-referencing of image data and text data (both modalities) will be further promoted and hallucination will be further reduced.

[0186] The second embodiment of the present invention has been described above.

[0187] (3. Hardware Configuration Example) Next, an example of the hardware configuration of the learning device 10 according to the first embodiment of the present invention will be described. Note that the hardware configuration of the learning device 10 according to the second embodiment of the present invention can also be implemented in the same manner as the hardware configuration of the learning device 10 according to the first embodiment of the present invention.

[0188] In the following, an example of the hardware configuration of the information processing device 900 will be described as an example of the hardware configuration of the learning device 10 according to the first embodiment of the present invention. Note that the example of the hardware configuration of the information processing device 900 described below is merely one example of the hardware configuration of the learning device 10. Therefore, the hardware configuration of the learning device 10 may be modified by removing unnecessary components from the hardware configuration of the information processing device 900 described below, or by adding new components.

[0189] Figure 12 shows the hardware configuration of an information processing device 900 as an example of a learning device 10 according to the first embodiment of the present invention. The information processing device 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, a host bus 904, a bridge 905, an external bus 906, an interface 907, an input device 908, an output device 909, a storage device 910, and a communication device 911.

[0190] The CPU 901 functions as both an arithmetic processing unit and a control unit, controlling the overall operation of the information processing unit 900 according to various programs. The CPU 901 may also be a microprocessor. The ROM 902 stores programs and arithmetic parameters used by the CPU 901. The RAM 903 temporarily stores programs used in the execution of the CPU 901 and parameters that change as needed during its execution. These are interconnected by a host bus 904, which consists of a CPU bus and other components.

[0191] The host bus 904 is connected to an external bus 906, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 905. It is not always necessary to configure the host bus 904, bridge 905, and external bus 906 separately; these functions may be implemented on a single bus.

[0192] The input device 908 consists of input means for the user to input information, such as a mouse, keyboard, touch panel, buttons, microphone, switches, and levers, and an input control circuit that generates input signals based on the user's input and outputs them to the CPU 901. The user operating the information processing device 900 can input various types of data to the information processing device 900 or instruct it to perform processing operations by operating this input device 908.

[0193] The output device 909 includes, for example, display devices such as CRT (Cathode Ray Tube) display devices, liquid crystal display (LCD) devices, OLED (Organic Light Emitting Diode) devices, lamps, and audio output devices such as speakers.

[0194] The storage device 910 is a device for storing data. The storage device 910 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, and a deletion device for deleting data recorded on the storage medium. The storage device 910 is composed of, for example, an HDD (Hard Disk Drive). This storage device 910 drives the hard disk and stores programs executed by the CPU 901 and various data.

[0195] The communication device 911 is a communication interface composed of, for example, a communication device for connecting to a network. The communication device 911 may support either wireless or wired communication.

[0196] The hardware configuration example of the learning device 10 according to the first embodiment of the present invention has been described above.

[0197] (4. Supplement) Although preferred embodiments of the present invention have been described in detail above with reference to the attached drawings, the present invention is not limited to these examples. It is clear to any person with ordinary skill in the art to which the present invention belongs that various modifications or alterations can be conceived within the scope of the technical idea described in the claims, and these are also understood to fall within the technical scope of the present invention. [Explanation of Symbols]

[0198] 10 Learning device 100 datasets 101 Input Section 111, 112 Weight parameters 113 Image mask determination unit 114 Image Feature Extraction Unit 115 Text mask determination unit 116 Text Feature Extraction Unit 117 Feature Fusion Section 118 Text generation unit 120 Loss calculation section 131 Update Department

Claims

1. An image mask determination unit that determines the position of at least a portion of the data in the first image data as an image mask position based on the first image data, A text mask determination unit determines the position of at least a portion of the data in the first text data as a text mask position based on the first text data paired with the first image data, An image feature extraction unit extracts features of the first image data based on the first image data, and extracts features of the second image data based on the second image data obtained by applying a mask to the image mask position of the first image data. A text feature extraction unit extracts features of the first text data based on the first text data, and extracts features of the second text data based on the second text data obtained by masking the text mask positions of the first text data, A feature fusion unit calculates a first fused feature based on the features of the first image data, the features of the second text data, and weight parameters, and calculates a second fused feature based on the features of the second image data, the features of the first text data, and the weight parameters. A loss calculation unit that calculates a loss based on the first fusion feature and the second fusion feature, An update unit updates the weight parameters based on the loss, An information processing device equipped with the following features.

2. The aforementioned feature fusion section is, Using the aforementioned weight parameters, the first fused feature is calculated based on a first fusion process applied to the features of the first image data and the features of the second text data. Using the aforementioned weight parameters, the second fused feature is calculated based on a second fusion process applied to the features of the second image data and the features of the first text data. The information processing apparatus according to claim 1.

3. The first fusion process includes a process for calculating a first autocorrelation of concatenated data obtained by concatenating the features of the second text data with a portion of the weight parameters, The second fusion process includes a process for calculating a second autocorrelation of concatenated data obtained by concatenating the features of the first text data with a portion of the weight parameters. The information processing apparatus according to claim 2.

4. The first fusion process includes a process for calculating a first cross-correlation between a portion of the first autocorrelation corresponding to a part of the weight parameters and the features of the first image data. The second fusion process includes a process for calculating a second cross-correlation between a portion of the second autocorrelation corresponding to a part of the weight parameters and the features of the second image data. The information processing apparatus according to claim 3.

5. The loss calculation unit, Based on the predicted data corresponding to the first fusion feature and the ground truth data corresponding to the first text data, a first loss is calculated. A second loss is calculated based on the predicted data corresponding to the second fusion feature and the ground truth data corresponding to the first image data. The loss is calculated based on the first loss and the second loss. The information processing apparatus according to claim 1.

6. The image feature extraction unit extracts features of the third image data based on the third image data which is not paired with the first text data. The text feature extraction unit extracts features of the third text data based on the third text data which is not paired with the first image data. The loss calculation unit calculates a third loss based on the characteristics of the first image data, the characteristics of the third image data, the characteristics of the first text data, and the characteristics of the third text data, and calculates the loss based on the first loss, the second loss, and the third loss. The information processing apparatus according to claim 5.

7. The loss calculation unit calculates the third loss based on a first similarity between the features of the first image data and the features of the first text data, a second similarity between the features of the first image data and the features of the third text data, and a third similarity between the features of the third image data and the features of the first text data. The information processing apparatus according to claim 6.

8. The loss calculation unit calculates the third loss such that the higher the first similarity, the smaller the third loss; the lower the second similarity, the smaller the third loss; and the lower the third similarity, the smaller the third loss. The information processing apparatus according to claim 7.

9. Based on the determination that the first image data contains a name that matches or is similar to the class name to which the object associated with the object detection frame attached to the first image data belongs, the image mask determination unit sets a mask probability for the first image data such that the mask probability inside the object detection frame is greater than the mask probability outside the object detection frame, and determines the image mask position based on the mask probability. The information processing apparatus according to claim 1.

10. Based on the determination that the text mask determination unit has determined that the first text data contains a name that matches or is similar to the class name to which the object associated with the object detection frame attached to the first image data belongs, the mask probability of the first text data is set such that the mask probability of the name is greater than the mask probability of data other than the name, and the text mask position is determined based on the mask probability. The information processing apparatus according to claim 1.

11. The aforementioned image feature extraction unit is composed of a neural network. The information processing apparatus according to claim 1.

12. Based on the first image data, the position of at least a portion of the data in the first image data is determined as the image mask position, Based on the first text data paired with the first image data, the position of at least a portion of the data in the first text data is determined as the text mask position, Based on the first image data, features of the first image data are extracted, and based on the second image data obtained by applying a mask to the image mask position of the first image data, Based on the first text data, features of the first text data are extracted, and based on the second text data obtained by masking the text mask positions of the first text data, Based on the features of the first image data, the features of the second text data, and the weight parameters, a first fused feature is calculated, and based on the features of the second image data, the features of the first text data, and the weight parameters, a second fused feature is calculated. The loss is calculated based on the first fusion feature and the second fusion feature. Based on the aforementioned loss, the weight parameters are updated, A computer-based information processing method, including [a specific example].

13. Computers An image mask determination unit that determines the position of at least a portion of the data in the first image data as an image mask position based on the first image data, A text mask determination unit determines the position of at least a portion of the data in the first text data as a text mask position based on the first text data paired with the first image data, An image feature extraction unit extracts features of the first image data based on the first image data, and extracts features of the second image data based on the second image data obtained by applying a mask to the image mask position of the first image data. A text feature extraction unit extracts features of the first text data based on the first text data, and extracts features of the second text data based on the second text data obtained by masking the text mask positions of the first text data, A feature fusion unit calculates a first fused feature based on the features of the first image data, the features of the second text data, and weight parameters, and calculates a second fused feature based on the features of the second image data, the features of the first text data, and the weight parameters. A loss calculation unit that calculates a loss based on the first fusion feature and the second fusion feature, An update unit updates the weight parameters based on the loss, A program that makes it function as such.

Images