Learning device, learning method and program

Abstract

This training device includes: a first acquisition unit that uses a first model to acquire feature amounts for a plurality of pieces of text; a second acquisition unit that uses a second model to acquire, for each of the pieces of text, a feature amount of a first image which is a positive example regarding the relevancy of the piece of text, and a feature amount of a second image in which is embedded a character string not included in the piece of text with respect to the first image; and a training unit that calculates a loss on the basis of the feature amount of the text, the feature amount of the first image, and the feature amount of the second image, and that updates parameters of the first model and the second model on the basis of the loss. The training device thereby reduces the effect due to an irrelevant character string embedded in an image, for a text- and image-embedding model.

Description

[Technical Field]

[0001] The present invention relates to a learning device, a learning method, and a program. [Background technology]

[0002] Technologies that embed image and text features in the same space and allow computers to understand the images and text based on those features are becoming widespread. Specifically, there are technologies that search for images from text or text from images by measuring the distance between the image and text in that space and using that distance as a search score (Non-Patent Document 1, Non-Patent Document 2).

[0003] These techniques use image recognition models and language models to create embedding maps, and then use these models to train neural network parameters using large datasets consisting of pairs of related images and text. This makes it possible to embed related images and text in close proximity in the same space, and has been highly evaluated in the fields of vision and language, which utilize visual and text information. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] CLIP, [online], Internet<URL:https: / / arxiv.org / abs / 2103.00020,https: / / openai.com / blog / clip / > [Non-patent document 2] Siqi Sun, Yen-Chun Chen, Linjie Li, Shuohang Wang, Yuwei Fang, and Liu, Jingjing, "LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval", Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 982-997, 2021 Summary of the Invention [Problem to be solved by the invention]

[0005] However, it has been reported that, among conventional models, particularly the model in Non-Patent Document 1, the accuracy of recognizing characters written in images is so high that when unrelated characters (characters not included in the text associated as correct answers with the training data images given during learning) are intentionally embedded in an image, the model may react strongly to the unrelated characters and fail to correctly recognize the visual information in the image. This phenomenon, which results vary depending on the size and position of the characters in the image, suggests the possibility that desired visual information cannot be obtained not only from malicious character embedding but also from naturally occurring character information such as corporate logos and signs, and this misidentification could pose a problem in practical operation.

[0006] For example, in an image search task using text as a query, if the image contains characters unrelated to the query, the system may react strongly to the unrelated characters and fail to correctly recognize the visual information in the image.

[0007] The present invention has been made in view of the above points, and aims to provide a text and image embedding model that reduces the influence of irrelevant character strings embedded in an image. [Means for solving the problem]

[0008] To solve the above problem, the learning device includes a first acquisition unit that acquires features of multiple texts using a first model; a second acquisition unit that acquires, for each of the texts, features of a first image that is a positive example for the relevance of the text and features of a second image in which a character string not contained in the text is embedded in the first image using a second model; and a learning unit that calculates a loss based on the features of the texts, the features of the first image, and the features of the second image, and updates parameters of the first model and the second model based on the loss. [Effects of the Invention]

[0009] For text and image embedding models, the influence of irrelevant strings embedded in images can be reduced. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an example of a hardware configuration of a search device 10 according to an embodiment of the present invention. [Figure 2] 1 is a diagram illustrating an example of a functional configuration of a search device 10 according to an embodiment of the present invention. [Figure 3] 10 is a flowchart illustrating an example of a processing procedure for learning a model parameter. [Figure 4] FIG. 10 is a diagram for explaining a method for calculating output values ​​and losses of the softmax function for text and images for positive examples and negative examples, respectively. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In this embodiment, a search device 10 that executes a search task will be described, taking as an example a search task in which, when a search target image is given, images related to a search text query are extracted.

[0012] Fig. 1 is a diagram showing an example of the hardware configuration of a search device 10 according to an embodiment of the present invention. The search device 10 in Fig. 1 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a processor 104, and an interface device 105, which are all interconnected via a bus B.

[0013] A program that realizes processing in the search device 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program does not necessarily have to be installed from the recording medium 101, but may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program as well as necessary files, data, etc.

[0014] When an instruction to start a program is received, the memory device 103 reads and stores the program from the auxiliary storage device 102. The processor 104 is a CPU or a GPU (Graphics Processing Unit), or a CPU and a GPU, and executes functions related to the search device 10 in accordance with the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.

[0015] Fig. 2 is a diagram showing an example of the functional configuration of the search device 10 according to the embodiment of the present invention. In Fig. 2, the search device 10 includes a search unit 11 and a model learning unit 12. These units are realized by processing in which one or more programs installed in the search device 10 are executed by a processor 104.

[0016] The search unit 11 executes a search task. The input to the search unit 11 is a search query Q, a set of images to be searched {I0, I1, ..., I mThe output from the search unit 11 is an ordered set {I0, I1, ..., I k} and the relevance of each related image to Q {S0,S1,...,S k} where m is the number of images to be searched, and k is the number of related images obtained by the search.

[0017] The search unit 11 includes a context encoding unit 111, an image encoding unit 112, and a ranking unit 113. The context encoding unit 111 and the image encoding unit 112 are implemented as neural networks. All computational processes in the neural networks are performed based on learned parameters corresponding to each model (neural network).

[0018] The context encoding unit 111 receives input of a character string constituting an arbitrary sentence as a search query Q, and outputs (generates) a feature quantity u of the search query based on parameters of a trained model serving as the context encoding unit 111. The specific neural network model serving as the context encoding unit 111 is not limited to a specific one as long as it encodes text information. For example, the text encoder model used in Non-Patent Document 1 may be used. This model inputs text and outputs d-dimensional feature quantities, but other models may also be used as long as they are context-aware pre-trained models using a transformer.

[0019] The image encoding unit 112 encodes each search target image I i is input, and the search target image I is calculated based on the parameters of the trained model as the image encoding unit 112. i The feature value v iwhere i = 0, 1, ..., m. The specific neural network model used as the image encoding unit 112 is not limited to a specific model as long as it receives an image as input and outputs a d-dimensional vector. However, the dimension d of the output vectors of the context encoding unit 111 and the image encoding unit 112 must match. For example, the image encoder model used in Non-Patent Document 1 may be used. Non-Patent Document 1 provides ResNet and ViT as models that input an image and output d-dimensional features. These may be used, or other models may be used.

[0020] The ranking unit 113 compares the feature value u output from the context encoding unit 111 with respect to the search query Q with each search target image I i The feature value v output from the image encoding unit 112 i (hereinafter referred to as the "feature set") {v1,...,v i ,…,v m} and the ordered set of relevant images for the search query Q, {I1,…,I i ,…,I k} and the relevance of each related image to Q {S1,...,S i ,…,S k} is output.

[0021] A Certain Image I i and the relevance S of the search query Q i is calculated by using an appropriate distance function f. i =f(u,v i ) As a concrete implementation example, f is calculated by i The input of f is two vectors of the same dimension, and the output is a scalar. Alternatively, a distance function other than the reciprocal of the dot product that can measure the distance between vectors may be used as f.

[0022] The model learning unit 12 learns the model parameters of the context encoding unit 111 and the model learning unit 12 .

[0023] As a preparation for learning, training data for the search task is collected in advance. For example, the data collected in "Thomee, B., Shamma, D.A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., and Li, L.-J. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64-73, 2016" is used. The training data consists of a text set T = {T0, T1, ..., T_c} and an image set I = {I0, I1, ..., I m '}.

[0024] Furthermore, text T i Related to (text T i Image set I (assuming it is a positive example in terms of its relevance to i ={I j |I j :T i For these pre-collected datasets, each T i Image set I related to i Randomly extract one image (positive example image) from I i + As, (T i ,I i + ) is used as a set of training data. In other words, the training data prepared in advance is a set of text and positive example images.

[0025] The model learning unit 12 updates the model parameters of the context encoding unit 111 and the image encoding unit 112 through supervised learning using such learning data. Note that the model parameters of the context encoding unit 111 and the image encoding unit 112 are assumed to be initialized in advance with appropriate initial values ​​(when the model structure of Non-Patent Document 1 is used, parameters of an existing trained model may be used as the initialization parameters).

[0026] The model learning unit 12 updates the model parameters based on all the learning data and repeats this any number of times (this repetitive process is called an "epoch", and the number of repetitions is called the "epoch number"). The parameter updating method may be the same as that used in general neural network learning.

[0027] The following describes the processing procedure for model learning executed by the search device 10. Fig. 3 is a flowchart for explaining an example of the processing procedure for learning model parameters.

[0028] In step S101, the model learning unit 12 randomly divides a plurality of pieces of learning data into a plurality of mini-batches.

[0029] Next, the model learning unit 12 executes a loop process L1 for each mini-batch. The mini-batch being processed in the loop process L1 is referred to as the "target batch." Within the loop process L1, the model learning unit 12 executes steps S102 to S109 for the target batch.

[0030] In step S102, the text T i is input to the context encoding unit 111, the vector (feature amount u i ) is obtained for each training data. For example, if the mini-batch size is 10, 10 features u i is obtained.

[0031] Next, the model learning unit 12 calculates the image I of each learning data included in the target batch. i + is input to the image encoding unit 112, the vector (feature value v i + ) is obtained for each training data (S103). For example, if the mini-batch size is 10, 10 features v i + is obtained.

[0032] Next, for each image of the learning data included in the target batch, the model learning unit 12 calculates an arbitrary text T - ∈T'-{T i}, T - is included in and T i A string s that is any noun not contained in k (S104). Here, for a certain training data, T i is the correct text. Therefore, for the training data, T'-{T i} is the set of texts in the target batch excluding the correct text. T - is one text in the set. - is one text other than the correct text (unrelated to the correct image) in the text set in the target batch. That is, T - is included in and T i A string s that is any noun not contained in k is a noun that is not included in the correct answer text.

[0033] For example, image I of a certain training data i + (correct image) is an image of a dog, and the text T i is "A dog is running around in the park", and the set of texts in a mini-batch of size 10 is T'. (1) The dog is running around in the park (2) The cat is lying down in the garden (3) : (10) In this case, T'-{T i} is the nine texts (2) to (10) of T' above. T - is a single piece of text (e.g., "The cat is lying down in the garden") arbitrarily (randomly) selected from (2) to (10). k For example, s is a noun that is included in "A cat is lying down in the garden" but not included in "A dog is running around in the park."k As for T - Instead of being selected from the entire vocabulary set (a vocabulary set not limited to the training data), i The noun may be selected from among nouns not included in

[0034] Note that the string s k The method of selecting can be random. Alternatively, word vectors can be calculated in advance using FastText, and the word with the furthest average distance from the words in the correct text in the space of the word vectors can be selected as the string s. k It may be possible to select it as follows.

[0035] Next, for each learning data of the target batch, the model learning unit 12 calculates the image I i + The string s related to the training data is generated in a random font size that does not exceed the image size of k I i + By embedding (superimposing) the negative example image on the replica of I ik - (S105). Several candidates for font size may be given in advance, or a minimum size may be defined and then the font size may be determined continuously and randomly within that range. i - Multiple I may be generated for one training data set, or only one. i - If a string s is generated, k It is sufficient that multiple values ​​are selected for one training data.

[0036] Next, the model learning unit 12 calculates the image I associated with each learning data of the target batch. ik - are input to the image encoding unit 112, and the image encoding unit 112 encodes each I ik - The feature value v generated for i k - is obtained (S106).

[0037] At this point, the target batch contains: A set of text features: {u1,u2,…,u b} (where b is the mini-batch size) Image feature set: {v1 + ,v2 + ,…,v b + ,v 11 - ,…,v bl -} (where l is the number of negative example images) This becomes:

[0038] Next, the model learning unit 12 calculates the output value of a softmax function of the inner product of text to image and image to text for each of the positive examples and negative examples (S107).

[0039] This calculation method will be explained using a diagram. Figure 4 is a diagram for explaining the calculation method of the output value and loss of the softmax function for text and images for positive and negative examples. Figure 4 shows the set of text features {u1, u2, ..., u b} are arranged in the column direction, and the image feature set {v1 + ,v2 + ,…,v b + ,v 11 - ,…,v bl -} are arranged in the row direction.

[0040] The model learning unit 12 calculates the feature value v of an image for text from the image. i and the set of text features {u1,u2,…,u b} one feature u j The dot product is calculated as follows: x i j=v i ·u j The inner product is used for each feature u i By calculating x i1 ,x i2 ,…,x ib The model learning unit 12 obtains x i1 ,x i2 ,…,x ib Based on this, the output value of the softmax function (hereinafter referred to as the "softmax output value") is calculated for each inner product of one row in Figure 4. This is used to calculate the feature set of the positive example image {v1 + ,v2 + ,…,v b +}, the softmax output value corresponding to the inner product of each row of the positive examples in Figure 4 is calculated.

[0041] The model learning unit 12 also calculates the feature value uj of a certain text and the feature value set {v1 + ,v2 + ,…,v b + ,v 11 - ,…,v bl -} and the softmax output value are calculated in the same way. b}, the softmax output value (text to image) corresponding to the dot product of all columns in Figure 4 is calculated.

[0042] Next, the model learning unit 12 calculates the loss (softmax cross entropy loss) for each row and each column of the positive examples in FIG. 4 based on the set of calculated softmax output values, and calculates the average or sum of these as the loss for the target batch (S108). In calculating the loss, the class label (correct answer label) for the softmax output value from image to text (row direction in FIG. 4) is set to 1 when i=j, as shown in FIG. 4, and is set to 0 otherwise. Also, the class label (correct answer label) for the softmax output value from text to image (column direction in FIG. 4) is set to v j + Only the part where i=j is set to 1.

[0043] Specifically, the cross-entropy loss function is H(p,q)=-Σp(x)logq(x) Here, p(x) is the true distribution and q(x) is the predictive distribution. The class labels are applied to this true distribution, and the softmax output value is applied to the predictive distribution to calculate the loss for each row or column of positive examples. Note that the average or sum of the row-wise losses is the image loss, and the average or sum of the column-wise losses is the text loss. The average or sum of the image loss and text loss is taken as the loss for the target batch.

[0044] Note that when using an existing trained model as the initial parameters of the model, the loss may be calculated using only the softmax output value from text to image.

[0045] Next, the model learning unit 12 updates the model parameters of the context encoding unit 111 and the image encoding unit 112 based on the loss in the target model (S108). Specifically, the model learning unit 12 calculates the gradient of each model parameter from the loss using backpropagation, and updates the model parameters using any optimization method.

[0046] When loop L1 has been executed for all mini-batches, model learning unit 12 determines whether a predetermined termination condition has been met (S110). If the termination condition has not been met (No in S110), model learning unit 12 repeats step S101 and subsequent steps. If the termination condition has been met (Yes in S110), model learning unit 12 ends the processing of FIG. 3.

[0047] As described above, according to this embodiment, the text and image embedding models are trained using images embedded with character strings that are negative examples (irrelevant) in terms of their relevance to the text. Therefore, the influence of irrelevant character strings embedded in images on the text and image embedding models can be reduced. As a result, it is possible to train models that are robust against, for example, adversarial character embedding attacks.

[0048] However, if training is too focused on reducing the influence of characters in an image, a model that is less affected by necessary information will be trained. Therefore, in this embodiment, training is also performed on positive examples. This makes it possible to reduce the influence of unrelated characters while maintaining the ability to recognize naturally embedded characters.

[0049] The following additional notes are provided regarding the above-described embodiments.

[0050] (Additional note 1) Memory and at least one processor coupled to said memory; Including, The processor: Obtaining features of a plurality of texts using a first model; For each of the texts, a feature amount of a first image that is a positive example for the relevance of the text and a feature amount of a second image in which a character string not included in the text is embedded in the first image are acquired using a second model; calculating a loss based on the feature amount of the text, the feature amount of the first image, and the feature amount of the second image; and updating parameters of the first model and the second model based on the loss; A learning device characterized by:

[0051] (Additional note 2) Obtaining features of a plurality of texts using a first model; For each of the texts, a feature amount of a first image that is a positive example for the relevance of the text and a feature amount of a second image in which a character string not included in the text is embedded in the first image are acquired using a second model; calculating a loss based on the feature amount of the text, the feature amount of the first image, and the feature amount of the second image; and updating parameters of the first model and the second model based on the loss; A recording medium that records a program that causes a computer to execute a process.

[0052] In this embodiment, the search device 10 is an example of a learning device. The model learning unit 12 is an example of a first acquisition unit, a second acquisition unit, and a learning unit.

[0053] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist of the present invention as described in the claims. [Explanation of symbols]

[0054] 10 Search device 11 Search section 12 Model Learning Section 100 Drive device 101 Recording media 102 Auxiliary storage 103 Memory Device 104 processors 105 Interface Device 111 Context encoding part 112 Image Encoding Unit 113 Rankings B Bus

Claims

1. a first acquisition unit that acquires feature quantities of a plurality of texts using a first model; a second acquisition unit that acquires, for each of the texts, a feature amount of a first image that is a positive example for the relevance of the text and a feature amount of a second image in which a character string not included in the text is embedded in the first image, using a second model; a learning unit that calculates a loss based on the feature amount of the text, the feature amount of the first image, and the feature amount of the second image, and updates parameters of the first model and the second model based on the loss; A learning device comprising:

2. the learning unit calculates, for each of the texts, an inner product of a feature of the text, a feature of the first image, and a feature of the second image; calculates, for each of the first images, an inner product of the first image and each of the feature of the text; and calculates the loss based on a cross-entropy loss of an output value of a softmax function of the inner product for each of the texts and a cross-entropy loss of an output value of a softmax function of the inner product for each of the first images.

2. The learning device according to claim 1.

3. the second acquisition unit generates, for each of the texts, the second image in which a character string not included in the text is embedded for the first image that is a positive example in terms of relevance of the text; 3. The learning device according to claim 1 or 2.

4. a first acquisition step of acquiring feature quantities of a plurality of texts using a first model; a second acquisition step of acquiring, for each of the texts, a feature of a first image that is a positive example for the relevance of the text and a feature of a second image in which a character string not included in the text is embedded in the first image, using a second model; a learning procedure for calculating a loss based on the feature amount of the text, the feature amount of the first image, and the feature amount of the second image, and updating parameters of the first model and the second model based on the loss; A learning method characterized in that the above is executed by a computer.

5. a first acquisition step of acquiring feature quantities of a plurality of texts using a first model; a second acquisition step of acquiring, for each of the texts, a feature of a first image that is a positive example for the relevance of the text and a feature of a second image in which a character string not included in the text is embedded in the first image, using a second model; a learning procedure for calculating a loss based on the feature amount of the text, the feature amount of the first image, and the feature amount of the second image, and updating parameters of the first model and the second model based on the loss; A program characterized by causing a computer to execute the above.

Images