Learning device, search device, learning method, and program

Abstract

Provided is a learning device for training a feature extraction model, wherein the feature extraction model receives input of training data including data of a plurality of modalities and outputs a global feature representing the entire input data, as well as one or more mask features based on part features, each representing a part of the input data. The learning device is provided with: a matching error calculation unit for calculating a matching error between modalities of global features of the plurality of modalities, output by the feature extraction model; a mask recognition error calculation unit for calculating a recognition error between the mask features output by the feature extraction model and information relating to parts corresponding to the mask features; and a model parameter optimization unit for updating parameters of the feature extraction model so as to minimize the matching error and the recognition error.

Description

Learning device, search device, learning method, and program

[0001] The present invention relates to cross-modal search techniques.

[0002] Cross-modal search technology is a technology that uses data of one modality (for example, image, video, or text) as a query to obtain data of another modality.

[0003] Cross-modal search makes it possible to obtain text that describes a query image or video, or to easily obtain images or videos described in the query text. Cross-modal search is one of the basic artificial intelligence (AI) technologies targeting media, and is expected to have a wide range of industrial applications.

[0004] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.Yuhao Chen, Guoqing Zhang, Yujiang Lu, ZhenxingWang, Yuhui Zheng, and RuiliWang. TIPCB: A simple but effective part-based convolutional baseline for text-based person search. CoRR, abs / 2105.11628, 2021.K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lfukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information ProcessinYing Zhang and Huchuan Lu. Deep cross-modal projection learning for image-text matching. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.Peike Li, Yunqiu Xu, Yunchao Wei, and Yi Yang. Self-correction for human parsing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In 2019 IEEE / CVF International Conference on Computer Vision (ICCV), 2019.Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lucic, and Cordelia Schmid. Vivit: Avideo vision transformer. In Proceedings of the IEEE / CVF International Conference on Computer Vision(ICCV), pages 6836-6846, October 2021.

[0005] However, conventional techniques have the problem that they are unable to take into account local correlations between data from different modalities when training feature extraction models used in cross-modal search.

[0006] The present invention has been made in view of the above points, and aims to provide a technique for training a feature extraction model in consideration of local correlations between data of different modalities.

[0007] According to the disclosed technology, there is provided a learning device that trains a feature extraction model, wherein the feature extraction model receives training data including data of a plurality of modalities as input, and outputs global features that capture the entire input data and mask features based on one or more part features that capture a portion of the input data, the learning device comprising: a matching error calculation unit that calculates a matching error between modalities of the global features of the plurality of modalities output by the feature extraction model; a mask recognition error calculation unit that calculates a recognition error between the mask features output by the feature extraction model and information related to the part that corresponds to the mask features; and a model parameter optimization unit that updates parameters of the feature extraction model so as to minimize the matching error and the recognition error.

[0008] The disclosed technology provides a technology for enhancing security in an originality guarantee method for distributing electronic rights.

[0009] 1 is a diagram illustrating an example of the configuration of a search device 100. A diagram for explaining an example of the operation of the ... problem. A diagram for explaining the problem. A diagram for explaining an example of the configuration of a learning device 200. A diagram for explaining an example of the configuration of a feature extraction model 10. A diagram for explaining an example of the configuration of a learning unit 220. A diagram for explaining an example of the configuration of a mask recognition error calculation unit 20. A diagram for explaining an example of the configuration of the mask recognition error calculation unit 20. A diagram for explaining a part image extraction unit 23. A diagram for explaining an example of the configuration of the mask recognition error calculation unit 20. A diagram for explaining a part label extraction unit 26. A diagram for explaining the part label extraction unit 26. A diagram for explaining an example of the configuration of a mask recognition error calculation unit 20 (text) that receives mask features of text as input. A diagram for explaining a method of building a gallery. A diagram for illustrating components necessary for extracting global features. A diagram for explaining the detailed configuration of a cross-modal search unit 120. A diagram for explaining an example of the configuration of the feature extraction model 10. A diagram for explaining components necessary for extracting global features. A diagram for explaining an example of the hardware configuration of a device.

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0011] 1 shows an example of the configuration of a search device 100 according to this embodiment. As shown in FIG. 1, the search device 100 includes an input unit 110, a cross-modal search unit 120, an output unit 130, and a storage unit 140.

[0012] The storage unit 140 stores a group of data to be searched. The group of data to be searched is, for example, a group of items. The group of items to be searched may be called a "gallery." The cross-modal search unit 120 includes a feature extraction model. The detailed configuration of the cross-modal search unit 120 will be described later.

[0013] An example of the operation of the search device 100 will be described with reference to Figures 2 to 4. In the example shown in Figure 2, text is input as a query from the input unit 110. The cross-modal search unit 120 uses features extracted from the text by a feature extraction model to search a gallery of images and sorts the images in order of similarity to the query. The output unit 130 outputs the sorted results as search results.

[0014] 3, an image is input as a query from the input unit 110. The cross-modal search unit 120 searches a gallery of texts using features extracted from the image by a feature extraction model, and sorts the texts in order of similarity to the query. The output unit 130 outputs the sorted results as search results.

[0015] 4, text is input as a query from the input unit 110. The cross-modal search unit 120 uses features extracted from the text by a feature extraction model to search a gallery of videos and sorts the videos in order of similarity to the query. The output unit 130 outputs the sorted results as search results.

[0016] Below, we will explain the conventional technology and its problems, and then we will explain in detail the configurations and operations of the learning device 200 and the search device 100 in this embodiment. Note that the technologies described in the non-patent documents listed as conventional technology are publicly known, but their analysis and explanations of the problems are not publicly known.

[0017] (Regarding the Prior Art and Its Issues) A ​​prior art for realizing cross-modal search is disclosed in Non-Patent Document 1. Non-Patent Document 1 discloses a method based on comparing feature quantities obtained by inputting data into a deep learning model defined for each modality.

[0018] The parameters of each model defined for each modality are determined by minimizing an error function that is designed to minimize the feature distance between data representing the same semantic content regardless of modality, and to maximize the feature distance between data representing different semantic content, given the training data.

[0019] In the technique disclosed in Non-Patent Document 1, the feature values ​​used as inputs for error calculation are extracted by taking into consideration the entire input data. In other words, in the method disclosed in Non-Patent Document 1, if there is a local correlation between data, this is not sufficiently taken into consideration during model training.

[0020] The above-mentioned problem will be explained with reference to a specific example shown in Fig. 5. In the example shown in Fig. 5, text describing a person captured in an image shown on the left side of the figure is shown on the right side of the figure. Here, the text contains descriptions of hairstyle and clothing, such as "black shoulder-length hair" and "long grey collar jacket," but the corresponding image area is only a part of the entire image and is not related to the remaining image area.

[0021] As such, there are cases in cross-modal search where there is a strong correlation between only parts of the data. However, as shown in Figure 5, the method disclosed in Non-Patent Document 1, which performs model training by comparing features extracted from the entire data, has the problem of not being able to fully capture the local correlation between the data.

[0022] In relation to the above-mentioned problem, Non-Patent Document 2 discloses a technology for performing learning by extracting multiple part features that capture only a portion of the input data for each modality in addition to global features that are extracted taking into account the entire input data.

[0023] A specific example of this technology will be described with reference to Fig. 6. In the example of Fig. 6, the person captured in the image shown on the left side of the figure is described in text on the right side of the figure. As shown in Fig. 6, global features and multiple part features are extracted for each of the image and text, and an error function similar to that in Non-Patent Document 1 is defined between the global features and between the part features, and model parameters are optimized to minimize the error defined by their sum.

[0024] This method is expected to work well if highly correlated parts between different modalities can be referenced at the time of model training. However, in many cases, information indicating local correlations between data is not provided at the time of model training, and if such information were to be provided manually, the cost would be significantly high. If information indicating local correlations is not known, implicit correspondence between highly correlated parts would also be performed during model training, but the method disclosed in Non-Patent Document 2 does not include a mechanism to facilitate this correspondence. In other words, even the method disclosed in Non-Patent Document 2 does not achieve model training that fully captures the local correlations between data.

[0025] That is, conventional techniques such as those described in Non-Patent Documents 1 and 2 have a problem in that they do not sufficiently take into account local correlations between data of different modalities when training feature extraction models used in cross-modal search.

[0026] The device configuration and device operation for solving the above problems will be described in detail below.

[0027] 7 shows an example of the configuration of the learning device 200 according to this embodiment. As shown in FIG. 7, the learning device 200 includes an input unit 210, a learning unit 220, an output unit 230, and a storage unit 240.

[0028] Learning data is input from the input unit 210, and the learning data is stored in the memory unit 240. The memory unit 240 may also be called a learning data DB (DataBase). The learning unit 220 holds a feature extraction model. The learning unit 220 uses the learning data read from the memory unit 240 to learn the feature extraction model. More specifically, the feature extraction model is a neural network model, and the parameters of the neural network are optimized through learning. The feature extraction model to be learned may also be considered an element other than the components of the learning device 200.

[0029] The output unit 230 outputs the trained feature extraction model (specifically, optimized parameters). The configuration and operation of the training unit 220 will be described below. Here, characteristic configuration examples of the training unit 220 will be described as first to fifth examples. An overview will be given in the first to fifth examples, followed by a more specific description.

[0030] In the explanation of the first to fifth examples, for convenience, reference will be made to the figures used in the examples described below. While the figures illustrate specific modalities such as images and text, the configurations and operations described in the first to fifth examples are not limited to a specific modality and can be applied to any modality.

[0031] <First Example> In the first example, a description will be given of the feature extraction model 10 included in the learning unit 220. When learning the feature extraction model 10, the learning unit 220 randomly masks a predetermined proportion of one or more part features out of one or more part features output by a backbone corresponding to each modality, and optimizes the parameters of the feature extraction model 10 so that the semantic content corresponding to the masked part features, that is, the masked part features, can be recognized from the remaining part features.

[0032] An example of the configuration of the feature extraction model 10 during learning is shown in Fig. 8. As shown in Fig. 8, the feature extraction model 10 has a backbone section 13, a feature masking section 11, and an attention section 12. The backbone section 13 has an image backbone 14 and a text backbone 15. The example in Fig. 8 shows a case where an image and text are used as data of two modalities.

[0033] The backbone unit 13 outputs global features capturing the whole and one or more part features capturing a part from the input data of each modality.

[0034] The feature masking unit 11 masks a predetermined proportion of part features among one or more part features for each modality, and outputs features corresponding to the masked part features as mask features.

[0035] The attention unit 12 receives part features and mask features for multiple modalities as input, converts the mask features based on the correlation between them, and outputs the converted mask features.

[0036] Fig. 9 shows an example of the configuration of the learning unit 220. Fig. 9 also shows the storage unit 240 (learning data DB).

[0037] As shown in FIG. 9, the learning unit 220 includes a feature extraction model 10, a mask recognition error calculation unit 20, a matching error calculation unit 30, and a model parameter optimization unit 40.

[0038] The matching error calculation unit 30 calculates an error function defined based on a comparison between modalities of the global features output from the feature extraction model 10 to which training data has been input, and obtains a matching error. The mask recognition error calculation unit 20 calculates a recognition error for the part corresponding to the mask feature output from the feature extraction model 10.

[0039] The model parameter optimization unit 40 updates the parameters of the feature extraction model 10 based on the matching error output by the matching error calculation unit 30 and the mask recognition error output by the mask recognition error calculation unit 20 so as to minimize these errors.

[0040] As described above, by simultaneously performing global feature matching between different modalities and recognizing the masked part from the remaining parts in each modality during model training, the remaining parts that are highly correlated with the masked part can be dynamically acquired during model training. In other words, a feature extraction model 10 that captures the local correlation between data can be acquired.

[0041] <Second Example> In the second example (and the third to fifth examples), a more detailed description will be given of the mask recognition error calculation unit 20. As shown in Fig. 10 , the mask recognition error calculation unit 20 in the second example includes a reconstruction vector extraction unit 21 and a reconstruction error calculation unit 22.

[0042] The reconstructed vector extraction unit 21 receives the mask features output from the feature extraction model 10 as input and outputs a reconstructed vector. Note that the part features, mask features, reconstructed vectors, and mask recognition errors shown in Fig. 10 correspond to each other and relate to a certain part in the image. This is also true for other figures.

[0043] The reconstruction error calculation unit 22 receives the part features output from the backbone unit 13 and the reconstruction vectors output from the reconstruction vector extraction unit 21 as inputs, and calculates a mask recognition error, which is the error between the part features and the reconstruction vectors.

[0044] In training the feature extraction model 10, the parameters are updated to minimize errors including this mask recognition error, and the model can be trained so that the part features of the masked image can be reconstructed from the remaining part features, including not only the part features of the image modality but also the part features of the text modality, and the remaining parts that are highly correlated with the masked parts can be dynamically acquired during model training.

[0045] <Third Example> In the third example, we will explain another example of the mask recognition error calculation unit 20. As shown in Fig. 11 , the mask recognition error calculation unit 20 in the third example has a reconstruction vector extraction unit 21, a reconstruction error calculation unit 22, and a part image extraction unit 23.

[0046] The reconstructed vector extraction unit 21 receives the mask feature output from the feature extraction model 10 as an input and outputs a reconstructed vector.

[0047] The part image extraction unit 23 divides the input image into parts corresponding to the part features, and outputs the divided images as part images, as shown in Fig. 12. Note that the part images are an example of part data.

[0048] The reconstruction error calculation unit 22 receives the part image and the reconstruction vector output from the reconstruction vector extraction unit 21 as input, and calculates a mask recognition error, which is the error between the part image and the reconstruction vector.

[0049] In training the feature extraction model 10, the parameters are updated to minimize errors including this mask recognition error, and the model can be trained so that a part image corresponding to the part features of a masked image can be reconstructed from the remaining part features including not only the part features of the image modality but also the part features of the text modality, and the remaining parts that are highly correlated with the masked parts can be dynamically acquired during model training.

[0050] <Fourth Example> In the fourth example, we will explain yet another example of the mask recognition error calculation unit 20. As shown in Fig. 13 , the mask recognition error calculation unit 20 in the fourth example has a classification vector extraction unit 24, a classification error calculation unit 25, and a part label extraction unit 26.

[0051] The classification vector extraction unit 24 receives mask features as input and outputs classification vectors. The part label extraction unit 26 outputs a single label (part label) corresponding to each part feature from the input image, as shown in FIG.

[0052] The classification error calculation unit 25 receives the classification vector and the part label as input and calculates the error between them.

[0053] When training the feature extraction model 10, the parameters are updated to minimize errors including this error, and the model can be trained so that the label corresponding to the part feature of the masked image can be inferred from the remaining part features, including not only the part features of the image modality but also the part features of the text modality, and the remaining parts that are highly correlated with the masked part can be dynamically acquired during model training.

[0054] <Fifth Example> The fifth example will be described mainly focusing on the differences from the fourth example. In the fifth example, the part label extraction unit 26 in the mask recognition error calculation unit 20 in the fourth example outputs an arbitrary number of labels corresponding to each part feature from the input image, as shown in FIG.

[0055] The classification error calculation unit 25 included in the mask recognition error calculation unit 20 calculates the error between the multiple labels and the part features.

[0056] This allows the model to be trained so that multiple labels corresponding to the part features of the masked image can be inferred from the remaining part features, including the text modal, and the remaining parts that are highly correlated with the masked parts can be dynamically acquired during model training.

[0057] Hereinafter, more specific examples of the operation of the learning device 200 and the search device 100 described above will be described as Examples 1 to 3.

[0058] Example 1 In Example 1, as shown in Figure 2, the learning device 100 uses a single text as a query and ranks a group of images in a gallery based on the degree of similarity to the content described in the text. To achieve this search process, it is necessary to train the feature extraction model 10 and build a gallery. Each step of the process will be described in detail below.

[0059] <Learning of Feature Extraction Model 10> The learning unit 220 of the learning device 200 learns (optimizes parameters of) the feature extraction model 10. The configuration of the feature extraction model 10 is as shown in Fig. 8, and the configuration of the learning unit 220 is as shown in Fig. 9.

[0060] The training data used for training the feature extraction model 10 is a data set consisting of a group of images to which a unique ID is assigned for each individual, and a group of texts about the individual, as shown in the training data DB portion of FIG. 9 .

[0061] In the feature extraction model 10 during training shown in Fig. 8, the image backbone unit 14 receives an image as input and outputs one global feature and one or more part features. The text backbone unit 15 receives text as input and outputs one global feature and one or more part features.

[0062] The feature masking unit 11 masks a predetermined proportion of one or more part features and outputs the masked part features as mask features. The attention unit 12 receives the part features and the mask features, converts the mask features based on the correlation between them, and outputs the converted mask features.

[0063] The image backbone unit 14 can be a neural network such as ResNet (Non-Patent Document 3) or ViT (Non-Patent Document 4), and the text backbone unit 15 can be a text encoder such as BERT (Non-Patent Document 5) or CLIP (Non-Patent Document 1).

[0064] In the feature masking unit 11, the mask features converted from the part features and output can be defined as learnable parameters of the same dimension as the part features. Different mask features may be prepared for the image backbone unit 13 and the text backbone unit 15, or the same mask features may be used. In addition, the proportion of one or more part features converted into mask features can be set to any value, and different values ​​or the same value may be used for the image and the text.

[0065] The attention unit 12 can be configured in any manner, and for example, the transformer disclosed in Non-Patent Document 6 can be used.

[0066] The learning process of the feature extraction model 10 described above is executed by the learning unit 220 shown in Fig. 9. Specifically, the process is as follows.

[0067] The matching error calculation unit 20 calculates an error based on matching between the image global features and the text global features output by the feature extraction model 10. The mask recognition error calculation unit 30 calculates an error related to the recognition of the image mask features and the text mask features output by the feature extraction model 10.

[0068] The model parameter optimization unit 40 optimizes the parameters of the feature extraction model 10 so as to simultaneously minimize the matching error and the mask recognition error. Specific examples of each unit are as follows.

[0069] The matching error calculation unit 30 can be realized as an arbitrary error function such that the error decreases as the distance between the image global features and the text global features capturing the same individual decreases, and the error decreases as the distance between the image global features and the text global features capturing different individuals increases. For example, the triplet error or the CMPM error disclosed in Non-Patent Document 7 can be used as the matching error calculation unit 30.

[0070] More specifically, the mask recognition error calculation unit 20 has a functional unit that takes as input the mask features of the text and calculates a mask error for the text, and a functional unit that takes as input the mask features of the image and calculates a mask error for the image. Hereinafter, the functional unit that takes as input the mask features of the text and calculates a mask error for the text will be referred to as the mask recognition error calculation unit 20 (text), and the functional unit that takes as input the mask features of the image and calculates a mask error for the image will be referred to as the mask recognition error calculation unit 20 (image).

[0071] <Mask Recognition Error Calculation Unit 20 (Text)> FIG. 16 shows an example of the configuration of the mask recognition error calculation unit 20 (text) that receives the mask features of text as input.

[0072] 16, the mask recognition error calculation unit 20 (text) has a classification vector extraction unit 28 and a cross-entropy calculation unit 29. The classification vector extraction unit 28 extracts classification vectors from mask features. The cross-entropy calculation unit 29 calculates cross-entropy.

[0073] That is, the mask recognition error can be calculated as an error function value of a classification vector obtained from the mask feature by the classification vector extraction unit 28, with a word or word string corresponding to the part feature being taken as the correct label. In the example of Fig. 16, for example, if "coat" is taken as the correct label, the mask recognition error can be obtained as an error function value between "coat" and the classification vector obtained from the mask feature corresponding to "coat."

[0074] The classification vector extraction unit 28 can be defined, for example, by a fully connected neural network that converts mask features into a vector with the same number of dimensions as the number of words to be classified, and a softmax function. Also, cross-entropy can be used as the error function, as in the example of FIG. 16.

[0075] Below, examples 1 to 3 of the mask recognition error calculation unit 20 (image) will be explained.

[0076] <Mask Recognition Error Calculation Unit 20 (Image): Example 1> In Example 1, the mask recognition error calculation unit 20 (image) is a method of outputting an error value between the part feature input to the feature masking unit 11 of the feature extraction model 10 and the reconstructed value of the mask feature output by the attention unit 12.

[0077] Specifically, the configuration of the mask recognition error calculation unit 20 (image) in Example 1 is as shown in Fig. 10, and includes a reconstruction vector extraction unit 21 and a reconstruction error calculation unit 22. The reconstruction vector extraction unit 21 receives a mask feature corresponding to a part feature as input and outputs a reconstruction vector. The reconstruction error calculation unit 22 calculates a reconstruction error between the reconstruction vector and the part feature.

[0078] For example, a fully connected neural network that aligns the number of dimensions of the mask feature and the part feature can be used as the reconstruction vector extraction unit 21. Furthermore, for the reconstruction error calculation unit 22, an error function that becomes smaller as the mask feature and the part feature become more similar, such as the L1 norm, the L2 norm, or the KL divergence (Kullback-Leibler divergence), can be used.

[0079] <Mask recognition error calculation unit 20 (image): Example 2> In Example 2, the mask recognition error calculation unit 20 (image) outputs an error value between a part image corresponding to a part feature input to the feature masking unit 11 of the feature extraction model 10 and a reconstructed value of the mask feature output by the attention unit 12.

[0080] Specifically, the mask recognition error calculation unit 20 (image) in Example 2 has the configuration shown in FIG. 11, and includes a reconstruction vector extraction unit 21, a reconstruction error calculation unit 22, and a part image extraction unit 23.

[0081] The reconstruction vector extraction unit 21 receives mask features corresponding to part features as input and outputs a reconstruction vector. The part image extraction unit 23 receives an image as input and outputs a part image corresponding to each mask feature. The reconstruction error calculation unit 22 calculates a reconstruction error between the reconstruction vector reconstructed from the mask features and the part image corresponding to the mask feature.

[0082] A fully connected neural network that aligns the number of dimensions between the mask feature and the vector obtained by rearranging the part images into one dimension can be used as the reconstruction vector extraction unit 21. Furthermore, an error function that becomes smaller as the mask feature and the part images become more similar, such as the L1 norm, L2 norm, or KL divergence, can be used as the reconstruction error calculation unit 22.

[0083] <Mask recognition error calculation unit 20 (image): Example 3> A module that estimates a part label corresponding to a part feature input to the feature masking unit 11 of the feature extraction model 10 is added to the mask recognition error calculation unit 20 (image) in Example 3. When a part label is recognized using the mask feature output by the attention unit 12, the mask recognition error calculation unit 20 (image) outputs, as a mask recognition error, an error value between the part label and a correct part label.

[0084] Specifically, the mask recognition error calculation unit 20 (image) in Example 3 has the configuration shown in FIG. 13, and includes a classification vector extraction unit 24, a part label extraction unit 26, and a classification error calculation unit 25.

[0085] The classification vector extraction unit 24 receives mask features as input and outputs a classification vector. The part label extraction unit 26 receives an image as input and extracts labels of part images corresponding to the classification vector. The classification error calculation unit 25 receives the classification vector and part labels as input and calculates the recognition error between them.

[0086] The part label extraction unit 26 may have any configuration, such as the configuration shown in Fig. 14. The configuration shown in Fig. 14 includes a segmentation unit 27 that performs segmentation as disclosed in, for example, Non-Patent Document 8.

[0087] The segmentation unit 27 applies segmentation to the input image, and outputs labels that frequently appear in part images corresponding to mask features as part labels, as shown in FIG.

[0088] The classification vector extraction unit 24 can use a fully connected neural network that converts mask features into a vector with the same number of dimensions as the number of labels to be classified, and a softmax function. The classification error calculation unit 25 can use cross-entropy.

[0089] The processing details of the part label extraction unit 26 are not limited to those shown in FIG. 14 . The processing details of the part label extraction unit 26 may also be those shown in FIG. 15 . In the example shown in FIG. 15 , the segmentation unit 27 outputs all labels that appear in the part image corresponding to the mask features as part labels. In this case, the classification vector extraction unit 24 can be configured with a fully connected neural network that converts the mask features into a vector with the same number of dimensions as the number of labels to be classified, and a sigmoid function. The classification error calculation unit 25 calculates the binary cross entropy for each label.

[0090] <Gallery Construction> Next, construction of a gallery, which is a data group to be searched, will be described. The gallery is constructed by a device (computer) equipped with a trained feature extraction model 10. The device in question may be the learning device 200, the search device 100, or another device.

[0091] A method for constructing a gallery will be described with reference to Fig. 17. As shown in Fig. 17, a feature vector is extracted from each image to be registered in the gallery using a feature extraction model 10 with optimized parameters.

[0092] In this case, it is sufficient to use only the components necessary for extracting global features in the feature extraction model 10, as shown in Fig. 18. As shown in Fig. 17, each element registered in the gallery consists of an ID, features, and attributes that are unique to each image. Any information can be registered as an attribute; for example, if the specific object is a person, the person's name, gender, attributes, etc. can be set.

[0093] <Regarding Search> Search is performed by the search device 100 shown in Fig. 1. Fig. 19 shows a detailed configuration of the cross-modal search unit 120. Fig. 19 also shows that a data group constituting a gallery is stored in the storage unit 140.

[0094] 19, the cross-modal search unit 120 has a trained feature extraction model 10 and a search unit 150. Query text is input to the feature extraction model 10 to extract features. Next, the search unit 150 compares the query features with the features of each element in the gallery to rank the elements in the gallery and output the results.

[0095] When extracting features from a query, it is sufficient to use only the components required to extract global features in the feature extraction model 10, as shown in FIG.

[0096] The search unit 150 may compare query features with gallery features in any manner, for example, by calculating the cosine similarity between the query features and the features of all gallery elements and ranking the elements in the memory unit 140 in order of highest similarity.

[0097] In Example 2, the learning device 100 uses a single image as a query and ranks a group of texts in a gallery based on the similarity to the content of the image, as shown in Fig. 3. To achieve this search process, it is necessary to train the feature extraction model 10 and build a gallery, as in Example 1.

[0098] The feature extraction model 10 can be trained using the same method as in Example 1. Gallery construction and search can be achieved by replacing the input modality in Example 1.

[0099] Example 3 In Example 3, the learning device 100 uses a single text as a query and ranks a group of videos in a gallery based on the degree of similarity to the content described in that text, as shown in Figure 4. To achieve this search process, it is necessary to train the feature extraction model 10 and build a gallery, as in Examples 1 and 2. Below, we will explain the differences from Example 1.

[0100] 20 shows an example of the configuration of the feature extraction model 10 in Example 3. As shown in Fig. 20 , the feature extraction model 10 in Example 3 includes a video backbone unit 16, a text backbone unit 15, a feature masking unit 11, and an attention unit 12.

[0101] The video backbone unit 16 receives a video as input and outputs one global feature and one or more part features. The text backbone unit 15 receives a text as input and outputs one global feature and one or more part features.

[0102] The feature masking unit 11 masks a predetermined proportion of one or more part features and outputs the masked part features as mask features. The attention unit 12 receives the part features and the mask features as input, converts the mask features based on the correlation between them, and outputs the converted mask features.

[0103] The text backbone unit 15, the feature masking unit 11, and the attention unit 12 can be realized in the same manner as in Example 1 (FIG. 8). The video backbone unit 16 can be realized using SlowFast (Non-Patent Document 9) or ViViT (Non-Patent Document 10).

[0104] Alternatively, the image backbone unit 14 of Example 1 ( FIG. 8 ) may be applied to each frame constituting the video in the video backbone unit 16. In this case, for example, the average of the global feature group obtained by the image backbone unit 14 is used as the global feature output by the video backbone unit 16, and the average of the part features extracted from the same position in the frame obtained by the image backbone unit 14 is used as the part feature output by the video backbone unit 16.

[0105] When building a gallery and when searching, only the parts of the entire feature extraction model 10 that are necessary for extracting video global features and text global features are selected and used, as shown in FIG.

[0106] (Hardware Configuration Example) Any of the devices described in this embodiment (search device 100, learning device 200) can be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud.

[0107] That is, the device can be realized by executing a program corresponding to the processing performed by the device using hardware resources such as a CPU (Central Processing Unit) and memory built into a computer. The program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The program can also be provided via a network such as the Internet or email.

[0108] Fig. 22 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 22 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected by a bus BS. The computer may further include a GPU (Graphics Processing Unit).

[0109] The program that realizes the processing on the computer is provided by a recording medium 1001, such as a CD-ROM (Compact Disc Read Only Memory) or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0110] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes functions related to the device in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0111] (Summary, effects, etc. of the embodiment) As explained above, the technology described in this embodiment makes it possible to generate a feature extraction model that takes into account local correlations between data from different modalities, thereby enabling accurate cross-modal searches.

[0112] The following additional notes are provided regarding the above-described embodiments.

[0113] <Addendum> (Addendum 1) A learning device that trains a feature extraction model, wherein the feature extraction model is a model that receives training data including data of a plurality of modalities as input, and outputs global features that capture the entire input data and mask features based on one or more part features that capture a part of the input data, the learning device comprising: a matching error calculation unit that calculates a matching error between modalities of the global features of the plurality of modalities output by the feature extraction model; a mask recognition error calculation unit that calculates a recognition error between the mask features output by the feature extraction model and information related to the part corresponding to the mask features; and a model parameter optimization unit that updates parameters of the feature extraction model so as to minimize the matching error and the recognition error. (Supplementary Item 2) The learning device according to Supplementary Item 1, wherein the feature extraction model comprises: a backbone unit that extracts, for each of the multiple modalities, global features that capture the entirety of input data and one or more part features that capture a part of the input data, a feature masking unit that masks a predetermined proportion of the one or more part features for each of the modalities and outputs the masked part features as mask features, and an attention unit that receives as input one or more part features and one or more mask features for the multiple modalities, transforms each mask feature based on a correlation between the part feature and the mask feature, and outputs the transformed mask feature. (Supplementary Item 3) The learning device according to Supplementary Item 1, wherein the mask recognition error calculation unit comprises: a reconstruction vector extraction unit that receives as input a mask feature and outputs a reconstruction vector, and a reconstruction error calculation unit that receives as input a part feature corresponding to the mask feature and the reconstruction vector and calculates an error. (Supplementary Item 4) The learning device according to Supplementary Item 1, wherein the mask recognition error calculation unit includes: a reconstruction vector extraction unit that receives a mask feature as an input and outputs a reconstruction vector; and a reconstruction error calculation unit that receives part data corresponding to the mask feature and the reconstruction vector as input and calculates an error.(Supplementary Item 5) The learning device according to Supplementary Item 1, wherein the mask recognition error calculation unit comprises: a classification vector extraction unit that receives a mask feature as input and outputs a classification vector, and a classification error calculation unit that receives the classification vector and a label of part data corresponding to the mask feature as input and calculates an error. (Supplementary Item 6) A search device comprising: a memory unit that stores a data set of a first modality to be searched, an input unit that receives a query of a second modality, and a search unit that extracts features from the query using the feature extraction model trained by the learning device according to any one of Supplementary Item 1 to 5, and searches the data set using the features. (Supplementary Item 7) A learning method executed by a learning device that learns a feature extraction model, wherein the feature extraction model receives learning data including data of a plurality of modalities as input, and outputs global features that capture the entire input data and mask features based on one or more part features that capture a portion of the input data, the learning method comprising: a matching error calculation step that calculates a matching error between modalities of the global features of the plurality of modalities output by the feature extraction model, a mask recognition error calculation step that calculates a recognition error between the mask features output by the feature extraction model and information on the part corresponding to the mask features, and a model parameter optimization step that updates parameters of the feature extraction model so as to minimize the matching error and the recognition error. (Supplementary Item 8) A non-transitory storage medium that stores a program for causing a computer to function as each unit in the learning device described in any one of Supplementary Items 1 to 5.

[0114] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0115] This patent application claims priority based on Japanese Patent Application No. 2023-132378, filed on August 15, 2023, the entire contents of which are incorporated herein by reference.

[0116] 10 Feature extraction model 11 Feature masking unit 12 Attention unit 13 Backbone unit 14 Image backbone 15 Text backbone 20 Mask recognition error calculation unit 21 Reconstruction vector extraction unit 22 Reconstruction error calculation unit 23 Part image extraction unit 24 Classification vector extraction unit 25 Classification error calculation unit 26 Part label extraction unit 27 Segmentation unit 28 Classification vector extraction unit 29 Cross-entropy calculation unit 30 Matching error calculation unit 40 Model parameter optimization unit 100 Search device 110 Input unit 120 Cross-modal search unit 130 Output unit 140 Memory unit 150 Search unit 200 Learning device 210 Input unit 220 Learning unit 230 Output unit 240 Memory unit 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims

1. A learning device that trains a feature extraction model, wherein the feature extraction model receives learning data including data of multiple modalities as input, and outputs global features that capture the entire input data and mask features based on one or more part features that capture a portion of the input data, the learning device comprising: a matching error calculation unit that calculates a matching error between modalities of the global features of the multiple modalities output by the feature extraction model; a mask recognition error calculation unit that calculates a recognition error between the mask features output by the feature extraction model and information related to the part corresponding to the mask features; and a model parameter optimization unit that updates parameters of the feature extraction model so as to minimize the matching error and the recognition error.

2. The learning device according to claim 1, wherein the feature extraction model comprises: a backbone unit that extracts, for each of the multiple modalities, global features that capture the entirety of input data and one or more part features that capture a portion of the input data; a feature masking unit that masks a predetermined percentage of the one or more part features for each of the multiple modalities and outputs the masked part features as mask features; and an attention unit that receives as input one or more part features and one or more mask features for the multiple modalities, converts each mask feature based on the correlation between the part feature and the mask feature, and outputs the converted mask feature.

3. The learning device according to claim 1, wherein the mask recognition error calculation unit comprises: a reconstruction vector extraction unit that receives a mask feature as input and outputs a reconstruction vector; and a reconstruction error calculation unit that receives a part feature corresponding to the mask feature and the reconstruction vector as input and calculates an error.

4. The learning device according to claim 1, wherein the mask recognition error calculation unit comprises: a reconstruction vector extraction unit that receives a mask feature as input and outputs a reconstruction vector; and a reconstruction error calculation unit that receives part data corresponding to the mask feature and the reconstruction vector as input and calculates an error.

5. The learning device according to claim 1, wherein the mask recognition error calculation unit comprises: a classification vector extraction unit that receives a mask feature as input and outputs a classification vector; and a classification error calculation unit that receives the classification vector and a label of part data corresponding to the mask feature as input and calculates an error.

6. A search device comprising: a memory unit for storing a data group of a first modality to be searched; an input unit for inputting a query of a second modality; and a search unit for extracting features from the query using the feature extraction model learned by a learning device described in any one of claims 1 to 5, and for searching the data group using the features.

7. A learning method executed by a learning device that trains a feature extraction model, the feature extraction model being a model that receives learning data including data of multiple modalities as input, and outputs global features that capture the entire input data and mask features based on one or more part features that capture a part of the input data, the learning method comprising: a matching error calculation step that calculates a matching error between modalities of the global features of the multiple modalities output by the feature extraction model; a mask recognition error calculation step that calculates a recognition error between the mask features output by the feature extraction model and information related to the part corresponding to the mask features; and a model parameter optimization step that updates parameters of the feature extraction model so as to minimize the matching error and the recognition error.

8. A program for causing a computer to function as each unit in the learning device according to any one of claims 1 to 5.

Images