Learning device, retrieval device, learning method, and program

Abstract

To perform training of a feature extraction model while considering local correlations among data from a plurality of different modalities.SOLUTION: A learning device performs training of a feature extraction model, which is a model which receives learning data including data from a plurality of modalities as input and outputs a global feature capturing the input data as a whole and a mask feature based on one or more part features capturing a portion of the input data. The learning device includes: a matching error calculation unit which calculates a matching error between the modalities of the global features of the plurality of modalities output by the feature extraction model; a mask recognition error calculation unit which calculates a recognition error between the mask feature output by the feature extraction model and information on a part corresponding to the mask feature; and a model parameter optimization unit which updates parameters of the feature extraction model so as to minimize the matching error and the recognition error.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to cross-modal search techniques. [Background technology]

[0002] Cross-modal search technology is a technology that uses data of one modality (e.g., image, video, or text) as a query to retrieve data of another modality.

[0003] Cross-modal search makes it possible, for example, to obtain text that describes a query image or video, or to easily obtain images or videos described in the query text. Cross-modal search is one of the basic AI technologies targeting media, and is thought to have a wide range of industrial applications. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. [Non-Patent Document 2] Yuhao Chen, Guoqing Zhang, Yujiang Lu, ZhenxingWang, Yuhui Zheng, and RuiliWang. TIPCB: A simple but effective part-based convolutional baseline for text-based person search. CoRR , abs / 2105.11628 , 2021 .

Outdoor Tools3

Outdoor Tools 4

Direct Environment 5

[0005] However, conventional techniques have the problem that they are unable to take into account local correlations between data from different modalities when training feature extraction models used in cross-modal search.

[0006] The present invention has been made in consideration of the above points, and has an object to provide a technique for learning a feature extraction model taking into account local correlations between data of different modalities. [Means for solving the problem]

[0007] According to the disclosed technology, there is provided a learning device that learns a feature extraction model, comprising: the feature extraction model is a model that receives learning data including data of a plurality of modalities as input, and outputs a global feature capturing the entire input data, and a mask feature based on one or more part features capturing a part of the input data; a matching error calculation unit that calculates a matching error between modalities of global features of a plurality of modalities output by the feature extraction model; a mask recognition error calculation unit that calculates a recognition error between a mask feature output by the feature extraction model and information on a part corresponding to the mask feature; a model parameter optimization unit that updates parameters of the feature extraction model so as to minimize the matching error and the recognition error; A learning device is provided, comprising: Effect of the Invention

[0008] According to the disclosed technology, a technology for enhancing security in an originality assurance method for distributing electronic rights is provided. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an example of the configuration of a search device 100. [Diagram 2] FIG. 2 is a diagram for explaining an example of the operation of the search device 100. [Diagram 3] FIG. 2 is a diagram for explaining an example of the operation of the search device 100. [Figure 4] FIG. 2 is a diagram for explaining an example of the operation of the search device 100. [Diagram 5] FIG. 1 is a diagram for explaining a problem. [Figure 6] FIG. 1 is a diagram for explaining a problem. [Figure 7] FIG. 2 illustrates an example of the configuration of a learning device 200. [Figure 8] FIG. 2 is a diagram illustrating an example of the configuration of a feature extraction model 10. [Figure 9] FIG. 2 illustrates an example of the configuration of a learning unit 220. [Figure 10] FIG. 2 is a diagram illustrating an example of the configuration of a mask recognition error calculation unit 20. [Figure 11] FIG. 2 is a diagram illustrating an example of the configuration of a mask recognition error calculation unit 20. [Figure 12]FIG. 2 is a diagram for explaining a part image extraction unit 23. [Figure 13] FIG. 2 is a diagram illustrating an example of the configuration of a mask recognition error calculation unit 20. [Figure 14] FIG. 2 is a diagram for explaining a part label extraction unit 26. [Figure 15] FIG. 2 is a diagram for explaining a part label extraction unit 26. [Figure 16] FIG. 2 is a diagram showing an example of the configuration of a mask recognition error calculation unit 20 (text) that receives as input a mask feature of text. [Figure 17] FIG. 13 is a diagram for explaining a method for constructing a gallery. [Figure 18] FIG. 1 shows the components required to extract global features. [Figure 19] FIG. 2 is a diagram showing a detailed configuration of a cross-modal search unit 120. [Figure 20] FIG. 2 is a diagram illustrating an example of the configuration of a feature extraction model 10. [Figure 21] FIG. 1 shows the components required to extract global features. [Figure 22] FIG. 2 illustrates an example of a hardware configuration of the apparatus. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0011] (About cross-modal search) 1 shows an example of the configuration of a search device 100 according to this embodiment. As shown in FIG. 1, the search device 100 includes an input unit 110, a cross-modal search unit 120, an output unit 130, and a storage unit 140.

[0012] The storage unit 140 stores a data group to be searched. The data group to be searched is, for example, an item group. The item group to be searched may be called a "gallery." The cross-modal search unit 120 includes a feature extraction model. A detailed configuration of the cross-modal search unit 120 will be described later.

[0013] An example of the operation of the search device 100 will be described with reference to Fig. 2 to Fig. 4. In the example shown in Fig. 2, text is input as a query from the input unit 110. The cross-modal search unit 120 searches a gallery made up of a group of images using features extracted from the text by a feature extraction model, and sorts the images in order of similarity to the query. The output unit 130 outputs the sorted result as a search result.

[0014] 3, an image is input as a query from the input unit 110. The cross-modal search unit 120 searches a gallery of texts using features extracted from the image by a feature extraction model, and sorts the texts in order of similarity to the query. The output unit 130 outputs the sorted result as a search result.

[0015] 4, text is input as a query from the input unit 110. The cross-modal search unit 120 searches a gallery of videos using features extracted from the text by a feature extraction model, and sorts the videos in order of similarity to the query. The output unit 130 outputs the sorted result as a search result.

[0016] Below, the conventional technology and problems are explained, and then the configurations and operations of the learning device 200 and the search device 100 in this embodiment are explained in detail. Note that the technologies described in the non-patent documents listed as the conventional technology are publicly known, but their analysis and explanations of the problems are not publicly known.

[0017] (Regarding conventional technology and its problems) A conventional technique for realizing cross-modal search is disclosed in Non-Patent Document 1. Non-Patent Document 1 discloses a method based on comparing feature quantities obtained by inputting data into a deep learning model defined for each modality.

[0018] The parameters of each model defined for each modality are determined through minimization of an error function that is designed, based on given training data, to minimize the feature distance between data expressing the same semantic content regardless of modality, and to increase the feature distance between data expressing different semantic content.

[0019] In the technology disclosed in Non-Patent Document 1, the feature amount that is the input for error calculation is extracted by considering the entire input data. In other words, in the method disclosed in Non-Patent Document 1, if there is a local correlation between data, it is not sufficiently taken into consideration during model learning.

[0020] The above-mentioned problem will be described with reference to a specific example shown in Fig. 5. In the example shown in Fig. 5, text describing a person captured in an image shown on the left side of the figure is shown on the right side of the figure. Here, the text includes descriptions of hairstyle and clothing worn, such as "black shoulder-length hair" and "long grey collar jacket", but the corresponding image area is only a part of the entire image and is not related to the remaining image area.

[0021] In this way, there are cases in cross-modal search where there is a strong correlation only between parts of the data. However, as shown in Figure 5, the method disclosed in Non-Patent Document 1, which performs model learning by comparing features extracted from the entire data, has the problem that it cannot fully capture the local correlation between the data.

[0022] In relation to the above-mentioned problem, Non-Patent Document 2 discloses a technology for performing learning by extracting multiple part features that capture only a portion of the input data in each modality in addition to global features that are extracted by taking into account the entire input data.

[0023] A specific example of this technology will be described with reference to Fig. 6. In the example of Fig. 6, the person captured in the image shown on the left side of the figure is described in text on the right side of the figure. As shown in Fig. 6, global features and multiple part features are extracted for each of the image and text, and an error function similar to that in Non-Patent Document 1 is defined between the global features and between the part features, and model parameters are optimized to minimize the error defined by their sum.

[0024] This method is expected to work well if parts with high correlation between different modalities can be referenced at the time of model learning. However, in many cases, information indicating local correlation of data is not given at the time of model learning, and if it is attempted to be given manually, there is a problem that the cost is so high that it cannot be ignored. If information indicating local correlation is not known, implicit correspondence of highly correlated parts will also be performed during model learning, but the method disclosed in Non-Patent Document 2 does not have a mechanism to promote such correspondence. In other words, even the method disclosed in Non-Patent Document 2 does not realize model learning that fully captures the local correlation between data.

[0025] That is, in the conventional techniques such as Non-Patent Documents 1 and 2, there is a problem in that the local correlation between data of different modalities is not sufficiently taken into consideration when training a feature extraction model used in cross-modal search.

[0026] The configuration and operation of the device for solving the above problems will be described in detail below.

[0027] (Configuration of learning device 200) 7 shows an example of the configuration of a learning device 200 according to this embodiment. As shown in FIG. 7, the learning device 200 includes an input unit 210, a learning unit 220, an output unit 230, and a storage unit 240.

[0028] Learning data is input from the input unit 210, and the learning data is stored in the storage unit 240. The storage unit 240 may be called a learning data DB. The learning unit 220 holds a feature extraction model. The learning unit 220 uses the learning data read from the storage unit 240 to learn the feature extraction model. More specifically, the feature extraction model is a neural network model, and parameters of the neural network are optimized by learning. The feature extraction model to be learned may be considered as an element other than the components of the learning device 200.

[0029] The output unit 230 outputs the trained feature extraction model (specifically, optimized parameters). The configuration and operation of the learning unit 220 will be described below. Here, characteristic configuration examples of the learning unit 220 will be described as first to fifth examples. In the first to fifth examples, an overview will be described first, and then a more specific description will be provided.

[0030] In the description of the first to fifth examples, for convenience, the drawings used in the embodiments described below are referred to. In the drawings, specific modalities such as images and texts are described, but the configurations and operations described in the first to fifth examples are not limited to a specific modality and can be applied to any modality.

[0031] <First example> In the first example, a description will be given of the feature extraction model 10 included in the learning unit 220. When learning the feature extraction model 10, the learning unit 220 randomly masks a predetermined ratio of one or more part features among one or more part features output by a backbone corresponding to each modality, and optimizes parameters of the feature extraction model 10 so that the semantic content corresponding to the mask feature, which is the masked part feature, can be recognized from the remaining part features.

[0032] An example of the configuration of the feature extraction model 10 during learning is shown in Fig. 8. As shown in Fig. 8, the feature extraction model 10 has a backbone section 13, a feature masking section 11, and an attention section 12. The backbone section 13 has an image backbone 14 and a text backbone 15. The example in Fig. 8 shows a case where an image and text are used as data of two modalities.

[0033] The backbone unit 13 outputs, from the input data of each modality, a global feature that captures the whole and one or more part features that capture a part of the data.

[0034] The feature masking unit 11 masks a predetermined ratio of part features among one or more part features for each modality, and outputs features corresponding to the masked part features as mask features.

[0035] The attention unit 12 receives part features and mask features for multiple modalities as input, converts the mask features based on the correlation between them, and outputs the converted mask features.

[0036] Fig. 9 shows an example of the configuration of the learning unit 220. Fig. 9 also shows the storage unit 240 (learning data DB).

[0037] As shown in FIG. 9, the learning unit 220 includes a feature extraction model 10, a mask recognition error calculation unit 20, a matching error calculation unit 30, and a model parameter optimization unit .

[0038] The matching error calculation unit 30 calculates an error function defined based on a comparison between modalities of global features output from the feature extraction model 10 to which learning data has been input, and obtains a matching error. The mask recognition error calculation unit 20 calculates a recognition error for a part corresponding to a mask feature output from the feature extraction model 10.

[0039] The model parameter optimization section 40 updates the parameters of the feature extraction model 10 based on the matching error output by the matching error calculation section 30 and the mask recognition error output by the mask recognition error calculation section 20 so as to minimize these errors.

[0040] As described above, by simultaneously performing global feature matching between different modalities and recognizing the masked part from the remaining parts in each modality during model training, the remaining parts that are highly correlated with the masked parts can be dynamically acquired during model training. In other words, a feature extraction model 10 that captures the local correlation between data can be acquired.

[0041] <Second example> In the second example (and the third to fifth examples), a more detailed description will be given of the mask recognition error calculation unit 20. As shown in FIG. 10, the mask recognition error calculation unit 20 in the second example has a reconstruction vector extraction unit 21 and a reconstruction error calculation unit 22.

[0042] The reconstructed vector extraction unit 21 receives the mask feature output from the feature extraction model 10 and outputs a reconstructed vector. The part feature, mask feature, reconstructed vector, and mask recognition error shown in Fig. 10 correspond to each other and relate to a certain part in the image. The same is true for other figures.

[0043] The reconstruction error calculation unit 22 receives the part feature output from the backbone unit 13 and the reconstructed vector output from the reconstructed vector extraction unit 21 as input, and calculates a mask recognition error, which is the error between the part feature and the reconstructed vector.

[0044] In training the feature extraction model 10, the parameters are updated to minimize errors including this mask recognition error, so that the model can be trained so that the part features of the masked image can be reconstructed from the remaining part features including not only the part features of the image modality but also the part features of the text modality, and the remaining parts that are highly correlated with the masked parts can be dynamically acquired during model training.

[0045] <Third Example> The third example describes another example of the mask recognition error calculation unit 20. As shown in Fig. 11, the mask recognition error calculation unit 20 in the third example has a reconstruction vector extraction unit 21, a reconstruction error calculation unit 22, and a part image extraction unit 23.

[0046] The reconstructed vector extraction unit 21 receives the mask feature output from the feature extraction model 10 as an input and outputs a reconstructed vector.

[0047] The part image extraction unit 23 divides the input image so as to correspond to the part features, and outputs the divided images as part images, as shown in Fig. 12. Note that the part images are an example of part data.

[0048] The reconstruction error calculation unit 22 receives the part image and the reconstructed vector output from the reconstructed vector extraction unit 21 as input, and calculates a mask recognition error, which is an error between the part image and the reconstructed vector.

[0049] In training the feature extraction model 10, the parameters are updated to minimize errors including this mask recognition error, so that the model can be trained so that a part image corresponding to the part features of a masked image can be reconstructed from the remaining part features including not only the part features of the image modality but also the part features of the text modality, and the remaining parts that are highly correlated with the masked parts can be dynamically acquired during model training.

[0050] <Fourth Example> The fourth example describes yet another example of the mask recognition error calculation unit 20. As shown in Fig. 13, the mask recognition error calculation unit 20 in the fourth example has a classification vector extraction unit 24, a classification error calculation unit 25, and a part label extraction unit 26.

[0051] The classification vector extraction unit 24 receives the mask feature and outputs a classification vector. The part label extraction unit 26 outputs a single label (part label) corresponding to each part feature from the input image, as shown in FIG.

[0052] The classification error calculation unit 25 receives the classification vector and the part label as input, and calculates the error between them.

[0053] In training the feature extraction model 10, the parameters are updated to minimize errors including this error, so that the model can be trained to infer a label corresponding to the part features of a masked image from the remaining part features including not only the part features of the image modality but also the part features of the text modality, and the remaining parts that are highly correlated with the masked parts can be dynamically acquired during model training.

[0054] <Fifth Example> The fifth example will be described mainly with respect to the differences from the fourth example. In the fifth example, the part label extraction unit 26 in the mask recognition error calculation unit 20 in the fourth example outputs an arbitrary number of labels corresponding to each part feature from the input image as shown in FIG.

[0055] A classification error calculation section 25 included in the mask recognition error calculation section 20 calculates the error between the multiple labels and the part features.

[0056] This allows the model to be trained so that multiple labels corresponding to the part features of the masked image can be inferred from the remaining part features, including the text modal, and the remaining parts that are highly correlated with the masked parts can be dynamically acquired during model training.

[0057] Hereinafter, more specific examples of the operations of the learning device 200 and the search device 100 described above will be described as Examples 1 to 3.

[0058] Example 1 In the first embodiment, the learning device 100 uses a single text as a query and ranks images in a gallery based on the similarity to the content described in the text, as shown in Fig. 2. To realize this search process, it is necessary to learn the feature extraction model 10 and build a gallery. Each process will be described in detail below.

[0059] <Learning 10 feature extraction models> The learning unit 220 of the learning device 200 learns (optimizes parameters) the feature extraction model 10. The configuration of the feature extraction model 10 is as shown in FIG. 8, and the configuration of the learning unit 220 is as shown in FIG.

[0060] The learning data used for training the feature extraction model 10 is a data set made up of an image group to which a unique ID is assigned for each individual, and a text group for the individual, as shown in the learning data DB portion of FIG.

[0061] In the feature extraction model 10 during learning shown in Fig. 8, the image backbone unit 14 receives an image as input and outputs one global feature and one or more part features. The text backbone unit 15 receives text as input and outputs one global feature and one or more part features.

[0062] The feature masking unit 11 masks a predetermined ratio of one or more part features and outputs the masked part features as mask features. The attention unit 12 receives the part features and the mask features as input, converts the mask features based on the correlation between them, and outputs the converted mask features.

[0063] For example, a neural network such as ResNet (Non-Patent Document 3) or ViT (Non-Patent Document 4) can be used as the image backbone unit 14. Also, for example, a text encoder such as BERT (Non-Patent Document 5) or CLIP (Non-Patent Document 1) can be used as the text backbone unit 15.

[0064] In the feature masking unit 11, the mask features converted from the part features and output can be defined as learnable parameters of the same dimension as the part features. Different mask features may be prepared for the image backbone unit 13 and the text backbone unit 15, or the same mask features may be used. In addition, the proportion of one or more part features to be converted into mask features can be set to any value, and different values ​​may be used for the image and the text, or the same value may be used.

[0065] The attention unit 12 can be configured in any manner, and for example, the transformer disclosed in Non-Patent Document 6 can be used.

[0066] The learning process of the feature extraction model 10 described above is executed by the learning unit 220 shown in Fig. 9. More specifically, the process is as follows.

[0067] A matching error calculation unit 20 calculates an error based on matching between the image global features and the text global features output by the feature extraction model 10. A mask recognition error calculation unit 30 calculates an error related to the recognition of the image mask features and the text mask features output by the feature extraction model 10.

[0068] The model parameter optimization unit 40 optimizes the parameters of the feature extraction model 10 so as to simultaneously minimize the matching error and the mask recognition error. Specific examples of each unit are as follows.

[0069] The matching error calculation unit 30 can be realized as an arbitrary error function such that the error becomes smaller as the distance between the image global feature and the text global feature capturing the same individual becomes smaller, and the error becomes smaller as the distance between the image global feature and the text global feature capturing different individuals becomes larger. For example, the triplet error or the CMPM error disclosed in Non-Patent Document 7 can be used as the matching error calculation unit 30.

[0070] More specifically, the mask recognition error calculation unit 20 has a functional unit that takes as input the mask features of the text and calculates a mask error for the text, and a functional unit that takes as input the mask features of the image and calculates a mask error for the image. Hereinafter, the functional unit that takes as input the mask features of the text and calculates a mask error for the text is referred to as the mask recognition error calculation unit 20 (text), and the functional unit that takes as input the mask features of the image and calculates a mask error for the image is referred to as the mask recognition error calculation unit 20 (image).

[0071] <Mask Recognition Error Calculation Unit 20 (Text)> FIG. 16 shows an example of the configuration of the mask recognition error calculation unit 20 (text) that receives the mask features of text as input.

[0072] 16, the mask recognition error calculation unit 20 (text) has a classification vector extraction unit 28 and a cross-entropy calculation unit 29. The classification vector extraction unit 28 extracts a classification vector from a mask feature. The cross-entropy calculation unit 29 calculates the cross-entropy.

[0073] That is, the mask recognition error can be calculated as an error function value of a classification vector obtained from the mask feature by the classification vector extraction unit 28, with a word or word string corresponding to the part feature being taken as a correct label. In the example of Fig. 16, for example, if "coat" is taken as the correct label, the mask recognition error is obtained as an error function value between "coat" and the classification vector obtained from the mask feature corresponding to "coat".

[0074] The classification vector extraction unit 28 can be defined, for example, by a fully connected neural network that converts the mask feature into a vector with the same number of dimensions as the number of words to be classified, and a softmax function. Also, the error function can be the cross-entropy, as in the example of FIG. 16.

[0075] Hereinafter, examples 1 to 3 of the mask recognition error calculation unit 20 (image) will be described.

[0076] <Mask recognition error calculation unit 20 (image): Example 1> In example 1, the mask recognition error calculation unit 20 (image) outputs an error value between the part feature input to the feature masking unit 11 of the feature extraction model 10 and the reconstructed value of the mask feature output by the attention unit 12.

[0077] Specifically, the configuration of the mask recognition error calculation unit 20 (image) in Example 1 is as shown in Fig. 10, and includes a reconstruction vector extraction unit 21 and a reconstruction error calculation unit 22. The reconstruction vector extraction unit 21 receives a mask feature corresponding to a part feature as an input and outputs a reconstruction vector. The reconstruction error calculation unit 22 calculates a reconstruction error between the reconstruction vector and the part feature.

[0078] For example, a fully connected neural network that aligns the number of dimensions of the mask feature and the part feature can be used as the reconstruction vector extraction unit 21. Also, for the reconstruction error calculation unit 22, an error function that becomes smaller as the mask feature and the part feature become more similar, such as the L1 norm, the L2 norm, or the KL divergence, can be used.

[0079] <Mask recognition error calculation unit 20 (image): Example 2> In example 2, the mask recognition error calculation unit 20 (image) outputs an error value between the part image corresponding to the part feature input to the feature masking unit 11 of the feature extraction model 10 and the reconstructed value of the mask feature output by the attention unit 12.

[0080] Specifically, the configuration of the mask recognition error calculation unit 20 (image) in Example 2 is as shown in FIG. 11, and includes a reconstruction vector extraction unit 21, a reconstruction error calculation unit 22, and a part image extraction unit .

[0081] The reconstruction vector extraction unit 21 receives mask features corresponding to the part features as input and outputs a reconstruction vector. The part image extraction unit 23 receives an image as input and outputs a part image corresponding to each mask feature. The reconstruction error calculation unit 22 calculates a reconstruction error between the reconstruction vector reconstructed from the mask features and the part image corresponding to the mask feature.

[0082] A fully connected neural network that aligns the number of dimensions of the mask feature and the vector obtained by rearranging the part image into one dimension can be used as the reconstruction vector extraction unit 21. Also, an error function that becomes smaller as the mask feature and the part image become more similar, such as the L1 norm, L2 norm, or KL divergence, can be used as the reconstruction error calculation unit 22.

[0083] <Mask recognition error calculation unit 20 (image): Example 3> A module for estimating a part label corresponding to a part feature input to the feature masking unit 11 of the feature extraction model 10 is added to the mask recognition error calculation unit 20 (image) of Example 3. The mask recognition error calculation unit 20 (image) outputs, as a mask recognition error, an error value between the part label and a correct part label when the part label is recognized by the mask feature output by the attention unit 12.

[0084] Specifically, the configuration of the mask recognition error calculation unit 20 (image) in Example 3 is as shown in FIG. 13, and includes a classification vector extraction unit 24, a part label extraction unit 26, and a classification error calculation unit 25.

[0085] The classification vector extraction unit 24 receives the mask feature as input and outputs a classification vector. The part label extraction unit 26 receives an image as input and extracts the label of the part image corresponding to the classification vector. The classification error calculation unit 25 receives the classification vector and the part label as input and calculates the recognition error between them.

[0086] The part label extraction unit 26 may have any configuration, for example, the configuration shown in Fig. 14. The configuration shown in Fig. 14 includes a segmentation unit 27 that performs segmentation as disclosed in Non-Patent Document 8, for example.

[0087] The segmentation unit 27 applies segmentation to the input image, and outputs labels that frequently appear in the part image corresponding to the mask feature as part labels, as shown in FIG.

[0088] The classification vector extraction unit 24 can use a fully connected neural network and a softmax function that converts the mask feature into a vector with the same number of dimensions as the number of labels to be classified. The classification error calculation unit 25 can use cross entropy.

[0089] The processing contents of the part label extraction unit 26 are not limited to those shown in Fig. 14. The processing contents of the part label extraction unit 26 may be those shown in Fig. 15. In the example shown in Fig. 15, the segmentation unit 27 outputs all labels appearing in the part image corresponding to the mask feature as part labels. In this case, the classification vector extraction unit 24 can be configured with a fully connected neural network that converts the mask feature into a vector with the same number of dimensions as the number of labels to be classified, and a sigmoid function. The classification error calculation unit 25 calculates the binary cross entropy for each label.

[0090] <About building a gallery> Next, construction of a gallery, which is a data group to be searched, will be described. The gallery is constructed by a device (computer) equipped with a trained feature extraction model 10. The device may be the learning device 200, the search device 100, or other devices.

[0091] A method for constructing a gallery will be described with reference to Fig. 17. As shown in Fig. 17, a feature vector is extracted from each image to be registered in the gallery using a feature extraction model 10 with optimized parameters.

[0092] In this case, in the feature extraction model 10, only components necessary for extracting global features need to be used, as shown in Fig. 18. As shown in Fig. 17, each element registered in the gallery consists of an ID unique to each image, features, and attributes. Any information can be registered as an attribute, and for example, if the specific object is a person, the name, gender, attributes, etc. of the person can be set.

[0093] <About search> The search is performed by the search device 100 shown in Fig. 1. Fig. 19 shows a detailed configuration of the cross-modal search unit 120. Fig. 19 also shows that a data group constituting a gallery is stored in the storage unit 140.

[0094] 19, the cross-modal search unit 120 has a trained feature extraction model 10 and a search unit 150. A query text is input to the feature extraction model 10 to extract features. Then, the search unit 150 compares the query features with the features of each element in the gallery to rank the elements in the gallery and output the results.

[0095] When extracting features from a query, in the feature extraction model 10, as shown in FIG. 18, only components necessary for extracting global features need to be used.

[0096] The search unit 150 may compare the query features with the gallery features in any manner. For example, the search unit 150 may calculate the cosine similarity between the query features and the features of all the gallery elements, and rank the elements in the storage unit 140 in descending order of similarity.

[0097] Example 2 In the second embodiment, the learning device 100 uses a single image as a query and ranks a group of texts in a gallery based on the similarity to the contents of the image, as shown in Fig. 3. To realize this search process, it is necessary to learn the feature extraction model 10 and build a gallery, as in the first embodiment.

[0098] The feature extraction model 10 can be trained in the same manner as in the first embodiment. The gallery can be constructed and searched by replacing the input modality in the first embodiment.

[0099] Example 3 In the third embodiment, the learning device 100 uses a single text as a query and ranks the videos in the gallery based on the similarity to the content described in the text, as shown in Fig. 4. To realize this search process, it is necessary to learn the feature extraction model 10 and build a gallery, as in the first and second embodiments. The following describes the differences from the first embodiment.

[0100] <Learning 10 feature extraction models> 20 shows an example of the configuration of the feature extraction model 10 in the third embodiment. As shown in FIG. 20, the feature extraction model 10 in the third embodiment has a video backbone unit 16, a text backbone unit 15, a feature masking unit 11, and an attention unit 12.

[0101] The video backbone unit 16 receives a video and outputs one global feature and one or more part features. The text backbone unit 15 receives a text and outputs one global feature and one or more part features.

[0102] The feature masking unit 11 masks a predetermined ratio of one or more part features and outputs the masked part features as mask features. The attention unit 12 receives the part features and the mask features as input, converts the mask features based on the correlation between them, and outputs the converted mask features.

[0103] The text backbone unit 15, the feature masking unit 11, and the attention unit 12 can be realized in the same manner as in the first embodiment (FIG. 8). The video backbone unit 16 can be realized using SlowFast (Non-Patent Document 9) or ViViT (Non-Patent Document 10).

[0104] Alternatively, the image backbone unit 14 of the first embodiment (FIG. 8) may be applied to each frame constituting the video in the video backbone unit 16. In this case, for example, the average of the global feature group obtained by the image backbone unit 14 is set as the global feature output by the video backbone unit 16, and the average of the part features extracted from the same position in the frame obtained by the image backbone unit 14 is set as the part feature output by the video backbone unit 16.

[0105] When building a gallery and performing a search, as shown in FIG. 21, only the portion of the feature extraction model 10 required for extracting the image global features and the text global features is selected and used.

[0106] (Hardware configuration example) Any of the devices described in this embodiment (search device 100, learning device 200) can be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud.

[0107] That is, the device can be realized by executing a program corresponding to the processing performed by the device, using hardware resources such as a CPU and memory built into a computer. The program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The program can also be provided via a network such as the Internet or email.

[0108] Fig. 22 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 22 has a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, etc., which are mutually connected by a bus BS. The computer may further include a GPU.

[0109] The program for realizing the processing on the computer is provided by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0110] The memory device 1003 reads out and stores the program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes the functions related to the device in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to a program. The input device 1007 is composed of a keyboard and mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0111] (Summary of the embodiment, effects, etc.) As described above, the technology described in this embodiment makes it possible to generate a feature extraction model that takes into account local correlations between data from multiple different modalities, thereby enabling accurate cross-modal search.

[0112] The following supplementary notes are further disclosed regarding the above embodiment.

[0113] <Additional Notes> (Additional note 1) A learning device that learns a feature extraction model, the feature extraction model is a model that receives learning data including data of a plurality of modalities as input, and outputs a global feature capturing the entire input data, and a mask feature based on one or more part features capturing a part of the input data; a matching error calculation unit that calculates a matching error between modalities of global features of a plurality of modalities output by the feature extraction model; a mask recognition error calculation unit that calculates a recognition error between a mask feature output by the feature extraction model and information on a part corresponding to the mask feature; a model parameter optimization unit that updates parameters of the feature extraction model so as to minimize the matching error and the recognition error; A learning device comprising: (Additional note 2) The feature extraction model is a backbone unit that extracts, for each of the multiple modalities, a global feature that captures the entirety of input data and one or more part features that capture a part of the input data; a feature masking unit that masks a predetermined ratio of one or more part features for each of the modalities and outputs the masked part features as mask features; and an attention unit that receives one or more part features and one or more mask features for the multiple modalities, converts each mask feature based on a correlation between the part feature and the mask feature, and outputs the converted mask feature. Item 1. A learning device according to item 1. (Additional note 3) The mask recognition error calculation unit a reconstruction vector extraction unit that receives a mask feature as an input and outputs a reconstruction vector; a reconstruction error calculation unit that calculates an error by inputting a part feature corresponding to the mask feature and the reconstruction vector; The learning device according to claim 1, comprising: (Additional note 4) The mask recognition error calculation unit a reconstruction vector extraction unit that receives a mask feature as an input and outputs a reconstruction vector; a reconstruction error calculation unit that calculates an error by inputting part data corresponding to the mask feature and the reconstruction vector; The learning device according to claim 1, comprising: (Additional note 5) The mask recognition error calculation unit a classification vector extraction unit that receives a mask feature as an input and outputs a classification vector; a classification error calculation unit that calculates an error using the label of part data corresponding to the mask feature and the classification vector as input; The learning device according to claim 1, comprising: (Additional note 6) A storage unit for storing a set of data of a first modality to be searched; an input unit for inputting a query of a second modality; a search unit that extracts features from the query using the feature extraction model trained by the learning device according to any one of appendixes 1 to 5, and performs a search on the data group using the features; A search device comprising: (Additional note 7) A learning method executed by a learning device that learns a feature extraction model, the feature extraction model is a model that receives learning data including data of a plurality of modalities as input, and outputs a global feature capturing the entire input data, and a mask feature based on one or more part features capturing a part of the input data; a matching error calculation step of calculating a matching error between modalities of global features of multiple modalities output by the feature extraction model; a mask recognition error calculation step of calculating a recognition error between a mask feature output by the feature extraction model and information on a part corresponding to the mask feature; a model parameter optimization step of updating parameters of the feature extraction model so as to minimize the matching error and the recognition error; A learning method that provides: (Additional note 8) A non-transitory storage medium storing a program for causing a computer to function as each part of the learning device described in any one of appendix 1 to 5.

[0114] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and variations are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0115] 10 Feature Extraction Model 11 Feature Masking Section 12 Attention Section 13 Backbone Section 14 Image Backbone 15 Text Backbone 20 Mask recognition error calculation unit 21 Reconstruction vector extraction unit 22 Reconstruction error calculation unit 23 Part Image Extraction Section 24 Classification Vector Extraction Unit 25 Classification error calculation unit 26 Part label extraction unit 27 Segmentation Department 28 Classification Vector Extraction Unit 29 Cross entropy calculation unit 30 Matching error calculation unit 40 Model Parameter Optimization Section 100 Search device 110 Input section 120 Cross-modal Search Unit 130 Output section 140 Storage section 150 Search Department 200 Learning Device 210 Input section 220 Learning Department 230 Output section 240 Storage section 1000 Drive Device 1001 Recording media 1002 Auxiliary storage 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input Device 1008 Output device

Claims

1. A learning device that learns a feature extraction model, the feature extraction model is a model that receives learning data including data of a plurality of modalities as input, and outputs a global feature capturing the entire input data and a mask feature based on one or more part features capturing a part of the input data; a matching error calculation unit that calculates a matching error between modalities of global features of a plurality of modalities output by the feature extraction model; a mask recognition error calculation unit that calculates a recognition error between a mask feature output by the feature extraction model and information on a part corresponding to the mask feature; a model parameter optimization unit that updates parameters of the feature extraction model so as to minimize the matching error and the recognition error; A learning device comprising:

2. The feature extraction model is a backbone unit that extracts, for each of the plurality of modalities, a global feature that captures the entirety of input data and one or more part features that capture a part of the input data; a feature masking unit that masks a predetermined ratio of one or more part features for each of the modalities and outputs the masked part features as mask features; and an attention unit that receives one or more part features and one or more mask features for the plurality of modalities, converts each mask feature based on a correlation between the part feature and the mask feature, and outputs the converted mask feature. The learning device according to claim 1 .

3. The mask recognition error calculation unit a reconstruction vector extraction unit that receives a mask feature as an input and outputs a reconstruction vector; a reconstruction error calculation unit that calculates an error by inputting a part feature corresponding to the mask feature and the reconstruction vector; The learning device according to claim 1 .

4. The mask recognition error calculation unit a reconstruction vector extraction unit that receives a mask feature as an input and outputs a reconstruction vector; a reconstruction error calculation unit that calculates an error by inputting part data corresponding to the mask feature and the reconstruction vector; The learning device according to claim 1 .

5. The mask recognition error calculation unit a classification vector extraction unit that receives a mask feature as an input and outputs a classification vector; a classification error calculation unit that calculates an error using the label of part data corresponding to the mask feature and the classification vector as input; The learning device according to claim 1 .

6. A storage unit for storing a data set of a first modality to be searched; an input unit for inputting a query of a second modality; a search unit that extracts features from the query using the feature extraction model trained by the learning device according to any one of claims 1 to 5, and searches the data set using the features; A search device comprising:

7. A learning method executed by a learning device that learns a feature extraction model, the feature extraction model is a model that receives learning data including data of a plurality of modalities as input, and outputs a global feature capturing the entire input data and a mask feature based on one or more part features capturing a part of the input data; a matching error calculation step of calculating a matching error between modalities of global features of multiple modalities output by the feature extraction model; a mask recognition error calculation step of calculating a recognition error between a mask feature output by the feature extraction model and information on a part corresponding to the mask feature; a model parameter optimization step of updating parameters of the feature extraction model so as to minimize the matching error and the recognition error; A learning method that provides:

8. A program for causing a computer to function as each unit in the learning device according to any one of claims 1 to 5.

Images