Information processing device and information processing method

Abstract

To create a common space reflecting a correspondence between plural pieces of information between different modals.SOLUTION: An information processing apparatus according to the present application comprises: an acquisition unit for acquiring pair data as a pair of first information and second information which is information of a type different from the first information and corresponds to the first information; an extraction unit for extracting a first feature amount which is a feature amount corresponding to the first information, from the first information, and extracting a second feature amount which is a feature amount corresponding to the second information, from the second information; and a creation unit for creating plural third feature amounts corresponding to the first information from the first feature amount by using a creation model which is a machine learning model outputting output information which follows a specified probability distribution depending on the input of input information, and mapping the plural third feature amounts and the second feature amount to a common space which is a common feature amount space.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing device and an information processing method. [Background technology]

[0002] In recent years, there has been active research into multimodal technology that combines different types of information (also called information belonging to different modalities) such as images and language. For example, known multimodal technology that combines images and language includes technology that generates explanatory text (captions) from images and technology that enables mutual search between images and text.

[0003] Furthermore, in order to realize the above-mentioned multimodal technology, a technology is used that expresses two different types of information, such as images and language (hereinafter also referred to as text), in a common feature space (also referred to as a multimodal space or a common space, hereinafter referred to as a "common space"). For example, a technology is known that uses a data set including a pair of images and text indicating the contents of the images to train a machine learning model so as to map each feature to the common space by associating one image with one piece of text corresponding to the image. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Alec Radford and 11 others, “Learning Transferable Visual Models From Natural Language Supervision”, [online], February 2021, [Retrieved February 28, 2023], Internet<URL:https: / / arxiv.org / abs / 2103.00020> Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the above-mentioned conventional technology, a machine learning model is trained to match one image with one piece of text corresponding to the image and map each of their features into a common space, so it is not necessarily possible to generate a common space that reflects the correspondence between multiple pieces of information across different modalities.

[0006] An object of the present application is to provide an information processing device and an information processing method capable of generating a common space that reflects the corresponding relationships between a plurality of pieces of information in different modalities. [Means for solving the problem]

[0007] The information processing device according to the present application includes an acquisition unit that acquires pair data that is a set of first information and second information, which is a type of information different from the first information and corresponds to the first information; an extraction unit that extracts a first feature that is a feature corresponding to the first information from the first information and extracts a second feature that is a feature corresponding to the second information from the second information; and a generation unit that generates a plurality of third features that correspond to the first information from the first features using a generative model that is a machine learning model that outputs output information that follows a specific probability distribution in response to input information, and maps the plurality of third features and the second feature to a common space that is a common feature space. Effect of the Invention

[0008] According to one aspect of the embodiment, it is possible to generate a common space that reflects the correspondence between a plurality of pieces of information across different modalities. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram for explaining an example of a learning process of two types of encoders capable of embedding image features and text features in a common space according to the conventional technology. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of the information processing device according to the embodiment. [Diagram 3] FIG. 3 is a flowchart illustrating an example of a common space generation process according to the embodiment. [Figure 4] FIG. 4 is a diagram for explaining an example of a common space generation process according to the embodiment. [Diagram 5] FIG. 5 is a flowchart illustrating an example of the search process according to the embodiment. [Figure 6] FIG. 6 is a diagram for explaining an example of the search process according to the embodiment. [Figure 7] FIG. 7 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, a detailed description will be given of an information processing device and an information processing method according to the present application (hereinafter, referred to as an "embodiment") with reference to the drawings. Note that the information processing device and the information processing method according to the present application are not limited to the embodiment. In addition, the same components in the following embodiments are given the same reference numerals, and duplicated descriptions are omitted.

[0011] (Embodiment) 1. Introduction In this specification, an "image" may be a video such as a video captured by a camera (imaging device), or each scene (still image) contained in the video. Also, in this specification, a "linguistic expression" may be character information (hereinafter, "text") such as a sentence, a phrase, or a keyword.

[0012] In recent years, a technology has been known that maps feature quantities of sets of information that are one-to-one associated between different modalities into a common space, which is a common feature space, by associating them. For example, a technology that associates image features that indicate the features of an image with language features that indicate the features of a language expression (hereinafter, the language expression is referred to as text) that expresses the content of the image in language and embeds them into a common space is called VSE (Visual-Semantic Embedding). In addition, a technology that searches for text that matches a query image or an image that matches a query sentence using a machine learning model (hereinafter, also referred to as a VSE model) generated using the VSE technology is known (hereinafter, referred to as "image-text search"). In addition, mapping features extracted from data such as images and text into a common space may be described as embedding data such as images and text into a common space.

[0013] Specifically, the VSE model includes an image encoder that extracts image features (hereinafter, may be referred to as "image features") indicating image features from an image, and a text encoder that extracts text features (hereinafter, may be referred to as "text features") indicating text features from text. For example, when searching for text from an image using the VSE model, an image is input to the image encoder of the VSE model to generate image features. Next, the VSE model maps the image features generated by the image encoder to a common space with the text features, and then outputs text corresponding to text features similar to the image features as a search result. Here, the image features and text features may be vectors or tensors. For example, the VSE model may output text corresponding to text features whose similarity (e.g., cosine similarity) between the image features and the text features exceeds a predetermined threshold as a search result.

[0014] Furthermore, when searching for an image from text using the VSE model, the text is input to a text encoder of the VSE model to generate text features. The VSE model then maps the generated text features to a common space with the image features, and outputs images corresponding to image features similar to the text features as search results. For example, the VSE model may output images corresponding to image features whose similarity (e.g., cosine similarity) between the text features and the image features exceeds a predetermined threshold as search results.

[0015] FIG. 1 is a diagram for explaining an example of a learning process of two types of encoders capable of embedding image features and text features in a common space according to the prior art. When an image is input, the image encoder shown in the upper left of FIG. 1 outputs image features indicating the features of the image. For example, the image features are expressed by a vector having a lower dimensionality than the image input to the image encoder. Also, when text is input, the text encoder shown in the lower left of FIG. 1 outputs text features indicating the features of the text. For example, the text features are expressed by a vector having a lower dimensionality than the text input to the text encoder.

[0016] The learning process shown in Figure 1 is called CLIP (Contrastive Language-Image Pre-training). In CLIP, a pair of an image and text showing the content of that image (also called paired data) is prepared. Then, the image encoder and the text encoder are trained so that the similarity between the image feature and the text feature corresponding to the paired data increases. On the other hand, the image encoder and the text encoder are trained so that the similarity between the image feature and the text feature not in the paired data decreases. In this way, in CLIP, the image encoder and the text encoder are trained so that the image feature and the text feature are embedded in a common space.

[0017] The image encoder shown in FIG. 1 may be realized by any known technology capable of extracting image features from an image. For example, the image encoder may include a convolutional neural network (CNN). The image encoder may extract image features from an image using the CNN. For example, the image encoder may include ResNet (Residual Network) (Kaiming He et al., 2015), AlexNet (Krizhevsky et al., 2012), VGGNet (Simonyan et al., 2014), GoogLeNet (Szegedy et al., 2014), SENet (Squeeze-and-Excitation Networks) (Jie Hu et al., 2018), EfficientNet (Tan et al., 2019), or ZFNet (Matthew et al., 2013) developed for object recognition. The image encoder may extract image features from the image using ResNet, AlexNet, VGGNet, GoogLeNet, SENet, EfficientNet, or ZFNet. For example, the image encoder may include Faster R-CNN (Shaoqing Ren et al., 2015), YOLO (You Look Only Onse) (Joseph Redmon et al., 2015), or SSD (Single Shot MultiBox Detector) (Wei Liu., 2015), which are developed for object detection. The image encoder may extract image features from the image using Faster R-CNN, YOLO, or SSD.

[0018] The text encoder shown in FIG. 1 may be realized by any known technology capable of extracting text features from text. For example, the text encoder may include a recurrent neural network (RNN). The text encoder may extract text features from text using the RNN. For example, the text encoder may include a gated recurrent unit (GRU) or a long short term memory (LSTM). The text encoder may extract text features from text using the GRU or the LSTM. For example, the text encoder may include a Transformer (Ashish Vaswani et al., 2017), a Bidirectional Encoder Representations from Transformers (BERT) based on a Transformer, Generative Pre-Training3 (GPT-3), or a Text-to-Text Transfer Transformer (T5). The text encoder may extract text features from text using the Transformer, BERT, GPT-3, or T5.

[0019] As described above, the conventional technology uses information on a pair of an image and text indicating the content of the image to have an image encoder and a text encoder learn the correspondence between the image and the text. That is, the conventional technology maps image features and text features, which are features of the information on a pair of an image and a text, in a common space by associating them with each other. This makes it possible for the conventional technology to generate a common space that reflects the correspondence between information on a pair of different modalities, such as an image and a text, that are associated with each other on a one-to-one basis.

[0020] However, in reality, information between different modalities has not only a one-to-one correspondence but also a many-to-many correspondence. For example, in the example shown in FIG. 1, in addition to the text "A pair of children sit on a giraffe while other children stand nearby." shown in the lower part of FIG. 1 (hereinafter, "There are two children riding on a giraffe, and there are other children around them"), multiple texts such as "A giraffe playing with its child in the shade of a tree" can be associated with the image shown in the upper part of FIG. 1. Also, in addition to the image shown in the upper part of FIG. 1, other images showing scenes at a zoo can be associated with the text "There are two children riding on a giraffe, and there are other children around them" shown in the lower part of FIG. 1. Therefore, a technology that can generate a better common space that reflects the many-to-many correspondence between different modalities is desired. In other words, a technology that can generate a common space that reflects the correspondence between multiple pieces of information between different modalities is desired.

[0021] More specifically, for example, if a machine learning model used in image-text search is trained on the assumption that a one-to-one correspondence between the modalities of an image and a modal of a text exists, it may adversely affect the ranking of search results. For example, when searching for images using the text "photo of a giraffe and a child" as a query, the third image may be ranked at the bottom of the search results in the ranking of three images trained in correspondence with the texts "photo of a giraffe," "photo of a child," and "photo of a giraffe and a child in the shade." This is because the query "photo of a giraffe and a child" does not include the word "shade" contained in the text associated with the third image. However, of the three images, the image that best matches the query "photo of a giraffe and a child" is the third image associated with the text "photo of a giraffe and a child in the shade," so it is desirable for the third image to be ranked at the top of the search results. Thus, in order to improve the accuracy of the ranking of search results, a technology is desired that can generate a common space that reflects the correspondence between multiple pieces of information between different modalities.

[0022] In recent years, technology related to generative models has become known. A generative model is a machine learning model that can learn training data and generate data similar to the training data. For example, when an image of a cat is used as training data, the generative model can generate an image of a cat. For example, generative adversarial networks (GANs), variational autoencoders (VAEs), diffusion models, and the like are known as examples of generative models.

[0023] Specifically, the generative model treats the input information as a random variable. In other words, the generative model assumes that the input information follows a specific probability distribution. That is, the generative model considers the probability with which the input information generated from a specific probability distribution fits into a specific class. At this time, if the probability distribution that the input information follows can be obtained, it is considered possible to generate pseudo-information equivalent to the input information using the probability distribution that the input information follows. In other words, the generative model is a machine learning model that generates pseudo-information that follows a specific probability distribution that the input information input to the generative model follows, and outputs it as output information. Here, the input information input to the generative model and the output information output by the generative model (that is, the pseudo-information generated by the generative model) are similar information because they are generated from the same probability distribution.

[0024] In addition, since the output information output by the generative model follows a specific probability distribution, the output information varies probabilistically. For example, when the generative model is used N times (N is a natural number equal to or greater than 2), the first output information output the first time, the second output information output the second time, ..., the Nth output information output the Nth time each vary probabilistically, so it is considered that there is a high probability that the first to Nth output information will be similar information but slightly different from each other.

[0025] In contrast, an information processing device according to an embodiment uses two types of trained encoders (e.g., an image encoder and a text encoder) each corresponding to two different types of information (e.g., an image and a text), and extracts two types of features (e.g., an image feature and a text feature) corresponding to each of the two different types of information having a corresponding relationship from the two different types of information. In addition, the information processing device uses a generative model to generate a plurality of pseudo features from one of the two types of features, and maps the generated pseudo features and the other feature in a common space. Specifically, the information processing device generates a generative model trained so that the similarity between each of the pseudo features and the other feature is below a predetermined threshold.

[0026] Thereby, the information processing device according to the embodiment can map a plurality of pseudo-features generated from one feature and the other feature in a common space in association with each other. That is, the information processing device can generate a common space that reflects the correspondence between a plurality of pieces of information similar to one piece of information and the other piece of information among two different types of information having a correspondence relationship. Similarly, the information processing device can map a plurality of pseudo-features generated from the other feature and a plurality of pseudo-features generated from the one feature in association with each other in a common space. That is, the information processing device can generate a common space that reflects the correspondence between a plurality of pieces of information similar to the other piece of information and the one piece of information among two different types of information having a correspondence relationship. Therefore, the information processing device can generate a common space that reflects the correspondence between a plurality of pieces of information between different modals.

[0027] 2. Configuration of Information Processing Device An example of the configuration of the information processing device 100 according to the embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the configuration of the information processing device 100 according to the embodiment. The information processing device 100 has a communication unit 110, a storage unit 120, and a control unit 130.

[0028] (Communication unit 110) The communication unit 110 is realized by a NIC (Network Interface Card), an antenna, etc. The communication unit 110 is connected to various networks by wire or wirelessly, and transmits and receives information to and from, for example, an information processing device (hereinafter, may be referred to as a "terminal device") used by a user.

[0029] (Storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. Specifically, the storage unit 120 stores various data. For example, the storage unit 120 stores information about images and text to be searched. The storage unit 120 also stores various programs. For example, the storage unit 120 stores information about a trained encoder and information about the generative model M1.

[0030] (Control unit 130) The control unit 130 is a controller, and is realized, for example, by a CPU (Central Processing Unit), MPU (Micro Processing Unit), etc., executing various programs stored in a storage device inside the information processing device 100 using a RAM as a working area. The control unit 130 is also a controller, and is realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0031] The control unit 130 has an acquisition unit 131, an extraction unit 132, a generation unit 133, and a search unit 134 as functional units, and may realize or execute the information processing action described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in Fig. 2, and may be other configurations as long as they perform the information processing described below. Also, each functional unit indicates a function of the control unit 130, and does not necessarily have to be physically distinct.

[0032] (Acquisition part 131) The acquisition unit 131 may acquire various types of information. Specifically, the acquisition unit 131 may acquire pair data that is a set of first information and second information that is a type of information different from the first information and corresponds to the first information. More specifically, the acquisition unit 131 may acquire a plurality of different pair data. For example, the acquisition unit 131 may acquire a plurality of different pair data from the storage unit 120 that stores a plurality of different pair data. For example, the set of the first information and the second information may be a set of an image and text, a set of a video and text, a set of an audio and video, a set of text and audio, or a set of two different types of language information (for example, Japanese and English).

[0033] (Extraction part 132) The extraction unit 132 may extract a first feature amount, which is a feature amount corresponding to the first information, from the first information acquired by the acquisition unit 131. Specifically, the extraction unit 132 may acquire a trained first encoder (hereinafter, abbreviated as "first encoder") corresponding to the first information. For example, the extraction unit 132 may acquire the first encoder from the storage unit 120 that stores information related to the first encoder. Then, the extraction unit 132 may extract the first feature amount from the first information acquired by the acquisition unit 131 using the first encoder. For example, the extraction unit 132 may input the first information acquired by the acquisition unit 131 to the first encoder and acquire the first feature amount output from the first encoder.

[0034] Furthermore, the extraction unit 132 may extract a second feature amount, which is a feature amount corresponding to the second information, from the second information acquired by the acquisition unit 131. Specifically, the extraction unit 132 may acquire a trained second encoder (hereinafter, abbreviated as "second encoder") corresponding to the second information. For example, the extraction unit 132 may acquire the second encoder from the storage unit 120 that stores information related to the second encoder. Next, the extraction unit 132 may extract the second feature amount from the second information acquired by the acquisition unit 131 using the second encoder in a manner similar to the first feature amount.

[0035] (Generation unit 133) The generating unit 133 may generate a third feature, which is a plurality of pseudo features corresponding to the first information, from the first feature extracted by the extracting unit 132, by using the generating model M1, which is a machine learning model that outputs output information following a specific probability distribution in response to the input of input information. Specifically, the generating unit 133 may acquire the generating model M1 from the storage unit 120 that stores information related to the generating model M1. Subsequently, the generating unit 133 may input the first feature as input information of the generating model M1, and generate a third feature to be output as output information of the generating model M1. For example, the generating unit 133 may input the first feature as input information of the generating model M1 a preset number of times (e.g., two or more times), and generate a third feature to be output as output information of the generating model M1. In this way, the generating unit 133 may generate a preset number of third features (e.g., two or more). The generating unit 133 may input the first feature as input information of the generative model M1, and generate multiple third feature amounts at once by batch processing to be output as output information of the generative model M1.

[0036] Next, the generating unit 133 may map the generated third feature quantities and the second feature quantities to a common space. Specifically, the generating unit 133 may map the third feature quantities to a feature space (hereinafter, may be referred to as a "second feature quantity space") corresponding to the second feature quantities generated by the second encoder. More specifically, the generating unit 133 may train the generative model M1 so that the similarity between each of the third feature quantities and the second feature quantities falls below a predetermined threshold. In this way, the generating unit 133 may map the third feature quantities and the second feature quantities to the second feature quantity space, which is a common space.

[0037] Similarly, the generating unit 133 may generate a fourth feature, which is a plurality of pseudo features corresponding to the second information, from the second feature extracted by the extracting unit 132, using the generating model M2, which is a machine learning model that outputs output information following a specific probability distribution in response to the input of input information. Then, the generating unit 133 may map the generated fourth feature, the third feature, and the second feature to a second feature space, which is a common space. For example, the generating unit 133 may train the generating model M2 so that the similarity between each of the plurality of fourth feature and the second feature is below a predetermined threshold. Alternatively, the generating unit 133 may calculate an average value of the plurality of third feature mapped to the common space. Then, the generating unit 133 may train the generating model M2 so that the similarity between each of the plurality of fourth feature and the average value of the plurality of third feature is below a predetermined threshold. In this manner, the generating unit 133 may map the plurality of fourth feature amounts, the plurality of third feature amounts, and the second feature amount into the second feature amount space, which is a common space.

[0038] In this way, the generating unit 133 may generate a plurality of fourth features corresponding to the second information from the second features using the generating model M2, which is a machine learning model that outputs output information following a specific probability distribution in response to the input of input information, and map the plurality of fourth features, the plurality of third features, and the second feature to a common space that is a common feature space. Furthermore, the generating unit 133 may generate the generating model M2 that has been trained so that the similarities between each of the plurality of fourth features and the average value of the plurality of third features are each below a predetermined threshold.

[0039] (Search section 134) The search unit 134 may acquire a query accepted from a user. For example, the search unit 134 may acquire a query input by a user from a terminal device used by the user via the communication unit 110. For example, the search unit 134 may acquire a query that is the first information.

[0040] Furthermore, when the search unit 134 acquires a query that is the first information, the search unit 134 may acquire a plurality of pieces of second information. Specifically, the search unit 134 may acquire a plurality of pieces of second information from the storage unit 120 that stores the plurality of pieces of second information.

[0041] Furthermore, the extraction unit 132 may extract a first feature from a query that is the first information acquired by the search unit 134. Furthermore, the extraction unit 132 may extract a plurality of second feature amounts from each of the plurality of second information acquired by the search unit 134.

[0042] Furthermore, the generating unit 133 may calculate a similarity between the query, which is the first information, and each of the multiple pieces of second information. For example, the generating unit 133 may calculate an average value of similarities between each of the multiple third feature amounts corresponding to the first information and the second feature amount corresponding to the second information, as the similarity between the query, which is the first information, and the second information. For example, the generating unit 133 may calculate an average value of cosine similarities between each of the multiple third feature amounts and the second feature amount, as the similarity between the query, which is the first information, and the second information.

[0043] Furthermore, the search unit 134 may determine the output order (also called ranking) of the second information to be output as the search result, based on the average value of the similarity calculated by the generation unit 133. For example, the search unit 134 may determine that the second information having a higher average value of the similarity is preferentially output as the search result.

[0044] In this way, the search unit 134 searches for second information corresponding to the first information from among the plurality of pieces of second information. Specifically, the generation unit 133 calculates an average value of the similarity between each of the plurality of third feature amounts and the second feature amount as the similarity between the first information and the second information. Furthermore, the search unit 134 determines information related to the second information to be output as a search result based on the average value of the similarity.

[0045] 3. Information Processing Method From here, an example of information processing by the information processing device 100 according to the embodiment will be described with reference to FIGS.

[0046] First, an example of a common space generation process by the information processing device 100 will be described in detail with reference to Fig. 3 and Fig. 4. Fig. 3 is a flowchart illustrating an example of a common space generation process according to an embodiment. Fig. 4 is a diagram for explaining an example of a common space generation process according to an embodiment. In Fig. 4, a case will be described where the first information is text and the second information is an image.

[0047] In Fig. 3, the acquiring unit 131 acquires pair data that is a set of first information and second information that is a type of information different from the first information and corresponds to the first information (step S101). For example, in Fig. 4, the acquiring unit 131 acquires pair data that is a set of text T1 and image G1 that corresponds to the text T1. For example, the text T1 may be an explanatory text that explains the contents of the image G1.

[0048] Also, in FIG. 3, the extraction unit 132 extracts a first feature from the first information, which is a feature corresponding to the first information, and extracts a second feature from the second information, which is a feature corresponding to the second information (step S102). For example, in FIG. 4, the extraction unit 132 may input the text T1 acquired by the acquisition unit 131 to a trained text encoder E1 (hereinafter, abbreviated as "text encoder E1") to acquire a text feature F11 output from the text encoder E1. As shown in FIG. 4, the text feature F11 is mapped to a text feature space SP1 corresponding to the text feature generated by the text encoder E1. Also, the extraction unit 132 may input the image G1 acquired by the acquisition unit 131 to a trained image encoder E2 (hereinafter, abbreviated as "image encoder E2") to acquire an image feature F21 output from the image encoder E2. As shown in FIG. 4, the image feature F21 is mapped to an image feature space SP2 corresponding to the image features generated by the image encoder E2.

[0049] 3, the generating unit 133 generates a plurality of third features corresponding to the first information from the first features using a generating model M1 that is a machine learning model that outputs output information that follows a specific probability distribution in response to input of input information, and maps the plurality of third features and the second feature to a common space that is a common feature space (step S103). Specifically, the generating unit 133 generates the generating model M1 that is trained so that the similarity between each of the plurality of third features and the second feature is below a predetermined threshold.

[0050] For example, in FIG. 4, the generating unit 133 uses the generation model M1 to generate third features F311 to F341, which are four pseudo features corresponding to the text T1, from the text feature F11 extracted by the extracting unit 132. Note that, in FIG. 4, a case where the generating unit 133 generates four third features F311 to F341 will be described, but the generating unit 133 may generate any number of third features as long as the number is two or more. Next, the generating unit 133 may map the four generated third features F311 to F341 and the image feature F21 to a common space. Specifically, the generating unit 133 may map the four third features F311 to F341 and the image feature F21 to the image feature space SP2. More specifically, the generating unit 133 may train the generative model M1 so that the similarities SM311-SM341 between the four third features F311-F341 and the image feature F21 are below a predetermined threshold. In this manner, the generating unit 133 may map the four third features F311-F341 and the image feature F21 into the image feature space SP2, which is a common space.

[0051] Although not shown in the figure, the generating unit 133 may similarly generate a fourth feature, which is a plurality of pseudo features corresponding to the image G1, from the image feature F21 extracted by the extracting unit 132, using the generating model M2. Then, the generating unit 133 may map the generated plurality of fourth features, the four third features F311 to F341, and the image feature F21 to an image feature space SP2 that is a common space. For example, the generating unit 133 may train the generating model M2 so that the similarity between each of the plurality of fourth features and the image feature F21 is below a predetermined threshold. Alternatively, the generating unit 133 may calculate the average value of the four third features F311 to F341 mapped to the common space. Then, the generating unit 133 may train the generating model M2 so that the similarity between each of the plurality of fourth features and the average value of the four third features F311 to F341 is below a predetermined threshold. In this manner, the generation unit 133 may map the plurality of fourth feature amounts, the four third feature amounts F311 to F341, and the image feature amount F21 into the image feature amount space SP2, which is a common space.

[0052] Next, an example of search processing by the information processing device 100 will be described in detail with reference to Figs. 5 and 6. Fig. 5 is a flowchart showing an example of search processing according to the embodiment. Fig. 6 is a diagram for explaining an example of search processing according to the embodiment. In Fig. 6, a case where the first information is text and the second information is an image, similar to Fig. 4, will be explained. Also, in Fig. 6, an example of image-text search processing by the information processing device 100 will be explained.

[0053] In Fig. 5, the search unit 134 acquires a query received from a user (step S201). For example, in Fig. 6, the search unit 134 acquires text T2, which is the query received from the user.

[0054] Also, in Fig. 5, when the search unit 134 acquires a text query, it acquires a plurality of images (step S202). For example, in Fig. 6, when the search unit 134 acquires text T2 that is a query, it acquires a plurality of images from the storage unit 120 that stores a plurality of images. In Fig. 6, the plurality of images is indicated as an image pool.

[0055] Also, in Fig. 5, the extraction unit 132 extracts text features from the query acquired by the search unit 134, and extracts image features from each image (step S203). For example, in Fig. 6, the extraction unit 132 extracts text features F12 from text T2, which is the query. Also, the extraction unit 132 randomly selects one image from the multiple images. For example, the extraction unit 132 selects image G2 from the multiple images. Then, the extraction unit 132 extracts image features F22 from image G2.

[0056] 5, the generation unit 133 uses a trained generation model M1 (hereinafter, abbreviated as "generation model M1") to generate a plurality of third features from the text feature extracted by the extraction unit 132 (step S204). For example, in FIG. 6, the generation unit 133 uses the generation model M1 to generate four third features F312 to F342 from the text feature F12 extracted by the extraction unit 132. As described in FIG. 4, the generation unit 133 maps the four third features F312 to F342 and the image feature F22 to a common space.

[0057] Also, in FIG. 5, the generating unit 133 calculates an average value of similarities between each of the multiple third feature amounts generated by the generating unit 133 and the image feature amount corresponding to each image as the similarity between the query acquired by the search unit 134 and each image (step S205). For example, in FIG. 6, the generating unit 133 calculates an average value of similarities between each of the four third feature amounts F312 to F342 generated by the generating unit 133 and the image feature amount F22 as the similarity between the text T2, which is the query acquired by the search unit 134, and the image G2. In the same manner, the generating unit 133 may calculate an average value of similarities between each of the four third feature amounts F312 to F342 generated by the generating unit 133 and the image feature amount corresponding to one image randomly selected from the multiple images. In this way, the generating unit 133 may calculate an average value of similarities between each of the four third feature amounts F312 to F342 generated by the generating unit 133 and the image feature amount corresponding to each image.

[0058] Also, in Fig. 5, the search unit 134 determines the ranking of images to be output as search results based on the average value of similarity calculated by the generation unit 133 (step S206). For example, in Fig. 6, the search unit 134 estimates that the similarity between the text T2, which is the query, and the image G2 is the highest since the average value of similarity between each of the four third features F312 to F342 and the image feature F22 is the highest among the average values ​​of similarity calculated by the generation unit 133. Therefore, in Fig. 6, the search unit 134 determines to output the image G2 first as a search result corresponding to the text T2, which is the query.

[0059] 4. Modifications The processing according to the above-described embodiment may be implemented in various different forms other than the above embodiment.

[0060] In the above-described embodiment, the information processing device 100 executes an image-text search using a generative model that has learned a many-to-many correspondence between different modals. However, even if a generative model that has learned a many-to-many correspondence between different modals is used, the ranking of search results may differ depending on the user (also called the searcher) who performs the search. For example, for a text query of "photograph of a giraffe and a child", it is considered that which of an image associated with the text "photograph of a giraffe and a child in the shade of a tree" and an image associated with the text "photograph of a giraffe and a child at the zoo" should be ranked higher in the search results depends on the user. Therefore, in the modified example, a case will be described in which the information processing device 100 displays the search results described in the above-described embodiment, and then tunes the generative model for each user using a pair of a search result and a query selected by the user from the search results as learning data.

[0061] Specifically, the acquiring unit 131 acquires user pair data that is a pair of the first information and the second information that is a search result selected by the user from the search results. In the example of Fig. 6, the acquiring unit 131 may acquire a pair of the text T2 that is a query received from the user and an image that is a search result selected by the user from among a plurality of search results.

[0062] Furthermore, the generation unit 133 generates a retrained generation model for each user based on the user pair data acquired by the acquisition unit 131. For example, the acquisition unit 131 may acquire a plurality of different user pair data. Furthermore, the generation unit 133 may generate a generation model M1' personalized for each user by retraining the generation model M1 based on a plurality of different user pair data. For example, the extraction unit 132 may extract a text feature from the text included in the user pair data. Furthermore, the extraction unit 132 may extract an image feature from the image included in the user pair data. Furthermore, the generation unit 133 may generate a plurality of third feature values ​​from the text feature values ​​corresponding to the text included in the user pair data using the generation model M1. Subsequently, the generation unit 133 may map the plurality of third feature values ​​and the image feature values ​​to a common space. Specifically, the generation unit 133 may generate a generation model M1' retrained such that the similarity between each of the plurality of third feature values ​​and the second feature value is below a predetermined threshold.

[0063] 5. Effects As described above, the information processing device 100 according to the embodiment includes an acquisition unit 131, an extraction unit 132, and a generation unit 133. The acquisition unit 131 acquires pair data that is a set of first information and second information, which is information of a different type from the first information and corresponds to the first information. The extraction unit 132 extracts a first feature amount that is a feature amount corresponding to the first information from the first information, and extracts a second feature amount that is a feature amount corresponding to the second information from the second information. The generation unit 133 generates a plurality of third feature amounts corresponding to the first information from the first feature amount using a generative model that is a machine learning model that outputs output information that follows a specific probability distribution in response to input of input information, and maps the plurality of third feature amounts and the second feature amount to a common space that is a common feature amount space.

[0064] Thereby, the information processing device 100 can map the third feature, which is a plurality of pseudo feature generated from the first feature, to the second feature in a common space in association with each other. That is, the information processing device 100 can generate a common space that reflects the correspondence between a plurality of pieces of information similar to the first information and the second information. Similarly, the information processing device 100 can map the fourth feature, which is a plurality of pseudo feature generated from the second feature, to the third feature, which is a plurality of pseudo feature generated from the first feature, in association with each other in a common space. That is, the information processing device 100 can generate a common space that reflects the correspondence between a plurality of pieces of information similar to the second information and a plurality of pieces of information similar to the first information. Therefore, the information processing device 100 can generate a common space that reflects the correspondence between a plurality of pieces of information between different modalities. In addition, the information processing device 100 can generate a common space that reflects the correspondence between multiple pieces of information across different modalities, thereby contributing to the achievement of Goal 9 of the Sustainable Development Goals (SDGs), which is to "build resilient infrastructure, promote inclusive and sustainable industry and technological innovation."

[0065] Furthermore, the generation unit 133 generates a generative model that is trained so that the similarity between each of the multiple third feature amounts and the second feature amount is below a predetermined threshold.

[0066] Thereby, the information processing device 100 can map the third feature, which is a plurality of pseudo feature generated from the first feature, to the second feature in a common space in association with each other. That is, the information processing device 100 can generate a common space that reflects the correspondence between a plurality of pieces of information similar to the first information and the second information. Similarly, the information processing device 100 can map the fourth feature, which is a plurality of pseudo feature generated from the second feature, to the third feature, which is a plurality of pseudo feature generated from the first feature, in association with each other in a common space. That is, the information processing device 100 can generate a common space that reflects the correspondence between a plurality of pieces of information similar to the second information and a plurality of pieces of information similar to the first information. Therefore, the information processing device 100 can generate a common space that reflects the correspondence between a plurality of pieces of information between different modalities.

[0067] The information processing device 100 further includes a search unit 134. The search unit 134 searches for second information corresponding to the first information from among the plurality of pieces of second information. The generation unit 133 calculates an average value of the similarity between each of the plurality of third feature amounts and the second feature amount as the similarity between the first information and the second information. The search unit 134 determines information related to the second information to be output as a search result based on the average similarity.

[0068] This allows the information processing device 100 to search for information across different modalities using a common space that reflects the corresponding relationships between multiple pieces of information across different modalities, thereby improving the accuracy of ranking search results.

[0069] The acquisition unit 131 also acquires user pair data that is a pair of the first information and the second information that is a search result selected by the user from the search results. The generation unit 133 generates a re-trained generative model for each user based on the user pair data.

[0070] This enables the information processing device 100 to improve the accuracy of ranking search results according to the user who performs the search (also called the searcher).

[0071] Furthermore, the pair of first information and second information is a pair of image and text, a pair of video and text, a pair of audio and video, a pair of text and audio, or a pair of two different types of language information.

[0072] This enables the information processing device 100 to generate a common space that reflects the correspondence between multiple pieces of information in different modalities, such as images and text, videos and text, audio and video, text and audio, or two different types of linguistic information.

[0073] [6. Hardware Configuration] Moreover, the information processing device 100 according to the embodiment described above is realized by a computer 1000 having a configuration as shown in Fig. 7, for example. Fig. 7 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device 100. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, a HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

[0074] The CPU 1100 operates and controls each unit based on a program stored in the ROM 1300 or the HDD 1400. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started up, programs that depend on the hardware of the computer 1000, and the like.

[0075] The HDD 1400 stores programs executed by the CPU 1100, data used by such programs, etc. The communication interface 1500 receives data from other devices via a predetermined communication network and sends it to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the predetermined communication network.

[0076] The CPU 1100 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse, via the input / output interface 1600. The CPU 1100 acquires data from the input devices via the input / output interface 1600. The CPU 1100 also outputs generated data to the output devices via the input / output interface 1600.

[0077] The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0078] For example, when the computer 1000 functions as the information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes programs loaded onto the RAM 1200 to realize the functions of the control unit 130. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from another device via a predetermined communication network.

[0079] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be embodied in other forms that incorporate various modifications and improvements based on the knowledge of those skilled in the art, including the forms described in the Disclosure of the Invention section.

[0080] [7. Other] Furthermore, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

[0081] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0082] For example, in the above-described embodiment, the information processing device 100 has the acquisition unit 131, the extraction unit 132, the generation unit 133, and the search unit 134 as functional units, but each unit can be configured to be distributed in a separate device. For example, the information processing device 100 can have the acquisition unit 131, the extraction unit 132, and the generation unit 133 as functional units as an information processing device that generates the generative model M1. In addition, an information processing device other than the information processing device 100 (hereinafter, referred to as a "search device") can have the acquisition unit 131, the extraction unit 132, the generation unit 133, and the search unit 134 as functional units as a device that executes a search. At this time, the information processing device 100 and the search device may be connected to various networks by wire or wirelessly and may transmit and receive information to each other. For example, the search device may receive information about the generative model generated by the information processing device 100 from the information processing device 100.

[0083] Furthermore, the above-described embodiments and modifications can be appropriately combined as long as the processing contents are not contradictory. [Explanation of symbols]

[0084] 100 Information processing device 110 Communications Department 120 Storage section 130 Control section 131 Acquisition Department 132 Extraction part 133 Generation part 134 Search Department

Claims

1. an acquisition unit that acquires pair data that is a set of first information and second information that is information of a different type from the first information and corresponds to the first information; an extraction unit that extracts a first feature amount that is a feature amount corresponding to the first information from the first information and extracts a second feature amount that is a feature amount corresponding to the second information from the second information; a generation unit that generates a plurality of third features corresponding to the first information from the first features using a generation model that is a machine learning model that outputs output information following a specific probability distribution in response to input of input information, and maps the plurality of third features and the second feature into a common space that is a common feature space; An information processing device comprising:

2. The generation unit is generating the generative model trained so that the similarities between each of the third feature amounts and the second feature amount are below a predetermined threshold; The information processing device according to claim 1 .

3. a search unit that searches for the second information corresponding to the first information from among a plurality of the second information; The generation unit is calculating an average value of the similarities between the second feature amount and each of the third feature amounts as a similarity between the first information and the second information; The search unit is determining information about the second information to be output as a search result based on the average similarity score; The information processing device according to claim 1 .

4. The acquisition unit is acquiring user pair data which is a pair of the second information, which is a search result selected by a user from the search results, and the first information; The generation unit is generating the generative model for each user that is retrained based on the user pair data; The information processing device according to claim 3 .

5. The pair of the first information and the second information is a pair of an image and a text, a pair of a video and a text, a pair of an audio and a video, a pair of a text and an audio, or a pair of two different kinds of language information. The information processing device according to claim 1 .

6. An information processing method implemented by a program executed by an information processing device, comprising: an acquisition step of acquiring a set of data of first information and second information, which is information of a different type from the first information and corresponds to the first information; an extraction step of extracting a first feature amount corresponding to the first information from the first information and extracting a second feature amount corresponding to the second information from the second information; a generation process of generating a plurality of third features corresponding to the first information from the first features using a generation model that is a machine learning model that outputs output information following a specific probability distribution in response to input of input information, and mapping the plurality of third features and the second feature into a common space that is a common feature space; An information processing method comprising:

Images