Information processing device, information processing method, and information processing program

Abstract

In the present invention, a related information search unit (112) acquires a related information cluster, which is a plurality of related information related to input information. For each piece of related information included in the related information cluster, a similarity calculation unit (130) calculates the similarity of the related information to the input information as an input similarity, calculates the similarity of the related information to another related information cluster as an inter-information similarity, and calculates the similarity of the related information to an information cluster of another modality as an inter-modality similarity. A ranking unit (113) ranks the related information cluster on the basis of the input similarity and the inter-modality similarity of each piece of the related information in the related information cluster. An output information generation unit (122) generates output information in response to the input information on the basis of the related information cluster that has been ranked.

Description

Information processing device, information processing method, and information processing program

[0001] The present disclosure relates to question answering technology.

[0002] There are known systems that use generative models to respond to user questions. When using a generative model to respond to user questions, it is difficult to generate an appropriate response if the information asked by the user is not included in the knowledge pre-trained by the generative model.

[0003] The following system is known that can handle inputs that query knowledge other than that learned by the generative model. The system searches external knowledge to obtain a set of information related to the input, and inputs the set of related information into the generative model along with the input.

[0004] When an external knowledge source that only has text information is used, it may not be able to respond appropriately to user input. However, there are few techniques for converting multiple types of modalities into external knowledge for interactive question-answering tasks.

[0005] Non-Patent Document 1 proposes a technology for providing answers using multimodal external knowledge. Non-Patent Document 1 discloses the following method: a database for storing multimodal external knowledge is prepared, text, images, and table data are vectorized for search purposes, and a response sentence is generated based on the search results.

[0006] In the method of Non-Patent Document 1, information similar to the user's input sentence is ranked higher in the search results, so similar information is likely to be output as the search results. However, in order to generate an appropriate response to the user's input sentence, a variety of information needs to be included in the search results. The method of Non-Patent Document 1 cannot address this diversity of search results.

[0007] Smaranda Muresan, Preslav Nakov, Aline Villavicencio, “MMCoQA: Conversational Question Answering over Text, Tables, and Images” In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022. (The Hong Kong Polytechnic University, Shandong University)

[0008] The present disclosure aims to enable generation of output information that takes into account the diversity of search results by utilizing external knowledge related to a user's input.

[0009] The information processing device of the present disclosure includes a related information search unit that acquires a group of related information, which is a plurality of pieces of related information related to information input by a user; a ranking unit that ranks information necessary for responding to the user from the group of related information in a higher order; and an output information generation unit that generates output information, which is the response, based on the ranked group of related information.

[0010] According to the present disclosure, by utilizing external knowledge related to a user's input, it is possible to generate output information that takes into account the diversity of search results.

[0011] 1. A configuration diagram of a question answering system 200 according to the first embodiment. A configuration diagram of a response device 100 according to the first embodiment. A functional configuration diagram of the response device 100 according to the first embodiment. A configuration diagram of a search model storage unit 191 according to the first embodiment. A configuration diagram of an external knowledge storage unit 192 according to the first embodiment. A configuration diagram of a dialogue data storage unit 193 according to the first embodiment. A configuration diagram of a search result storage unit 194 according to the first embodiment. A configuration diagram of a generative model storage unit 195 according to the first embodiment. A configuration diagram of a terminal device 210 according to the first embodiment. A flowchart of a question answering method according to the first embodiment. A flowchart of step S110 according to the first embodiment. A flowchart of step S120 according to the first embodiment. A flowchart of step S130 according to the first embodiment. A hardware configuration diagram of the response device 100 according to the first embodiment. A hardware configuration diagram of the terminal device 210 according to the first embodiment.

[0012] In the embodiments and drawings, the same or corresponding elements are denoted by the same reference numerals. The description of elements denoted by the same reference numerals as those already described will be omitted or simplified as appropriate. Arrows in the drawings primarily indicate the flow of data or the flow of processing.

[0013] First Embodiment A question answering system 200 will be described with reference to FIGS.

[0014] ***Description of Configuration*** The configuration of a question answering system 200 will be described with reference to Fig. 1. The question answering system 200 includes a terminal device 210 and a response device 100. The terminal device 210 and the response device 100 communicate with each other via a network 201. An example of the network 201 is the Internet.

[0015] A user's input (question) is transmitted from the terminal device 210 to the response device 100 , and an output (response) to the input is transmitted from the response device 100 to the terminal device 210 .

[0016] The question answering system 200 is an information processing system. The response device 100 is an information processing device.

[0017] The configuration of the response device 100 will be described with reference to FIG. 2 . The response device 100 is a computer including hardware such as a processor 101, a memory 102, an auxiliary storage device 103, a communication device 104, an input / output interface 105, and a media interface 106. These pieces of hardware are connected to each other via signal lines. Examples of computers include desktop computers, portable computers, embedded microcomputers, and SoCs. Examples of desktop computers include personal computers and server computers. Examples of portable computers include smartphones and tablet computers. SoC is an abbreviation for System on Chip.

[0018] The processor 101 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 101 is a CPU or a DSP. The processor 101 may be a single processor or a multiprocessor. IC is an abbreviation for Integrated Circuit. CPU is an abbreviation for Central Processing Unit. DSP is an abbreviation for Digital Signal Processor.

[0019] The memory 102 is a volatile or non-volatile storage device. The memory 102 is also called a primary storage device or a main memory. For example, the memory 102 is a RAM. Data stored in the memory 102 is saved in the secondary storage device 103 as needed. RAM is an abbreviation for Random Access Memory.

[0020] The auxiliary storage device 103 is a non-volatile storage device. For example, the auxiliary storage device 103 is a ROM, HDD, flash memory, SSD, or a combination thereof. Data stored in the auxiliary storage device 103 is loaded into the memory 102 as needed. ROM is an abbreviation for Read Only Memory. HDD is an abbreviation for Hard Disk Drive. SSD is an abbreviation for Solid State Drive.

[0021] The communication device 104 is a receiver and a transmitter. For example, the communication device 104 is a communication chip or NIC. The communication of the response device 100 is performed using the communication device 104. NIC is an abbreviation for Network Interface Card.

[0022] The input / output interface 105 is a port to which an input device and an output device are connected. For example, the input / output interface 105 is a USB terminal, the input devices are a keyboard and a mouse, and the output device is a display. Input and output of the response device 100 is performed using the input / output interface 105. USB is an abbreviation for Universal Serial Bus.

[0023] The media interface 106 is a port to which an external storage medium is connected. The media interface 106 reads programs and data stored in the external storage medium and provides the read programs and data to the processor 101 through the memory 102. Examples of external storage media include CDs, DVDs, and flash memories. CD is an abbreviation for Compact Disc. DVD is an abbreviation for Digital Versatile Disc.

[0024] The response device 100 includes elements such as a search unit 110 and an inference unit 120. These elements are realized by software.

[0025] The auxiliary storage device 103 stores a response program for causing the computer to function as the search unit 110 and the inference unit 120. The response program is loaded into the memory 102 and executed by the processor 101. The auxiliary storage device 103 also stores an OS. At least a portion of the OS is loaded into the memory 102 and executed by the processor 101. The processor 101 executes the inference program while executing the OS. OS is an abbreviation for Operating System.

[0026] Data (input data, output data, etc.) of the response program is stored in the storage unit 190. The auxiliary storage device 103 functions as the storage unit 190. However, storage devices such as the memory 102, a register in the processor 101, a cache memory in the processor 101, and an external storage medium may function as the storage unit 190 instead of or together with the memory 102.

[0027] The response program can be recorded (stored) in a computer-readable manner on a non-volatile recording medium such as an optical disk or flash memory.

[0028] The functional configuration of the response device 100 will be described with reference to Fig. 3. The search unit 110 includes an acquisition unit 111, a related information search unit 112, a similarity calculation unit 130, and a ranking unit 113. The similarity calculation unit 130 includes an input similarity calculation unit 131, an inter-information similarity calculation unit 132, and an inter-modal similarity calculation unit 133. The inference unit 120 includes an acquisition unit 121, an output information generation unit 122, and an output unit 123. The storage unit 190 includes a search model storage unit 191, an external knowledge storage unit 192, a dialogue data storage unit 193, a search result storage unit 194, and a generation model storage unit 195.

[0029] The configuration of the search model storage unit 191 will be described with reference to FIG. 4. The search model storage unit 191 stores multiple search models having different modal types. Each of the multiple search models is associated with a modal type. Examples of modal types include text, image, audio, table, and three-dimensional information. Examples of three-dimensional information include point clouds, meshes, and distance information. The search model vectorizes information for search. If the modal type is text, the search model is a model that vectorizes text information. If the modal type is audio, the search models include a model that converts audio information into text information, a model that vectorizes text information, and a model that vectorizes audio information in the same latent space as text. If the modal type is image, the search models include a model that converts image information into text, a model that vectorizes text information, and a model that vectorizes image information in the same latent space as text. A model that converts image information into text is called a captioning model. For example, the search model storage unit 191 stores a CLIP model and a GPT-4 model as search models whose modal type is image. CLIP is an abbreviation for Contrastive Language-Image Pre-Training. GPT is an abbreviation for Generative Pretrained Transformer. For example, the search model storage unit 191 stores the Whisper model and the GPT-4 model as search models whose modal type is speech. The Whisper model converts speech information into text information. The GPT-4 model vectorizes text information. The search models are acquired from the search model storage unit 191 by, for example, the acquisition unit 111.

[0030] Models are generated through various types of machine learning. Models are also called trained models or AI models. AI stands for artificial intelligence.

[0031] The configuration of the external knowledge storage unit 192 will be described with reference to FIG. 5 . The external knowledge storage unit 192 stores a plurality of external knowledge pieces of a plurality of modal types. Each of the plurality of external knowledge pieces is associated with a modal type of the external knowledge. The external knowledge piece is information other than information learned by a generative model, which will be described later. External knowledge pieces whose modal type is not text may be converted into text information and stored, or may be stored as modal type data. For example, if the external knowledge piece is an image of "Player X playing a game on August 1st," the text information "Player X playing a game on August 1st" may be stored as the external knowledge piece, or the image may be stored as the external knowledge piece. The external knowledge piece is acquired from the external knowledge storage unit 192 by, for example, the acquisition unit 111.

[0032] The configuration of the dialogue data storage unit 193 will be described with reference to FIG. 6 . The dialogue data storage unit 193 stores (past) dialogue data. The dialogue data includes multiple pieces of utterance information. Each piece of utterance information is associated with a speaker identifier and a modal type. The speaker identifier identifies the speaker. For example, the speaker identifier distinguishes between a user and a model. However, the speaker identifier may identify individual users and individual models. The user and the model do not need to alternate as speakers. For example, a user may speak continuously. The utterance information is information transmitted from a user or a model. Utterance information whose modal type is not text may be converted into text information and stored, or may be stored as modal type data as is. For example, an image or sound may be stored as the utterance information. The dialogue data is stored by, for example, the output unit 123. The output unit 123 stores the user's input information and the output information of the output information generation unit 122 as utterance information in the dialogue data storage unit 193. The dialogue data storage unit 193 is acquired from the dialogue data storage unit 193 by the acquisition unit 111, for example.

[0033] The configuration of the search result storage unit 194 will be described with reference to FIG. 7 . The search result storage unit 194 stores multiple search results for multiple modal types. Each of the multiple search results is associated with a modal type, a pre-score, and a post-score. The pre-score is the search score before re-ranking. The search score will be described later. The post-score is the search score after re-ranking. The search results are related information, which will be described later. The search results are stored by, for example, the ranking unit 113. For example, the ranking unit 113 stores related information, which will be described later, as the search results, stores input similarity, which will be described later, as the pre-score, and stores search score, which will be described later, as the post-score. The search results are acquired from the search result storage unit 194 by, for example, the acquisition unit 121.

[0034] The configuration of the generative model storage unit 195 will be described with reference to FIG. 8 . The generative model storage unit 195 stores generative models. The generative model storage unit 195 may store multiple generative models that generate information of different modal types. Each of the multiple generative models is associated with a modal type of the information to be generated. A generative model generates and outputs information of a specific modal type (such as text or image) in response to an input. A generative model is also referred to as a generative AI or a generative AI model. A generative model is acquired from the generative model storage unit 195 by, for example, the acquisition unit 121.

[0035] The configuration of the terminal device 210 will be described with reference to Fig. 9. The terminal device 210 is a computer equipped with hardware such as a processor 211, a memory 212, an auxiliary storage device 213, a communication device 214, and an input / output interface 215. These pieces of hardware are connected to each other via signal lines.

[0036] The processor 211 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 211 is a CPU or a DSP. The memory 212 is a volatile or non-volatile storage device. The memory 212 is also called a primary storage device or a main memory. For example, the memory 212 is a RAM. Data stored in the memory 212 is saved in the secondary storage device 213 as needed. The secondary storage device 213 is a non-volatile storage device. The secondary storage device 213 is also called storage. For example, the secondary storage device 213 is a ROM, a HDD, a flash memory, an SSD, or a combination thereof. Data stored in the secondary storage device 213 is loaded into the memory 212 as needed. The communication device 214 is a receiver and a transmitter. For example, the communication device 214 is a communication chip or a NIC. Communication of the terminal device 210 is performed using the communication device 214. The input / output interface 215 is a port to which an input device and an output device are connected. For example, the input / output interface 215 is a USB terminal, the input device is a keyboard and a mouse, and the output device is a display. Input and output to and from the terminal device 210 are performed using the input / output interface 215. The media interface 216 is a port to which an external storage medium is connected. The media interface 216 reads programs and data stored in the external storage medium and provides the read programs and data to the processor 211 via the memory 212.

[0037] The terminal device 210 includes elements such as a reception unit 221, a dialogue unit 222, and an output unit 223. These elements are realized by software.

[0038] The auxiliary storage device 213 stores a terminal program for causing the computer to function as a reception unit 221, a dialogue unit 222, and an output unit 223. The terminal program is loaded into the memory 212 and executed by the processor 211. The auxiliary storage device 213 also stores an OS. At least a portion of the OS is loaded into the memory 212 and executed by the processor 211. The processor 211 executes the terminal program while executing the OS.

[0039] Data for the terminal program is stored in the storage unit 229. The memory 212 functions as the storage unit 229. However, a storage device such as the auxiliary storage device 213, a register in the processor 211, a cache memory in the processor 211, or an external storage medium may function as the storage unit 229 instead of or together with the memory 212.

[0040] The terminal program can be recorded (stored) in a computer-readable manner on a non-volatile recording medium such as an optical disk or flash memory.

[0041] ***Explanation of Operation*** The operation procedure of the question answering system 200 corresponds to a question answering method. The operation procedure of the response device 100 corresponds to a response method. The operation procedure of the terminal device 210 corresponds to a question method. The question answering method includes a question method and a response method. Each of the question answering method, question method, and response method is an information processing method.

[0042] The question answering method will be described with reference to Fig. 10. In step S101, the user inputs utterance information indicating a question to the terminal device 210. The input utterance information is referred to as input information. The receiving unit 221 of the terminal device 210 receives the input information.

[0043] In step S102, the dialogue unit 222 of the terminal device 210 transmits input data to the response device 100. The input data is communication data including input information.

[0044] In step S103 , the acquisition unit 111 of the response device 100 receives the input data transmitted from the terminal device 210 .

[0045] In step S110, the output information generating unit 122 of the response device 100 generates output information in response to the input information. The output information is speech information indicating a response to the question indicated in the input information. Step S110 will be described in detail later.

[0046] In step S104, the output unit 123 of the response device 100 transmits output data to the terminal device 210. The output data is communication data including output information.

[0047] In step S105 , the dialogue unit 222 of the terminal device 210 receives the output data transmitted from the response device 100 .

[0048] In step S106, the output unit 223 of the terminal device 210 outputs the output information. For example, the output unit 223 displays the output information on the display of the terminal device 210.

[0049] The procedure of step S110 will be described with reference to Fig. 11. In step S111, the related information search unit 112 acquires a group of related information for the input information. The group of related information includes a plurality of pieces of related information of a plurality of modal types. The related information is information related to the input information.

[0050] The group of related information for the input information is acquired by using the input information to search the external knowledge storage unit 192. A plurality of pieces of external knowledge found in the search become the group of related information for the input information.

[0051] A group of related information for input information is acquired in the following procedure. First, the acquisition unit 111 acquires a search model for the modal type of the input information from the search model storage unit 191, and the related information search unit 112 vectorizes the input information using the acquired search model. The vectorized input information is referred to as vectorized input information. Furthermore, for each piece of external knowledge stored in the external knowledge storage unit 192, the acquisition unit 111 acquires a search model for the modal type of the external knowledge from the search model storage unit 191, and the related information search unit 112 vectorizes the external knowledge using the acquired search model. The vectorized external knowledge is referred to as vectorized external knowledge. Then, the related information search unit 112 performs a search on a set of vectorized external knowledge using the vectorized input information. The multiple pieces of vectorized external knowledge found in the search or the multiple pieces of external knowledge corresponding to the multiple vectorized external knowledge found in the search become a group of related information for the input information.

[0052] In step S120, the ranking unit 113 calculates a search score for each piece of related information. The search score is also called a similarity score.

[0053] The procedure of step S120 will be described with reference to Fig. 12. In step S121, the similarity calculation unit 130 selects one piece of related information that has not been selected yet from the group of related information for the input information.

[0054] In step S122, the input similarity calculation unit 131 calculates the input similarity of the selected related information. The input similarity is the similarity between the input information and the related information.

[0055] For example, the input similarity calculation unit 131 calculates the cosine similarity between the input information and the related information as the input similarity.

[0056] For example, related information D for input information Q ｉ Input similarity Sim １ (D ｉ , Q) can be expressed by equation (1). "Vec(x)" represents vectorized information x. "Vec" represents a vectorized model (search model).

[0057]

[0058] In step S123, the information similarity calculation unit 132 calculates the information similarity of the selected related information. The information similarity is the similarity between the related information and the other related information group. The other related information group is one or more pieces of related information other than the target related information in the related information group. Specifically, the other related information group is one or more pieces of related information for which a search score has already been calculated. Each piece of related information included in the other related information group is referred to as other related information.

[0059] For example, the inter-information similarity is calculated as follows. First, the inter-information similarity calculation unit 132 calculates the cosine similarity between the selected related information and the other related information for each piece of other related information. Then, the inter-information similarity calculation unit 132 calculates the inter-information similarity based on the calculated cosine similarities. For example, the inter-information similarity calculation unit 132 determines the maximum cosine similarity among the calculated cosine similarities as the inter-information similarity.

[0060] For example, the selected related information D ｉ Similarity between information Sim ２ (D ｉ, S) can be expressed by the formula (2). "S" represents a group of other related information. "D Ｓ " indicates other related information.

[0061]

[0062] The inter-information similarity of each piece of related information may be calculated in descending order of input similarity, or may be calculated in random order and then sorted in descending order of input similarity.

[0063] In step S124, the inter-modal similarity calculation unit 133 calculates the inter-modal similarity of the selected related information. The inter-modal similarity is the similarity between the related information and the multi-modal information group. The multi-modal information group is one or more pieces of related information in the related information group whose modal type is different from that of the target related information. Specifically, the multi-modal information group is one or more pieces of related information whose modal type is different from that of the target related information among one or more pieces of related information for which search scores have already been calculated. Each piece of related information included in the multi-modal information group is referred to as multi-modal information.

[0064] For example, the inter-modal similarity is calculated as follows. First, the inter-modal similarity calculation unit 133 compares the modal type of the selected related information with the modal type of each other related information. The modal type of each piece of related information is obtained from the external knowledge storage unit 192. Next, the inter-modal similarity calculation unit 133 extracts a group of other modal information from the group of related information based on the comparison result. Next, the inter-modal similarity calculation unit 133 calculates the cosine similarity between the selected related information and the other modal information for each piece of other modal information. Then, the inter-modal similarity calculation unit 133 calculates the inter-modal similarity based on the calculated cosine similarities. For example, the inter-modal similarity calculation unit 133 determines the maximum cosine similarity among the calculated cosine similarities as the inter-modal similarity.

[0065] For example, the selected related information D ｉ Inter-modal similarity Sim ３ (D ｉ , S Ｎ ) can be expressed by the formula (3).Ｎ " represents a multimodal information group. ｎ " represents multimodal information.

[0066]

[0067] Steps S122 to S124 may be executed in reverse order.

[0068] In step S125, the ranking unit 113 calculates a search score for the selected related information based on the input similarity, inter-information similarity, and inter-modal similarity of the selected related information.

[0069] For example, the ranking unit 113 calculates the search score by calculating a calculation formula using the input similarity, the inter-information similarity, and the inter-modal similarity.

[0070] For example, the selected related information D ｉ Search score Sim(D ｉ ) can be expressed by Equation (4), where "k" and "l" represent hyperparameters.

[0071]

[0072] In step S126, the similarity calculation unit 130 determines whether there is any unselected related information. If there is any unselected related information, the process proceeds to step S121. If there is no unselected related information, the process of step S120 ends.

[0073] 11, the description will continue from step S112. In step S112, the ranking unit 113 ranks the related information group based on the search score of each piece of related information in the related information group.

[0074] For example, the ranking unit 113 ranks the multiple pieces of related information that make up the related information group in descending order of search score.

[0075] In step S130, the inference unit 120 generates output information for the input information based on the ranked related information group.

[0076] The output information can be generated using a generative model with the input information and the ranked related information group as input.

[0077] The procedure of step S130 will be described with reference to Fig. 13. In step S131, the output information generation unit 122 vectorizes the input information. The output information generation unit 122 also vectorizes each piece of related information in the related information group.

[0078] The input information and each related information are vectorized as follows: The acquisition unit 121 acquires a search model for the modal type of the input information from the search model storage unit 191, and the output information generation unit 122 vectorizes the input information using the acquired search model. For each related information, the acquisition unit 121 acquires a search model for the modal type of the related information from the search model storage unit 191, and the output information generation unit 122 vectorizes the related information using the acquired search model.

[0079] The vectorized input information is referred to as "vectorized input information." A related information group made up of vectorized related information is referred to as "vectorized related information group."

[0080] In step S132, the output information generation unit 122 selects a generative model to be used.

[0081] Specifically, a generative model that generates information of the same modal type as the modal type requested as the modal type of the output information is selected. For example, if an image output is requested, a generative model that generates an image is selected. Examples of generative models that generate images include image generation models such as GPT-4 and Dall-E3.

[0082] For example, the modal type required as the modal type of the output information is determined based on the input information.

[0083] In step S133, the output information generator 122 uses the vectorized input information and the ranked vectorization-related information group as inputs to the selected generative model, and executes inference using the selected generative model, thereby generating output information of the required modal type.

[0084] ***Effects of First Embodiment*** The first embodiment relates to a question answering technology. In particular, the first embodiment relates to a multimodal question answering device that complementarily combines multimodal information to respond appropriately to a user's input for a question that requires multimodal information as an external knowledge source.

[0085] In the first embodiment, an algorithm is proposed for reranking information related to a user input while preserving the diversity of the information. One of the objectives of the first embodiment is to effectively acquire information necessary for responding to a user input and generate a response.

[0086] The response device 100 includes an external knowledge storage unit 192, a related information search unit 112, an input similarity calculation unit 131, an inter-information similarity calculation unit 132, an inter-modal similarity calculation unit 133, a ranking unit 113, and an output information generation unit 122. The external knowledge storage unit 192 stores a plurality of modal knowledge. The related information search unit 112 searches for information related to an input. The input similarity calculation unit 131 calculates the similarity between each piece of related information and dialogue data. The inter-information similarity calculation unit 132 calculates the similarity between each piece of related information and other related information and obtains the maximum value. The inter-modal similarity calculation unit 133 calculates the similarity between each piece of related information and knowledge of a different modality and obtains the maximum value. The ranking unit 113 changes the ranking of the search results obtained by the related information search unit 112 (reranking). The output information generation unit 122 inputs the dialogue data and related information into a generation model to generate a response.

[0087] When determining the reranking score, the response device 100 calculates the similarity between each piece of information included in the subset of scored information for reranking and the target information, and obtains the maximum value. By subtracting the maximum similarity value from the reranking score, it is possible to prevent similar information from being ranked high multiple times. Note that instead of subtracting the maximum similarity value, for example, a predetermined value may be added to the maximum value, or the maximum value may be multiplied by a predetermined value.

[0088] When determining the reranking score, the response device 100 calculates the similarity between the target information and each piece of information of a modality different from the target information that is included in the subset of information that has already been scored for reranking, and obtains the maximum value of the similarities. By adding this maximum similarity to the reranking score, it is possible to search for information that can achieve a high reranking score only when combined with other modal information. In other words, a search that takes into account the diversity and modal information of the search results is possible. Therefore, the response device 100 can generate output information that takes into account the diversity of the search results by utilizing external knowledge (of a different modal type) related to the user's input. In other words, the response device 100 can obtain search results that maintain diversity in terms of the semantics and modality of the information from external knowledge based on the user's input, and generate output information using the search results.

[0089] The method of Non-Patent Document 1 cannot take into account information complementarity between modals. For example, for an input sentence such as "What color were Player X's gloves when he played against Team Y?", much of the information that can be obtained is text information similar to the input sentence, such as "Player X played against Team Y on August 1st." Therefore, an image depicting "Player X playing against Team Y on August 1st" may not be ranked as a top search result input to the generative model, even though it contains information that reveals the color of the gloves. Therefore, it is necessary to obtain the information "Player X played against Team Y on August 1st" from the text information obtained from the sentence input by the user, and rerank the search results for images depicting "Player X playing against Team Y on August 1st." The method of Embodiment 1 enables such reranking.

[0090] *** Supplementary Note to First Embodiment *** The response device 100 stores dialogue data between the user and the response device 100, and in response to new input from the user, searches external knowledge for information related to the input data based on past dialogue data and the user's new input information. In addition, the response device 100 reranks search results by combining the similarity between the input data and the search results, the similarity between the search results and other search results, and the similarity between the search results and search results of different modalities. The response device 100 then inputs the reranked results and the dialogue data into a generative model to generate response information to the user's input data.

[0091] The response device 100 includes a dialogue data storage unit 193, an external knowledge storage unit 192, a related information search unit 112, a ranking unit 113, and an output information generation unit 122. The dialogue data storage unit 193 receives input from a user. The external knowledge storage unit 192 stores knowledge obtained from multiple modalities. The related information search unit 112 searches for information necessary to respond to the input from the user. The ranking unit 113 ranks necessary information at the top. The output information generation unit 122 generates a response to the user.

[0092] The ranking unit 113 calculates a score using the similarity between the dialogue information between the user and the system and the external knowledge, the maximum value of the similarity between the previously-published search results and the information to be re-ranked, and the inter-modal similarity when the information to be re-ranked and the previously-published search results have different modals.The ranking unit 113 then re-ranks the search results using the calculated score.

[0093] The question answering system 200 may include a plurality of terminal devices 210. Each of the plurality of terminal devices 210 communicates with the response device 100 via the network 201.

[0094] The external knowledge search method may be an automatic search method, a token-based search method, or the like. The automatic search method is a method based on a machine learning method or a pre-trained model. The token-based search method is a method such as TF-IDF (Term Frequency-Inverse Document Frequency). When a model based on machine learning or the like is used, the model is stored in the search model storage unit 191. The external knowledge search method is performed by replacing all modals with single modal information and using a vectorization model of that modal type in order to search across different modals. Alternatively, the external knowledge search method is performed by using a model that can vectorize different modals into the same latent space. The model for search is stored in the search model storage unit 191.

[0095] The input similarity calculation unit 131 may include past dialogue information in the input data Q in addition to the input information.

[0096] The search score is calculated based on the input similarity, the inter-information similarity, and the inter-modal similarity.

[0097] The search model storage unit 191, the external knowledge storage unit 192, the dialogue data storage unit 193, the search result storage unit 194, and the generative model storage unit 195 may each be provided externally. These "storage units" may also be read as "databases."

[0098] The search model may be a model that converts input information and each piece of external knowledge into modal type information other than text and vectorizes it.

[0099] The speech information may be replaced with information other than speech (question, response).

[0100] The input similarity, information similarity, and modal similarity may be calculated using methods other than cosine similarity. For example, known methods such as Pearson's correlation coefficient and deviation pattern similarity may be used.

[0101] The operation procedures of the question answering system 200 correspond to the processing procedures by the question answering program. The operation procedures of the response device 100 correspond to the processing procedures by the response program. The operation procedures of the terminal device 210 correspond to the processing procedures by the terminal program. The question answering program includes a terminal program and a response program. The question answering program, the terminal program, and the response program are each an information processing program. The terminal program is also called a question program.

[0102] The hardware configuration of the response device 100 will be described with reference to Fig. 14. The response device 100 includes a processing circuit 109. The processing circuit 109 is hardware that realizes a search unit 110 and an inference unit 120. The processing circuit 109 may be dedicated hardware, or may be a processor 101 that executes a program stored in a memory 102.

[0103] When the processing circuit 109 is dedicated hardware, the processing circuit 109 may be, for example, a single circuit, a multiple circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof. ASIC is an abbreviation for Application Specific Integrated Circuit. FPGA is an abbreviation for Field Programmable Gate Array.

[0104] The response device 100 may include multiple processing circuits that replace the processing circuit 109 .

[0105] In the processing circuit 109, some functions may be realized by dedicated hardware, and the remaining functions may be realized by software or firmware.

[0106] Thus, the functionality of the response device 100 can be implemented in hardware, software, firmware, or a combination thereof.

[0107] The hardware configuration of the terminal device 210 will be described with reference to Fig. 15. The terminal device 210 includes a processing circuit 219. The processing circuit 219 is hardware that implements a reception unit 221, a dialogue unit 222, and an output unit 223. The processing circuit 219 may be dedicated hardware, or may be a processor 211 that executes a program stored in a memory 212.

[0108] When processing circuitry 219 is dedicated hardware, processing circuitry 219 may be, for example, a single circuit, multiple circuits, a programmed processor, parallel programmed processors, an ASIC, an FPGA, or a combination thereof.

[0109] The terminal device 210 may include multiple processing circuits that replace the processing circuit 219.

[0110] In the processing circuit 219, some functions may be realized by dedicated hardware, and the remaining functions may be realized by software or firmware.

[0111] Thus, the functions of the terminal device 210 can be realized by hardware, software, firmware, or a combination of these.

[0112] The first embodiment is an example of a preferred embodiment and is not intended to limit the technical scope of the present disclosure. The first embodiment may be implemented in part or in combination with other embodiments. The procedures described using flowcharts and the like may be modified as appropriate.

[0113] The "part" of each element of the response device 100 and the terminal device 210 may be read as "processing," "step," "circuit," or "circuitry."

[0114] 100 Response device, 101 Processor, 102 Memory, 103 Auxiliary storage device, 104 Communication device, 105 Input / output interface, 106 Media interface, 109 Processing circuit, 110 Search unit, 111 Acquisition unit, 112 Related information search unit, 113 Ranking unit, 120 Inference unit, 121 Acquisition unit, 122 Output information generation unit, 123 Output unit, 130 Similarity calculation unit, 131 Input similarity calculation unit, 132 Inter-information similarity calculation unit, 133 Inter-modal similarity calculation unit, 190 Storage unit, 191 Search model storage unit, 192 External knowledge storage unit, 193 Dialogue data storage unit, 194 Search result storage unit, 195 Generative model storage unit, 200 Question answering system, 201 Network, 210 Terminal device, 211 Processor, 212 Memory, 213 Auxiliary storage device, 214 communication device, 215 input / output interface, 216 media interface, 219 processing circuit, 221 reception unit, 222 dialogue unit, 223 output unit.

Claims

1. A related information retrieval unit that retrieves a group of related information, which is a set of related information related to the user's input information, A ranking unit that ranks the information necessary to respond to the user among the aforementioned related information group, An output information generation unit generates output information which is the response based on the ranked group of related information, A similarity calculation unit calculates, for each related piece of related information included in the related information group, the similarity between the input information and the related piece of related information as input similarity; the similarity between the related piece of related information and one or more related pieces of related information other than the related piece of related information group as inter-information similarity; and the similarity between the related piece of related information and one or more related pieces of related information whose modal type is different from the modal type of the related piece of related information group as inter-modal similarity. Equipped with, The ranking unit ranks the group of related information based on the input similarity, inter-information similarity, and inter-modal similarity of each related piece of information within the group of related information. Information processing device.

2. The output information generation unit takes the input information and the ranked related information group as input and generates the output information using a generation model. The information processing apparatus according to claim 1.

3. The related information retrieval unit retrieves the related information group from a storage unit that stores multiple pieces of information of multiple modal types, including information of a modal type different from the modal type of the input information. The information processing apparatus according to claim 1.

4. Obtain a group of related information, which consists of multiple pieces of related information related to the input information. For each related piece of related information included in the group of related information, the similarity between the input information and the related piece of related information is calculated as the input similarity; the similarity between the related piece of related information and one or more related pieces of related information other than the related piece of related information is calculated as the inter-information similarity; and the similarity between the related piece of related information and one or more related pieces of related information whose modal type is different from the modal type of the related piece of related information is calculated as the inter-modal similarity. Based on the input similarity, inter-information similarity, and inter-modal similarity of each related piece of information within the group of related information, the group of related information is ranked. Based on the ranked set of related information, output information is generated for the input information. Information processing methods.

5. Related information retrieval process to obtain a group of related information, which is a set of multiple related pieces of information related to the input information, A similarity calculation process that calculates the similarity between the input information and the related information for each related information included in the related information group as input similarity, calculates the similarity between the related information and one or more related information items other than the related information in the related information group as inter-information similarity, and calculates the similarity between the related information and one or more related information items other than the related information in the related information group whose modal type is different from the modal type of the related information as inter-modal similarity, A ranking process is performed to rank the group of related information based on the input similarity, inter-information similarity, and inter-modal similarity of each related piece of information within the group of related information. An output information generation process that generates output information for the input information based on the ranked related information group, An information processing program that causes a computer to execute something.