Information processing apparatus, information processing method, and information processing program

Abstract

The association information search unit (112) acquires a plurality of association information, i.e., an association information group, associated with input information. The similarity calculation unit (130) calculates, for each association information included in the association information group, a similarity between the input information and the association information as an input similarity, a similarity between another association information group and the association information as an inter-information similarity, and a similarity between another modality information group and the association information as an inter-modality similarity. The ranking unit (113) ranks the association information group based on the input similarity and the inter-modality similarity of each of the association information in the association information group. The output information generation unit (122) generates output information for the input information based on the ranked association information group.

Description

Technical Field

[0001] This disclosure relates to question and response technology. Background Technology

[0002] There are known systems that use generative models to respond to questions from users.

[0003] When using generative models to respond to user questions, it is difficult to generate appropriate responses if the knowledge learned by the generative model beforehand does not contain the information requested by the user.

[0004] The following system is known to correspond to inputs that query knowledge other than that learned by the generative model.

[0005] The system retrieves relevant information about the input by searching external knowledge, and then inputs this relevant information along with the input into the generative model.

[0006] When using external knowledge sources that only have textual information, it is sometimes impossible to respond appropriately to user input.

[0007] However, in conversational question-and-response tasks, there are relatively few techniques that use multiple modalities as external knowledge.

[0008] Non-patent literature 1 proposes a technique for answering questions using multimodal external knowledge.

[0009] Non-patent document 1 discloses the following method: A database for storing multimodal external knowledge is prepared; text, image, and table data are vectorized separately for retrieval; and a response statement is generated based on the retrieval results.

[0010] In the method of Non-Patent Document 1, information similar to the user's input statement is ranked higher in the search results, thus making it easy to output similar information as search results.

[0011] However, in order to generate appropriate responses to user input, a wide variety of information needs to be included in the search results. The method in Non-Patent Document 1 cannot handle this diversity of search results.

[0012] Existing technical documents

[0013] Non-patent literature

[0014] Non-patent document 1: Smaranda Muresan, Preslav Nakov, Aline Villavicencio, "MMCoQA: Conversational Question Answering over Text, Tables, and Images" InProceedings of the 60th Annual Meeting of the Association for ComputationalLinguistics (Volume 1: Long Papers), 2022. (The Hong Kong PolytechnicUniversity, Shandong University) Summary of the Invention

[0015] The problem that the invention aims to solve

[0016] The purpose of this disclosure is to generate output information that takes into account the diversity of search results by utilizing external knowledge associated with user input.

[0017] Methods for solving problems

[0018] The information processing apparatus disclosed herein includes: an association information retrieval unit that acquires multiple association information related to user input information, namely an association information group; a sorting unit that sorts the information required to respond to the user in the association information group into a top order; and an output information generation unit that generates output information as the response based on the sorted association information group.

[0019] Invention Effects

[0020] According to this disclosure, by utilizing external knowledge associated with user input, it is possible to generate output information that takes into account the diversity of search results. Attached Figure Description

[0021] Figure 1 This is a structural diagram of the question-and-response system 200 in Implementation Method 1.

[0022] Figure 2 This is a structural diagram of the response device 100 in Embodiment 1.

[0023] Figure 3 This is a functional structure diagram of the response device 100 in Implementation Method 1.

[0024] Figure 4 This is a structural diagram of the retrieval model storage unit 191 in Embodiment 1.

[0025] Figure 5 This is a structural diagram of the external knowledge storage unit 192 in Embodiment 1.

[0026] Figure 6 This is a structural diagram of the dialogue data storage unit 193 in Embodiment 1.

[0027] Figure 7 This is a structural diagram of the retrieval result storage unit 194 in Embodiment 1.

[0028] Figure 8 This is a structural diagram of the generation model storage unit 195 in Embodiment 1.

[0029] Figure 9 This is a structural diagram of the terminal device 210 in Embodiment 1.

[0030] Figure 10 This is a flowchart of the question-and-response method in Implementation Method 1.

[0031] Figure 11 This is a flowchart of step S110 in implementation method 1.

[0032] Figure 12 This is a flowchart of step S120 in implementation method 1.

[0033] Figure 13 This is a flowchart of step S130 in implementation method 1.

[0034] Figure 14 This is a hardware structure diagram of the response device 100 according to Embodiment 1.

[0035] Figure 15 This is a hardware structure diagram of the terminal device 210 in Implementation Method 1. Detailed Implementation

[0036] In the embodiments and accompanying drawings, the same or corresponding elements are labeled with the same reference numerals. Descriptions of elements labeled with the same reference numerals as those already described are appropriately omitted or simplified. Arrows in the figures primarily indicate data flow or processing flow.

[0037] Implementation method 1.

[0038] based on Figures 1 to 15 The question and response system 200 is described.

[0039] Structural description

[0040] based on Figure 1 The structure of the question and response system 200 will be explained.

[0041] The question and answer system 200 includes a terminal device 210 and an answering device 100.

[0042] Terminal device 210 and response device 100 communicate with each other via network 201. An example of network 201 is the Internet.

[0043] User input (question) is sent from terminal device 210 to response device 100, and output (response) to the input is sent from response device 100 to terminal device 210.

[0044] The question and response system 200 is an information processing system.

[0045] The response device 100 is an information processing device.

[0046] based on Figure 2 The structure of the response device 100 will be described.

[0047] The response device 100 is a computer equipped with hardware such as a processor 101, a memory 102, an auxiliary storage device 103, a communication device 104, an input / output interface 105, and a media interface 106. These hardware components are interconnected via signal lines.

[0048] Examples of computers include stationary computers, mobile computers, microcomputers for embedded devices, and SoCs (System-on-a-Chip).

[0049] Examples of stationary computers are personal computers and server computers.

[0050] Examples of portable computers are smartphones and tablets.

[0051] SoC is short for System on Chip.

[0052] Processor 101 is an IC that performs computational processing and controls other hardware. For example, processor 101 is a CPU or DSP.

[0053] Processor 101 can be a single processor or a multi-processor system.

[0054] IC is short for Integrated Circuit.

[0055] CPU is short for Central Processing Unit.

[0056] DSP is short for Digital Signal Processor.

[0057] Memory 102 is a volatile or non-volatile storage device. Memory 102 is also referred to as main storage device or main memory. For example, memory 102 is RAM. Data stored in memory 102 is stored in auxiliary storage device 103 as needed.

[0058] RAM is short for Random Access Memory.

[0059] Auxiliary storage device 103 is a non-volatile storage device. For example, auxiliary storage device 103 is ROM, HDD, flash memory, SSD, or a combination thereof. Data stored in auxiliary storage device 103 is loaded into memory 102 as needed.

[0060] ROM is short for Read Only Memory.

[0061] HDD is short for Hard Disk Drive.

[0062] SSD is short for Solid State Drive.

[0063] Communication device 104 is a receiver and a transmitter. For example, communication device 104 is a communication chip or NIC. Communication of response device 100 is performed using communication device 104.

[0064] NIC is short for Network Interface Card.

[0065] Input / output interface 105 is a port for connecting input devices and output devices. For example, input / output interface 105 is a USB terminal, the input devices are a keyboard and mouse, and the output device is a monitor. The input and output of response device 100 are performed using input / output interface 105.

[0066] USB is short for Universal Serial Bus.

[0067] Media interface 106 is a port for connecting to external storage media. Media interface 106 reads programs and data stored in the external storage media and provides the read programs and data to processor 101 via memory 102. Examples of external storage media are CDs, DVDs, or flash memory.

[0068] CD is short for Compact Disc.

[0069] DVD is short for Digital Versatile Disc.

[0070] The response device 100 includes elements such as a retrieval unit 110 and a reasoning unit 120. These elements are implemented through software.

[0071] The auxiliary storage device 103 stores a response program for enabling the computer to function as the retrieval unit 110 and the reasoning unit 120. The response program is loaded into the memory 102 and executed by the processor 101.

[0072] The OS is also stored in the auxiliary storage device 103. At least a portion of the OS is loaded into the memory 102 and executed by the processor 101.

[0073] Processor 101 executes inference programs while executing the OS.

[0074] OS is short for Operating System.

[0075] The response procedure's data (input data, output data, etc.) is stored in storage unit 190.

[0076] The auxiliary storage device 103 functions as the storage unit 190. However, storage devices such as the memory 102, registers within the processor 101, cache memory within the processor 101, and external storage media may also replace the memory 102 or function as the storage unit 190 together with the memory 102.

[0077] The response program can be recorded (saved) in a computer-readable manner on non-volatile recording media such as optical discs or flash memory.

[0078] based on Figure 3 The functional structure of the response device 100 will be explained.

[0079] The retrieval unit 110 includes an acquisition unit 111, a related information retrieval unit 112, a similarity calculation unit 130, and a ranking unit 113.

[0080] The similarity calculation unit 130 includes an input similarity calculation unit 131, an information similarity calculation unit 132, and an intermodal similarity calculation unit 133.

[0081] The reasoning unit 120 includes an acquisition unit 121, an output information generation unit 122, and an output unit 123.

[0082] The storage unit 190 includes a retrieval model storage unit 191, an external knowledge storage unit 192, a dialogue data storage unit 193, a retrieval result storage unit 194, and a generation model storage unit 195.

[0083] based on Figure 4The structure of the retrieval model storage unit 191 will be explained.

[0084] The retrieval model storage unit 191 stores multiple retrieval models with different modal types. Each of the multiple retrieval models corresponds to a modal type.

[0085] Examples of modal types include text, images, speech, tables, and 3D information. Examples of 3D information include point clouds, meshes, and distance information.

[0086] Retrieval models vectorize information for retrieval purposes. When the modality is text, the retrieval model is one that vectorizes the text information. When the modality is audio, retrieval models include models that convert audio information into text, models that vectorize the text information, and models that vectorize the audio information in the same latent space as the text. When the modality is image, retrieval models include models that textify image information, models that vectorize the text information, and models that vectorize the image information in the same latent space as the text. Models that textify image information are called descriptive generative models.

[0087] For example, the retrieval model storage unit 191 stores CLIP and GPT-4 models as image retrieval models. CLIP is short for Contrastive Language-Image Pre-Training. GPT is short for Generative Pretrained Transformer.

[0088] For example, the retrieval model storage unit 191 stores the Whisper model and the GPT-4 model as retrieval models whose modality type is speech. The Whisper model converts speech information into text information. The GPT-4 model vectorizes the text information.

[0089] The retrieval model is obtained, for example, by the acquisition unit 111 from the retrieval model storage unit 191.

[0090] The model is generated through various machine learning methods. It is also called a learned model or an AI model. AI stands for Artificial Intelligence.

[0091] based on Figure 5 The structure of the external knowledge storage unit 192 will be explained.

[0092] The external knowledge storage unit 192 stores multiple external knowledge items of multiple modal types. Each of the multiple external knowledge items corresponds to a modal type of external knowledge.

[0093] External knowledge refers to information beyond what the generative model learns, as discussed later.

[0094] External knowledge that is not textual can be converted into textual information and stored, or it can be stored directly as modal data. For example, if the image of "Athlete X in the August 1st match" is external knowledge, the textual information such as "Athlete X in the August 1st match" can be stored as external knowledge, or the image can be stored as external knowledge.

[0095] External knowledge is obtained, for example, from the external knowledge storage unit 192 by the acquisition unit 111.

[0096] based on Figure 6 The structure of the dialogue data storage unit 193 will be explained.

[0097] The dialogue data storage unit 193 stores (past) dialogue data. The dialogue data contains multiple speech information entries. Each speech information entry corresponds to a speaker identifier and modality type.

[0098] Speaker identifiers identify the speaker. For example, a speaker identifier distinguishes between a user and a model. However, a speaker identifier can also identify individual users and individual models. Users and models do not need to alternate as speakers. For example, users can speak continuously.

[0099] The speech information is information sent from the user or the model.

[0100] Speech information that is not in a modal format (text) can be converted to text and stored, or it can be stored directly as modal data. For example, images or speech can also be stored as speech information.

[0101] Dialogue data is stored, for example, by output unit 123. Output unit 123 stores the user's input information and the output information of output information generation unit 122 as speech information in dialogue data storage unit 193.

[0102] The dialogue data storage unit 193 obtains data from the dialogue data storage unit 193, for example, by the acquisition unit 111.

[0103] based on Figure 7 The structure of the retrieval result storage unit 194 will be explained.

[0104] The search results storage unit 194 stores multiple search results for multiple modalities. Each search result corresponds to a modality, a preceding score, and a following score.

[0105] The previous score is the search score before reordering. The search score will be described later.

[0106] The re-ranked score is the search score after reordering.

[0107] The search results are related information described later.

[0108] The search results are stored, for example, by the sorting unit 113. For example, the sorting unit 113 stores the association information described later as search results, stores the input similarity described later as a pre-score, and stores the search score described later as a post-score.

[0109] The search results are obtained, for example, by the acquisition unit 121 from the search results storage unit 194.

[0110] based on Figure 8 The structure of the generation model storage unit 195 will be explained.

[0111] The generative model storage unit 195 stores generative models. The generative model storage unit 195 can also store multiple generative models with different modal types of the information to be generated. Each of the multiple generative models corresponds to a modal type of the information to be generated.

[0112] Generative models generate and output information of a specific modality (text or image, etc.) in response to input. Generative models are also known as generative AI or generative AI models.

[0113] The generated model is obtained, for example, from the generated model storage unit 195 by the acquisition unit 121.

[0114] based on Figure 9 The structure of the terminal device 210 will be described.

[0115] Terminal device 210 is a computer equipped with hardware such as processor 211, memory 212, auxiliary storage device 213, communication device 214, and input / output interface 215. These hardware components are interconnected via signal lines.

[0116] Processor 211 is an IC that performs computational processing and controls other hardware. For example, processor 211 is a CPU or DSP.

[0117] Memory 212 is a volatile or non-volatile storage device. Memory 212 is also referred to as main storage device or main memory. For example, memory 212 is RAM. Data stored in memory 212 is stored in auxiliary storage device 213 as needed.

[0118] Auxiliary storage device 213 is a non-volatile storage device. Auxiliary storage device 213 is also referred to as a memory. For example, auxiliary storage device 213 is ROM, HDD, flash memory, SSD, or a combination thereof. Data stored in auxiliary storage device 213 is loaded into memory 212 as needed.

[0119] Communication device 214 is both a receiver and a transmitter. For example, communication device 214 is a communication chip or NIC. Communication of terminal device 210 is performed using communication device 214.

[0120] Input / output interface 215 is a port for connecting input devices and output devices. For example, input / output interface 215 is a USB terminal, the input devices are a keyboard and mouse, and the output device is a monitor. The input and output of terminal device 210 are performed using input / output interface 215.

[0121] Media interface 216 is a port for connecting to external storage media. Media interface 216 reads programs and data stored in external storage media and provides the read programs and data to processor 211 through memory 212.

[0122] The terminal device 210 includes elements such as a receiving unit 221, a dialogue unit 222, and an output unit 223. These elements are implemented through software.

[0123] The auxiliary storage device 213 stores a terminal program for enabling the computer to function as a reception unit 221, a dialogue unit 222, and an output unit 223. The terminal program is loaded into the memory 212 and executed by the processor 211.

[0124] The OS is also stored in the auxiliary storage device 213. At least a portion of the OS is loaded into the memory 212 and executed by the processor 211.

[0125] The processor 211 executes the OS and the terminal program simultaneously.

[0126] The terminal program's data is stored in storage unit 229.

[0127] The memory 212 functions as a storage unit 229. However, auxiliary storage devices such as the auxiliary storage device 213, registers within the processor 211, cache memory within the processor 211, and external storage media may also replace the memory 212 or function as a storage unit 229 together with the memory 212.

[0128] Terminal programs can be recorded (saved) in a computer-readable manner on non-volatile recording media such as optical discs or flash memory.

[0129] Description of the action

[0130] The steps of the question-and-response system 200 are equivalent to a question-and-response method. The steps of the response device 100 are equivalent to a response method. The steps of the terminal device 210 are equivalent to a question-and-response method. A question-and-response method includes a question-and-response method and a response method. Each of the question-and-response method, the question-and-response method, and the response method is an information processing method.

[0131] based on Figure 10 This section explains the question-and-answer methods.

[0132] In step S101, the user inputs speech information indicating a question into the terminal device 210. The input speech information is referred to as input information.

[0133] The receiving section 221 of the terminal device 210 receives input information.

[0134] In step S102, the dialogue unit 222 of the terminal device 210 sends input data to the response device 100. The input data is communication data containing input information.

[0135] In step S103, the acquisition unit 111 of the response device 100 receives input data sent from the terminal device 210.

[0136] In step S110, the output information generation unit 122 of the response device 100 generates output information in response to the input information. The output information is speech information representing the response to the question shown in the input information.

[0137] The details of step S110 will be described later.

[0138] In step S104, the output unit 123 of the response device 100 sends output data to the terminal device 210. The output data is communication data containing output information.

[0139] In step S105, the dialogue unit 222 of the terminal device 210 receives output data sent from the response device 100.

[0140] In step S106, the output unit 223 of the terminal device 210 outputs output information.

[0141] For example, the output unit 223 displays the output information on the display of the terminal device 210.

[0142] based on Figure 11 Explain the steps in step S110.

[0143] In step S111, the association information retrieval unit 112 obtains the association information group for the input information.

[0144] A group of associated information consists of multiple associated information of multiple modal types.

[0145] Associated information is information that is associated with the input information.

[0146] The input information is retrieved by using the external knowledge storage unit 192 to obtain a set of related information for the input information. Multiple external knowledge items found during the retrieval become a set of related information for the input information.

[0147] The associated information group for the input information is obtained through the following steps.

[0148] First, the acquisition unit 111 acquires a retrieval model for the modality type of the input information from the retrieval model storage unit 191, and the related information retrieval unit 112 uses the acquired retrieval model to vectorize the input information. The vectorized input information is called vectorized input information.

[0149] Furthermore, according to each piece of external knowledge stored in the external knowledge storage unit 192, the acquisition unit 111 acquires a retrieval model of the modality type for the external knowledge from the retrieval model storage unit 191, and the related information retrieval unit 112 uses the acquired retrieval model to vectorize the external knowledge. The vectorized external knowledge is referred to as vectorized external knowledge.

[0150] Then, the association information retrieval unit 112 uses the vectorized input information to perform a retrieval of a set of vectorized external knowledge. Multiple sets of vectorized external knowledge detected during the retrieval, or multiple sets of external knowledge corresponding to the multiple sets of vectorized external knowledge detected during the retrieval, become an association information group for the input information.

[0151] In step S120, the sorting unit 113 calculates the retrieval score for each piece of related information. The retrieval score is also called the similarity score.

[0152] based on Figure 12 Explain the steps in step S120.

[0153] In step S121, the similarity calculation unit 130 selects one unselected piece of related information from the group of related information for the input information.

[0154] In step S122, the input similarity calculation unit 131 calculates the input similarity of the selected associated information.

[0155] Input similarity is the similarity between input information and related information.

[0156] For example, the input similarity calculation unit 131 calculates the cosine similarity between the input information and the associated information as the input similarity.

[0157] For example, related information D i The input similarity Sim1(D) relative to the input information Q i Q can be represented by equation (1). “Vec(x)” represents the vectorized information x. “Vec” represents the vectorized model (retrieval model).

[0158] [Mathematical Expression 1]

[0159] In step S123, the information similarity calculation unit 132 calculates the information similarity between the selected related information.

[0160] Information similarity is the degree of similarity between related information and other related information groups.

[0161] Other related information groups are one or more related information items other than the object related information in a related information group. Specifically, other related information groups are one or more related information items for which a search score has been calculated. Each related information item contained in an other related information group is referred to as other related information.

[0162] For example, the similarity between information can be calculated as follows.

[0163] First, the information similarity calculation unit 132 calculates the cosine similarity between the selected associated information and other associated information according to each other associated information.

[0164] Then, the information similarity calculation unit 132 calculates the information similarity based on the calculated cosine similarity. For example, the information similarity calculation unit 132 determines the largest cosine similarity among the calculated cosine similarities as the information similarity.

[0165] For example, the selected related information D i Information similarity Sim2 (D i ,S) can be represented by equation (2). “S” represents other related information groups. “D” S " indicates other related information.

[0166] [Mathematical Expression 2]

[0167] In addition, the similarity between related information can be calculated sequentially starting from related information with high input similarity, or it can be calculated in different orders and then sorted according to the input similarity from high to low.

[0168] In step S124, the intermodal similarity calculation unit 133 calculates the intermodal similarity of the selected associated information.

[0169] Intermodal similarity is the similarity between related information and other modal information groups.

[0170] Other modal information groups are one or more pieces of association information in an association information group whose modal type differs from that of the object association information. Specifically, other modal information groups are one or more pieces of association information whose modal type differs from that of the object association information among those whose retrieval scores have been calculated. Each piece of association information contained in an other modal information group is referred to as other modal information.

[0171] For example, calculate intermodal similarity as follows.

[0172] First, the inter-modal similarity calculation unit 133 compares the modal type of the selected associated information with the modal types of each of the other associated information. The modal type of each associated information is obtained from the external knowledge storage unit 192.

[0173] Next, based on the comparison results, the intermodal similarity calculation unit 133 extracts other modal information groups from the associated information group.

[0174] Next, the intermodal similarity calculation unit 133 calculates the cosine similarity between the selected association information and other modal information according to each other modal information.

[0175] Then, the inter-modal similarity calculation unit 133 calculates the inter-modal similarity based on the calculated cosine similarity. For example, the inter-modal similarity calculation unit 133 determines the largest cosine similarity among the calculated cosine similarities as the inter-modal similarity.

[0176] For example, the selected related information D i intermodal similarity Sim3 (D i ,S N ) can be expressed by equation (3). “S N "Indicates other modal information groups." n " indicates other modal information.

[0177] [Mathematical Expression 3]

[0178] Steps S122 to S124 can also be executed in a different order.

[0179] In step S125, the sorting unit 113 calculates the retrieval score of the selected related information based on the input similarity, information similarity, and modal similarity of the selected related information.

[0180] For example, the ranking unit 113 uses input similarity, inter-information similarity, and inter-modal similarity to calculate a formula, thereby calculating a retrieval score.

[0181] For example, the selected related information D i The search score Sim(D) i ) can be represented by equation (4). “k” and “l” represent hyperparameters.

[0182] [Mathematical Expression 4]

[0183] In step S126, the similarity calculation unit 130 determines whether there is any unselected related information.

[0184] If there is unselected associated information, proceed to step S121.

[0185] If no unselected associated information exists, the process in step S120 ends.

[0186] return Figure 11 The explanation continues from step S112.

[0187] In step S112, the sorting unit 113 sorts the associated information group based on the retrieval scores of each associated information in the associated information group.

[0188] For example, the sorting unit 113 sorts the multiple pieces of related information that constitute the related information group in descending order of search scores.

[0189] In step S130, the inference unit 120 generates output information for the input information based on the sorted association information group.

[0190] It can take input information and sorted related information groups as input and use a generative model to generate output information.

[0191] based on Figure 13 Explain the steps in step S130.

[0192] In step S131, the output information generation unit 122 vectorizes the input information.

[0193] In addition, the output information generation unit 122 vectorizes each piece of related information in the related information group.

[0194] The input information and related information are vectorized as follows.

[0195] The acquisition unit 121 acquires a retrieval model for the modality type of the input information from the retrieval model storage unit 191, and the output information generation unit 122 uses the acquired retrieval model to vectorize the input information.

[0196] According to each piece of related information, the acquisition unit 121 acquires the retrieval model of the modality type for the related information from the retrieval model storage unit 191, and the output information generation unit 122 uses the acquired retrieval model to vectorize the related information.

[0197] The vectorized input information is called vectorized input information.

[0198] A group of related information composed of vectorized related information is called a vectorized related information group.

[0199] In step S132, the output information generation unit 122 selects the generation model to be used.

[0200] Specifically, a generative model is selected to generate information of the same modality type as the modality type required as the output information.

[0201] For example, when requesting image output, a generative model for generating the image is selected. Examples of generative models for generating images include GPT-4, DaLL·E3, and other image generation models.

[0202] For example, the modality type required to determine the modality type of the output information based on the input information.

[0203] In step S133, the output information generation unit 122 takes the vectorized input information and the sorted group of vectorized association information as input to the selected generation model and performs inference using the selected generation model. Thus, output information of the required modality type is generated.

[0204] Effects of Implementation Method 1

[0205] Implementation method 1 involves question and response technology.

[0206] Implementation 1 specifically relates to a multimodal question-and-response device that, for questions that require multimodal information as an external knowledge source, supplementarily combines multimodal information to appropriately respond to user input.

[0207] Implementation 1 proposes an algorithm that reorders information associated with user input while maintaining the diversity of information.

[0208] One of the objectives of Implementation Method 1 is to efficiently obtain the information required to respond to user input in order to generate a response.

[0209] The response device 100 includes an external knowledge storage unit 192, an associated information retrieval unit 112, an input similarity calculation unit 131, an information similarity calculation unit 132, an intermodal similarity calculation unit 133, a sorting unit 113, and an output information generation unit 122.

[0210] External knowledge storage unit 192 stores multiple modal knowledge.

[0211] The related information retrieval unit 112 retrieves information related to the input.

[0212] The similarity calculation unit 131 calculates the similarity between each piece of related information and the dialogue data.

[0213] The information similarity calculation unit 132 calculates the similarity between each related piece of information and other related information and obtains the maximum value.

[0214] The intermodal similarity calculation unit 133 calculates the similarity between each piece of related information and knowledge that differs from the modality and obtains the maximum value.

[0215] The sorting unit 113 changes the sorting of the search results obtained by the related information retrieval unit 112 (re-sorts).

[0216] The output information generation unit 122 inputs dialogue data and related information into the generation model to generate a response.

[0217] When determining the reordering score, the response device 100 calculates the similarity between each piece of information contained in the subset of information for which the reordering scoring has been completed and the object information, and obtains the maximum value of each similarity. By subtracting the maximum similarity from the reordering score, it is possible to avoid similar information being ranked higher multiple times.

[0218] Alternatively, you can replace subtracting the maximum similarity value; for example, you can add a specified value to the maximum value or multiply the maximum value by a specified value.

[0219] When determining the re-ranking score, the response device 100 calculates the similarity between each piece of information in a subset of the information for which re-ranking has been applied, and the object information, and obtains the maximum similarity between each of these modalities and the object information. By adding this maximum similarity to the re-ranking score, it is possible to retrieve information that can improve the re-ranking score when combined with other modal information. In other words, it is possible to perform a retrieval that takes into account diversity and modal information in the retrieval results.

[0220] Therefore, by utilizing external knowledge (of different modalities) associated with the user's input, the response device 100 is able to generate output information that takes into account the diversity of search results. In other words, the response device 100 can obtain search results that maintain diversity from the perspective of the meaning and modality of information based on external knowledge and the user's input, and use these search results to generate output information.

[0221] The method in non-patent literature 1 cannot take into account the information complementarity between modes.

[0222] For example, information that can be obtained from an input statement like "What color were the gloves of player X when they played against team Y?" is mostly textual information similar to the input statement, such as "player X played against team Y on August 1st". Therefore, an image depicting a scene where "player X played against team Y on August 1st", even though it contains information about the color of the gloves, may not become the higher-level information to be retrieved as input to the generative model.

[0223] Therefore, it is necessary to obtain information such as "Player X played against Team Y on August 1st" from the text information obtained based on the user's input statement, and to reorder the search priority of images depicting the scene of "Player X played on August 1st" to the top.

[0224] The method of Implementation 1 is capable of performing such a reordering.

[0225] Supplement to Implementation Method 1

[0226] The response device 100 stores dialogue data between the user and the response device 100. In response to new input from the user, it retrieves information associated with the input data from external knowledge based on past dialogue data and the user's new input information.

[0227] In addition, the response device 100 reorders the search results by combining the similarity between the input data and the search results, the similarity between the search results and other search results, and the similarity between the search results and search results with different modalities.

[0228] Then, the response device 100 inputs the reordered results and dialogue data into the generation model to generate response information to the user's input data.

[0229] The response device 100 includes a dialogue data storage unit 193, an external knowledge storage unit 192, a related information retrieval unit 112, a sorting unit 113, and an output information generation unit 122.

[0230] The dialogue data storage unit 193 accepts input from users.

[0231] External knowledge storage unit 192 stores knowledge obtained from multiple modalities.

[0232] The related information retrieval unit 112 retrieves the information needed to respond to input from the user.

[0233] The sorting unit 113 sorts the required information into the top order.

[0234] The output information generation unit 122 generates a response to the user.

[0235] The ranking unit 113 calculates a score using the similarity between the dialogue information between the user and the system and external knowledge, the maximum similarity between the existing search results and the information to be re-ranked, and the inter-modal similarity when the re-ranked information and the existing search results have different modalities. Then, the ranking unit 113 uses the calculated score to re-rank the search results.

[0236] The question-and-response system 200 may also have multiple terminal devices 210. The multiple terminal devices 210 communicate with the response device 100 via the network 201.

[0237] External knowledge retrieval methods can utilize automated retrieval methods or token-based retrieval methods. Automated retrieval methods are based on machine learning methods or pre-learned models. Token-based retrieval methods include TF-IDF (Term Frequency-Inverse Document Frequency). When using models such as machine learning, the model is stored in the retrieval model storage unit 191.

[0238] Regarding methods for retrieving external knowledge, in order to perform retrieval across different modalities, all modalities are replaced with single modal information, and a vectorization model using that modality type is used. Alternatively, external knowledge retrieval methods can use models capable of vectorizing different modalities in the same latent space. The model used for retrieval is stored in the retrieval model storage unit 191.

[0239] The input similarity calculation unit 131 may also include past dialogue information in the input data Q in addition to the input information.

[0240] The retrieval score is calculated based on input similarity, information similarity, and modality similarity.

[0241] The retrieval model storage unit 191, the external knowledge storage unit 192, the dialogue data storage unit 193, the retrieval result storage unit 194, and the generation model storage unit 195 can also be set externally respectively.

[0242] These "storage units" can also be replaced with "databases".

[0243] Retrieval models can also be models that convert input information and various external knowledge into information of modal types other than text and then vectorize them.

[0244] The information conveyed in a speech can also be replaced with information other than the speech itself (questioning, answering).

[0245] Methods other than cosine similarity can also be used to calculate input similarity, inter-information similarity, and inter-modal similarity. For example, well-known methods such as Pearson correlation coefficient and biased pattern similarity can be used.

[0246] The steps of operation of the query-response system 200 are equivalent to the steps of processing based on the query-response procedure. The steps of operation of the response device 100 are equivalent to the steps of processing the response procedure. The steps of operation of the terminal device 210 are equivalent to the steps of processing the terminal procedure. The query-response procedure includes a terminal procedure and a response procedure. The query-response procedure, the terminal procedure, and the response procedure are each information processing procedures. The terminal procedure is also called the query procedure.

[0247] based on Figure 14 The hardware structure of the response device 100 will be described.

[0248] The response device 100 includes a processing circuit 109.

[0249] The processing circuit 109 is the hardware that implements the retrieval unit 110 and the inference unit 120.

[0250] The processing circuit 109 can be dedicated hardware or a processor 101 that executes the program stored in the memory 102.

[0251] When the processing circuit 109 is dedicated hardware, the processing circuit 109 may be, for example, a single circuit, a composite circuit, a programmable processor, a parallel programmable processor, an ASIC, an FPGA, or a combination thereof.

[0252] ASIC is short for Application Specific Integrated Circuit.

[0253] FPGA is short for Field Programmable Gate Array.

[0254] The response device 100 may include multiple processing circuits that replace the processing circuit 109.

[0255] In the processing circuit 109, some functions may be implemented by dedicated hardware, while the remaining functions may be implemented by software or firmware.

[0256] Thus, the function of the response device 100 can be implemented through hardware, software, firmware, or a combination thereof.

[0257] based on Figure 15 The hardware structure of the terminal device 210 will be described.

[0258] The terminal device 210 includes a processing circuit 219.

[0259] The processing circuit 219 is the hardware that implements the receiving unit 221, the dialogue unit 222, and the output unit 223.

[0260] The processing circuit 219 can be dedicated hardware or a processor 211 that executes the program stored in the memory 212.

[0261] When the processing circuit 219 is dedicated hardware, the processing circuit 219 may be, for example, a single circuit, a composite circuit, a programmable processor, a parallel programmable processor, an ASIC, an FPGA, or a combination thereof.

[0262] The terminal device 210 may also have multiple processing circuits that replace the processing circuit 219.

[0263] In the processing circuit 219, some functions may be implemented by dedicated hardware, while the remaining functions may be implemented by software or firmware.

[0264] In this way, the functions of the terminal device 210 can be implemented through hardware, software, firmware, or a combination thereof.

[0265] Embodiment 1 is an example of a preferred embodiment and is not intended to limit the scope of this disclosure. Embodiment 1 may be implemented in part or in combination with other embodiments. The steps described using flowcharts or the like may also be appropriately modified.

[0266] The term "part" in the elements of the response device 100 and the terminal device 210 can also be replaced with "process", "procedure", "circuit" or "line".

[0267] Label Explanation

[0268] 100 Response device, 101 Processor, 102 Memory, 103 Auxiliary storage device, 104 Communication device, 105 Input / output interface, 106 Media interface, 109 Processing circuit, 110 Retrieval unit, 111 Acquisition unit, 112 Related information retrieval unit, 113 Sorting unit, 120 Reasoning unit, 121 Acquisition unit, 122 Output information generation unit, 123 Output unit, 130 Similarity calculation unit, 131 Input similarity calculation unit, 132 Inter-information similarity calculation unit, 133 Inter-modal similarity calculation unit. Similarity calculation unit, 190 storage unit, 191 retrieval model storage unit, 192 external knowledge storage unit, 193 dialogue data storage unit, 194 retrieval result storage unit, 195 generation model storage unit, 200 question and answer system, 201 network, 210 terminal device, 211 processor, 212 memory, 213 auxiliary storage device, 214 communication device, 215 input / output interface, 216 media interface, 219 processing circuit, 221 receiving unit, 222 dialogue unit, 223 output unit.

Claims

1. An information processing device comprising: The related information retrieval department retrieves multiple related information sets associated with the user's input information. The sorting unit sorts the information required for the user's response in the associated information group into the highest priority; and The output information generation unit generates output information as the response based on the sorted associated information group.

2. The information processing apparatus according to claim 1, wherein, The information processing device includes a similarity calculation unit. This unit calculates the similarity between the input information and each associated information in the associated information group as the input similarity; calculates the similarity between the input information and one or more other associated information groups (i.e., other associated information groups) in the associated information group as the inter-information similarity; and calculates the similarity between the associated information and one or more other associated information groups (i.e., other modality information groups) in the associated information group whose modality type is different from that of the associated information as the inter-modality similarity. The sorting unit sorts the associated information group based on the input similarity, the inter-information similarity, and the inter-modal similarity of each associated information in the associated information group.

3. The information processing apparatus according to claim 1 or 2, wherein, The output information generation unit takes the input information and the sorted associated information group as input and uses a generation model to generate the output information.

4. The information processing apparatus according to any one of claims 1 to 3, wherein, The associated information retrieval unit obtains the associated information group from a storage unit that stores multiple pieces of information of multiple modal types, in a manner that includes information of a modal type different from the modal type of the input information.

5. An information processing method, wherein, Retrieving multiple pieces of related information associated with the input information constitutes a group of related information. For each piece of related information contained in the related information group, the similarity between the input information and the related information is calculated as the input similarity. The similarity between the input information and one or more other related information groups (i.e., other related information groups) in the related information group (i.e., other modality information groups) and the related information is calculated as the inter-information similarity. The associated information group is sorted based on the input similarity, inter-information similarity, and inter-modal similarity of each associated information in the associated information group. Based on the sorted group of related information, output information is generated for the input information.

6. An information processing program for causing a computer to perform the following processes: Related information retrieval processing obtains multiple related information groups associated with the input information; The similarity calculation process involves calculating the similarity between the input information and the associated information for each associated information contained in the associated information group as the input similarity; calculating the similarity between the associated information and one or more other associated information groups (i.e., other associated information groups) in the associated information group (i.e., other associated information groups) as the inter-information similarity; and calculating the similarity between the associated information and one or more other associated information groups (i.e., other associated information groups) whose modality type is different from that of the associated information as the inter-modality similarity. The sorting process involves sorting the associated information group based on the input similarity, the inter-information similarity, and the inter-modal similarity of each associated information in the associated information group. as well as The output information generation process generates output information for the input information based on the sorted associated information group.

Images