Information processing device, information processing method, and program

Abstract

To provide an information processing device, an information processing method, and a program that can improve the accuracy of vector searches for multimodal documents. [Solution] An information processing device comprising: a generation unit that generates evaluation data including an evaluation query and correct answer information corresponding to the evaluation query based on the content of target data including text and images; and an optimization unit that optimizes search parameters for controlling the search process on the target data so as to improve the search accuracy using the evaluation data.

Description

[Technical Field]

[0001] This invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Conventionally, techniques for improving the search accuracy of databases in which knowledge is stored are known. For example, Patent Document 1 discloses a technique for extracting knowledge based on the similarity between vectorized knowledge stored in a vector database and vectorized creation information for creating medical-related information. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Patent application No. 2024-114086 [Overview of the project] [Problems that the invention aims to solve]

[0004] Generally speaking, in vector searches for multimodal documents containing both text and images, image vectors are suitable for visual queries and text vectors for text-based queries. However, applying fixed weights to both modalities does not adequately reflect the optimal information balance for each document. Conventional techniques have sometimes resulted in low accuracy in vector searches for multimodal documents because they do not consider this optimal information balance for each document.

[0005] This invention has been made in consideration of these circumstances, and one of its objectives is to provide an information processing device, an information processing method, and a program that can improve the accuracy of vector searches for multimodal documents. [Means for solving the problem]

[0006] One aspect of the present invention is an information processing device comprising: a generation unit that generates evaluation data including an evaluation query and correct answer information corresponding to the evaluation query based on the content of target data including text and images; and an optimization unit that optimizes search parameters for controlling the search process on the target data so as to improve the search accuracy using the evaluation data. [Effects of the Invention]

[0007] According to one aspect of the present invention, it is possible to provide an information processing device, an information processing method, and a program that can improve the accuracy of vector searches for multimodal documents. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of the overall configuration and operation overview of an information processing system including the information processing device 100 according to this embodiment. [Figure 2] This is a block diagram showing an example of the functional configuration of the information processing device 100. [Figure 3] This figure shows an example of a prompt 164 that the evaluation data generation unit 120 inputs to the LLM server 200. [Figure 4] This figure shows an example of the structure of the generated evaluation data 166. [Figure 5] This diagram shows an overview of the search simulation execution. [Figure 6] This figure shows an example of the structure of the determined optimization search parameters 168. [Figure 7] This flowchart shows an example of the search parameter optimization process. [Figure 8] A flowchart illustrating an example of the search process flow using optimized search parameters. [Modes for carrying out the invention]

[0009] Hereinafter, embodiments of an information processing apparatus, an information processing method, and a program according to the present invention will be described with reference to the drawings.

[0010] [Problems in the Background Art] In recent years, the importance of technologies for retrieving information from multimodal documents (e.g., technical reports, presentation materials, operation manuals, etc.) in which text and images are mixed, particularly Retrieval-Augmented Generation (RAG), has been increasing. In multimodal retrieval, a hybrid retrieval that combines text-based retrieval (text vector retrieval) and image-based retrieval (image vector retrieval) based on text information is effective. However, there has been a problem that the optimal balance (weighting ratio) between text retrieval and image retrieval for maximizing retrieval accuracy varies greatly depending on the content and characteristics of the documents to be retrieved (whether text-based or image-based). Image vectors are suitable for visual queries, and text vectors are suitable for text-based queries, but applying fixed weights to both modalities was not the best solution because the optimal information balance varies from document to document.

[0011] In the prior art, since this weighting ratio is set as a fixed value or manually tuned by an expert using a large amount of labeled data, it is difficult to achieve optimal search accuracy or requires a great deal of effort. The information processing apparatus 100 according to the present embodiment solves such a conventional problem and enables the accuracy of vector search for multimodal documents to be automatically maximized without requiring a complicated manual tuning operation. The gist of the present embodiment is to automatically generate evaluation data for search accuracy evaluation by utilizing a large language model (LLM) or the like based on the content of the multimodal document itself to be searched, and to automatically search for and determine the optimal weighting ratio between text search and image search using the evaluation data, which is a "self-optimization" mechanism. As a result, it becomes possible to intelligently adjust the focus (bias) of the search according to the characteristics of each document.

[0012] [Overall Configuration and Operation Outline of the System] FIG. 1 is a diagram showing an example of the overall configuration and operation outline of an information processing system including the information processing apparatus 100 according to the present embodiment. This system is mainly composed of a user terminal device 10, an information processing apparatus 100, and an LLM server 200. These devices are communicably connected to each other via a network (not shown), such as the Internet, a LAN (Local Area Network), a VPN (Virtual Private Network), or a dedicated line.

[0013] The user terminal device 10 is a client device operated by the user, such as a personal computer, a smartphone, a tablet terminal, or the like. The user transmits a search query to the information processing apparatus 100 via the user terminal device 10 (S5: Query) and receives an answer provided from the information processing apparatus 100 (S7: Answer).

[0014] The LLM server 200 is a server device that hosts a Large-Scale Language Model (LLM) and performs inference processing in response to requests from external sources. In this embodiment, the LLM server 200 primarily plays two roles in response to requests from the information processing device 100. The first role is to analyze the content of the documents to be searched and automatically generate evaluation data in Phase 1, which will be described later (used in S2). The second role is to generate natural language answers to the user's queries in Phase 2, based on the information retrieved by the information processing device 100 (chunks, which are fragments of documents) (used in S6 and S7). The LLM server 200 may be provided as a cloud service using an API (Application Programming Interface), or it may be built in a local environment within the organization.

[0015] The information processing device 100 is a server device that performs search processing on multimodal documents and has the function of automatically optimizing the search accuracy. The operation of the information processing device 100 is broadly composed of two phases: "Phase 1: Optimization of search parameters" and "Phase 2: Search execution". In Figure 1, the information processing device 100, which is a single device, is shown to perform the processing of Phase 1 and Phase 2. However, the present invention is not limited to such a configuration, and the functions of the information processing device 100 may be distributed among multiple devices, and these multiple devices may perform the processing of Phase 1 and Phase 2 separately.

[0016] (Phase 1: Optimization of search parameters) Phase 1 is the process of automatically determining the search parameters in advance to achieve the highest search accuracy for the dataset to be searched. First, the information processing device 100 receives the target document 162 to be searched as input (S1: input). The target document 162 is multimodal data that includes text information and image information.

[0017] Next, the information processing device 100 generates evaluation data based on the content of the input target document 162 (S2: generation of evaluation data). The evaluation data is a dataset consisting of pairs of evaluation queries (question sentences) and correct answer information for those queries. One of the features of this embodiment is that this evaluation data is automatically generated by utilizing the language understanding capabilities of the LLM server 200. This makes it possible to comprehensively prepare high-quality evaluation data that reflects the characteristics of the target document 162 while significantly reducing the cost of manually creating evaluation data.

[0018] Next, the information processing device 100 uses the generated evaluation data to search for the optimal values ​​of the search parameters (S3: Search for search parameters). The search parameters here include weight information for adjusting the contribution of text similarity and image similarity in multimodal search. The information processing device 100 repeatedly performs search simulations on the evaluation data while varying the values ​​of the search parameters and calculates the search accuracy for each set value of the search parameters.

[0019] Then, the information processing device 100 determines the combination of search parameter values ​​that yielded the highest search accuracy as the optimized search parameter (S4: Determination of optimized search parameter). The determined optimized search parameter is stored as the optimized search parameter 168 in the storage unit 160 within the information processing device 100.

[0020] (Phase 2: Search execution) Phase 2 is the actual search processing process, in which the system responds to search requests from users using the optimized search parameters determined in Phase 1. First, the information processing device 100 receives a search query from the user terminal device 10 (S5: query).

[0021] Next, the information processing device 100 uses the optimized search parameters saved in Phase 1 to perform a search on the target document 162 (S6: Search Execution). Specifically, it integrates the similarity of text search and image search according to the optimized weighting ratio (for example, by integrating using ray fusion technology) and extracts document fragments (chunks) that are highly relevant to the search query. This enables highly accurate multimodal search that is suited to the characteristics of the target document 162.

[0022] Finally, the information processing device 100 sends the chunks obtained as search results to the LLM server 200, causing the LLM server 200 to generate an answer to the search query. The generated answer is sent to the user terminal device 10 (S7: Answer). In this way, the system employs a RAG architecture and can provide users with accurate and natural answers based on optimized search results.

[0023] [Functional Configuration of Information Processing Equipment] Figure 2 is a block diagram showing an example of the functional configuration of the information processing device 100. The information processing device 100 includes, for example, an input / output unit 110, an evaluation data generation unit 120, a vector conversion unit 130, an optimization unit 140, a search execution unit 150, and a storage unit 160. These components are functional modules realized by a hardware processor, such as a CPU (Central Processing Unit) in the information processing device 100, reading and executing a program (software) stored in the storage unit 160. Some or all of these components may be realized by dedicated hardware (including circuitry) such as an ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or GPU (Graphics Processing Unit).

[0024] The memory unit 160 is a storage device such as an HDD (Hard Disk Drive), SSD (Solid State Drive), or RAM (Random Access Memory), and stores various data and programs necessary for processing. For example, the memory unit 160 stores the target document 162, the prompt 164, the evaluation data 166, and the optimization search parameters 168.

[0025] The target document 162 is multimodal data to be searched and is an example of "target data" in the claims. The prompt 164 is instruction information for instructing the LLM server 200 to process, and in particular includes a prompt for generating evaluation data (see Figure 3 below). The evaluation data 166 is a dataset that includes evaluation queries and correct answer information, which is automatically generated by the evaluation data generation unit 120 (see Figure 4 below). The optimized search parameters 168 are the optimal search parameters determined by the optimization unit 140 (see Figure 6 below).

[0026] The input / output unit 110 is a communication interface that performs data input and output with external devices. For example, it retrieves the target document 162, sends and receives queries and answers with the user terminal device 10, and sends and receives prompts and inference results with the LLM server 200.

[0027] The evaluation data generation unit 120 generates evaluation data 166 based on the content of the target document 162. The evaluation data generation unit 120 is an example of a "generation unit" in the claims. The evaluation data generation unit 120 works in cooperation with the LLM server 200 to analyze the content of the target document 162 using a large-scale language model, generates question sentences related to the content as evaluation queries, and generates answers to those question sentences as correct answer information.

[0028] In particular, in this embodiment, the evaluation data generation unit 120 generates location information (e.g., page number, section number, etc.) from the content of the target document 162 as an answer to the question. When generating location information, the evaluation data generation unit 120 may generate not only a single location piece (e.g., a single page number) but also multiple location pieces (e.g., multiple page numbers) as an answer to the question, or it may generate location information from multiple different layers (e.g., a section number on a certain page and the page number on the next page) as an answer to the question.

[0029] The vector conversion unit 130 converts text data and image data into feature vectors (embedding vectors) that represent their semantic content. The vector conversion unit 130 converts the text portion of a chunk into a text vector and the image portion (for example, a chart image or graph image) into an image vector. This makes it possible to calculate not only the similarity with text vectors (text similarity) but also the similarity with image vectors (image similarity) for text-based search queries.

[0030] The optimization unit 140 optimizes search parameters to control the search process for the target document 162 so that the search accuracy using the evaluation data 166 is improved. Specifically, the optimization unit 140 optimizes the search parameters based on the text similarity between the evaluation query vector and the text vector, and the image similarity between the evaluation query vector and the image vector.

[0031] The search parameters include a first weight (α) that adjusts the contribution of text similarity in the search process, and a second weight (β) that adjusts the contribution of image similarity. The optimization unit 140 calculates an integrated score by adding the value obtained by multiplying the text similarity by the first weight α and the value obtained by multiplying the image similarity by the second weight β, and optimizes the search parameters based on this integrated score. More specifically, the optimization unit 140 performs a search simulation on the evaluation data 166 while changing the values ​​of the search parameters (α, β), and optimizes the search parameters so that the proportion of correct information (e.g., Recall@k) that is included within the top predetermined number (k) of search results ranked by the integrated score increases.

[0032] The search execution unit 150 performs a search on the target document 162 in response to a search request from the user. The search execution unit 150 is an example of the "search unit" in the claims. The search execution unit 150 receives a search query from the user terminal device 10 and, based on the search query, performs a search on the target document 162 according to optimized search parameters (optimized search parameters 168).

[0033] [Details of the information processing device 100's processing] Next, with reference to Figures 3 to 6, the details of the processing performed by the information processing device 100 will be described. Figure 3 shows an example of a prompt 164 that the evaluation data generation unit 120 inputs to the LLM server 200. The prompt 164 includes, for example, an instruction area A1 and a procedure area A2. In instruction area A1 (#instruction), the purpose of the task of generating evaluation data including an evaluation query (questionnaire) and correct answer information is defined.

[0034] In procedure area A2 (#Procedure), specific steps are instructed. In procedure (1), the instruction is to "identify the correct information for the target document as the page number (location information) of the target document." The LLM understands the content of the input document, generates various questions (evaluation queries), and identifies the location (page number, etc.) where the answers are written as the correct information.

[0035] Step (2) instructs the system to "convert the content of the identified correct position into a text vector corresponding to the text portion and an image vector corresponding to the image portion, and generate it as a chunk." This vector conversion may be performed by the vector conversion unit 130 of the information processing device 100. Step (3) instructs the system to "associate the evaluation query with the correct information (chunk, text vector, image vector) to generate evaluation data."

[0036] Figure 4 shows an example of the structure of the generated evaluation data 166. The evaluation data table 166A maintains the correspondence between evaluation queries and correct answer information (chunk IDs). For example, the query "How many days of paid leave can I take each year?" is associated with the correct answer chunk ID "C007".

[0037] The chunk information table 166B holds detailed information for each chunk. For example, the chunk information table 166B includes "chunk ID," "original data information (such as page number)," "text vector," and "image vector." In this way, evaluation data 166 for objectively evaluating search accuracy is automatically prepared. That is, the correct answer information is converted into text vectors corresponding to the text portion and image vectors corresponding to the image portion and stored.

[0038] Next, the optimization unit 140 uses the generated evaluation data 166 to search for the optimal values ​​of the search parameters (first weight α, second weight β). The operation of this search process will be explained below based on the main mathematical formulas that constitute it.

[0039] First, the optimization unit 140 calculates the similarity between the query and the chunk. In this embodiment, the optimization unit 140 can map text and images to a common vector space by using a multimodal embedding model. Hereinafter, the symbol q represents the user's query (text), and the symbol c represents a candidate chunk within the target document. Chunk c is part of the text Ctext and the image part C image is composed of

[0040] First, the optimization unit 140 calculates two types of similarities between the query q and the chunk c, namely the text similarity S tt (q, c) and the image similarity S ti (q, c). The cosine similarity is used as the similarity index

[0041] The text similarity S text between the query text q and the text part C of the chunk tt (q, c) is calculated according to the following (Equation 1).

[0042]

Equation

[0043] The image similarity (text-to-image similarity) S image between the query text q and the image part C of the chunk ti (q, c) is calculated according to the following (Equation 2).

[0044]

Equation

[0045] Here, "·" represents the inner product of vectors, and ||·|| represents the vector norm (e.g., L2 norm). These similarities (S tt , S ti ) indicate how semantically consistent the query is with the text content and image content of the chunk

[0046] Next, the optimization unit 140 uses the calculated text similarity S tt (q, c) and the image similarity S ti(q, c) is integrated to calculate a combined score (weighted fusion score) that shows the final relevance between query q and chunk c. In this embodiment, weighted linear addition based on late fusion technology is used. Combined score Score α,β (q,c) is calculated using the following equation (3).

[0047]

number

[0048] Here, α is the text similarity S. tt The weight for (first weight), β is the image similarity S ti This is the weight (second weight) for α. Equation (3) allows adjustment of which to trust more (give more weight to)—text information or image information—to suit the domain of the documents being searched and the characteristics of the dataset. A larger value of α means that the results of text searches are given more weight, and a larger value of β means that the results of image searches are given more weight.

[0049] The optimization unit 140 automatically determines the optimal combination of search parameters (α, β) to maximize the search accuracy using the evaluation data 166. First, let ε be the evaluation dataset automatically generated by the evaluation data generation unit 120. ε consists of an evaluation query q and a set of correct chunks R for that query. q This is the set of pairs (ε={(q,R q )}) .

[0050] Next, we define a metric for evaluating search accuracy. Here, we use Recall@k. Recall@k is defined as the number of ground truth chunks (R) within the top k search results. q This is an index that shows the proportion of chunks included in the search. Given certain search parameters (α, β), the combined score (Equation 3) α,β Rank the chunks sorted in descending order based on (q,c). α,βLet (q). The optimization unit 140 searches for a combination of search parameters (α*, β*) that maximizes the average Recall@k across the entire evaluation dataset ε. This is formulated as the optimization problem shown in (Equation 4) below.

[0051]

number

[0052] Here, argmax represents the operation of finding the search parameters (α*, β*) that maximize the objective function (mean Recall@k). Equation 4 represents a mechanism that automatically adjusts the search parameters (α, β) so that Recall@k, an objective search accuracy metric, is maximized, using an evaluation dataset ε automatically generated by LLM or the like.

[0053] Figure 5 shows an overview of the execution of the search simulation. The optimization unit 140 executes a specific search algorithm to find the optimal solution to (Equation 4). For example, in Figure 5, the optimization unit 140 uses grid search (brute-force search) to change the value of α from 0.0 to 1.0 in predetermined increments (e.g., increments of 0.1), performs a search simulation for each value (β=1-α), and calculates the average Recall@k. In the example in Figure 5, the search simulation results show that the average Recall@k is maximized when the settings are α=0.8 and β=0.2. As a result, the optimization unit 140 determines the search parameters (α, β) = (0.8, 0.2) as the optimized search parameters (α*, β*).

[0054] Figure 6 shows an example of the structure of the determined optimization search parameters 168. The optimization unit 140 repeatedly performs search simulations and calculates the search accuracy (average Recall@k) for all candidate values ​​and records the results. As mentioned above, in the example in Figure 5, the average Recall@k is 0.65 when (α=1.0, β=0.0) (text search only), but the average Recall@k is maximized at 0.82 when (α=0.8, β=0.2). This indicates that combining text and images in an 8:2 ratio is optimal for the target document 162.

[0055] The optimization unit 140 determines the search parameters (α, β) obtained as the optimal solution to (Equation 4) as the optimized search parameters (α*, β*). In the example in Figure 6, the search parameters (α, β) = (0.8, 0.2) are determined. The determined optimized search parameters 168 are stored in the storage unit 160 and used when the search is executed in phase 2.

[0056] The search execution unit 150 receives a search query for the target document 162 from the user terminal device 10 and executes a search process for the target document 162 according to the optimized search parameters 168 stored in the storage unit 160. More specifically, first the vector conversion unit 130 converts the search query into a vector, and the search execution unit 150 selects chunks that have a high correlation with the converted vector (i.e., the integrated score Score in (Equation 3)). α,β The search execution unit 150 extracts the chunk where (q,c) is maximized. At this time, the search execution unit 150 may extract not just a single chunk, but multiple chunks (for example, the top k items). Once the search execution unit 150 has extracted the chunks, it sends the extracted chunks to the LLM server 200 and causes the LLM server 200 to generate a response to the search query. When the search execution unit 150 receives the response from the LLM server 200, it sends the received response to the user terminal device 10.

[0057] [Process Flow] Next, with reference to Figures 7 and 8, the processing flow performed by the information processing device 100 will be described. Figure 7 is a flowchart showing the flow of the search parameter optimization process. The processing shown in the flowchart of Figure 7 is performed, for example, when a new target document 162 to be searched is added or updated.

[0058] First, the information processing device 100 acquires the target data (S100). Next, the information processing device 100 generates evaluation data based on the content of the target data (S102). Next, the information processing device 100 sets candidate values ​​for the search parameters (α, β) (S104). Next, it performs a search simulation on the evaluation data using the set candidate values ​​(S106). Specifically, it calculates the similarity using (Equation 1) and (Equation 2), calculates the integrated score using (Equation 3), and generates a ranking.

[0059] Next, the information processing device 100 calculates the search accuracy (e.g., Recall@k) based on the results of the search simulation (S108). Next, the information processing device 100 determines whether or not the search simulation has been performed for all candidate values ​​(S110). If there are candidate values ​​that have not been simulated (S110: NO), the process returns to S104. If the simulation has been performed for all candidate values ​​(S110: YES), the information processing device 100 determines the search parameters (α, β) that maximize the search accuracy as the optimization parameters (α*, β*) (S112). This corresponds to the process of finding the optimal solution in (Equation 4). This completes the process in this flowchart.

[0060] Next, with reference to Figure 8, the details of the search process in Phase 2 will be explained. Figure 8 is a flowchart showing an example of the search process flow using optimized search parameters. The process shown in the flowchart in Figure 8 is executed, for example, when a user inputs a search query for the target document 162 using the user terminal device 10 and sends it to the information processing device 100.

[0061] First, the information processing device 100 receives a search query from the user terminal device 10 (S200). Next, the information processing device 100 vectorizes the search query (S202). Next, for each chunk, the information processing device 100 calculates the text similarity between the search query vector and the vector (S202). tt The system calculates the image similarity (Sti) (S204). This corresponds to the calculations of (Equation 1) and (Equation 2). Next, the information processing device 100 reads the search parameters (α*, β*) optimized in phase 1 from the storage unit 160 (S206).

[0062] Next, the information processing device 100 calculates an integrated score for each chunk using the read optimization parameters (S208). This corresponds to the calculation in (Equation 3). This provides a relevance score with weights optimized to match the characteristics of the target document. Next, the information processing device 100 ranks the chunks based on the integrated score (S210). Finally, the information processing device 100 outputs the top k ranked chunks as search results (S212). The output chunks are used as contextual information for answer generation by the LLM server 200.

[0063] [Differentiation] (1) Searchable data In the embodiments described above, document files were used as an example of target data, but the scope of the present invention is not limited to this. For example, it can be applied to any data composed of multiple modalities, such as web pages, emails, chat logs, and videos (video frame images and text from speech recognition). For example, in video search, the weights of video (image vectors representing frame images) and audio (text vectors representing speech recognition text) may be automatically optimized.

[0064] (2) Number of modalities Furthermore, while the above embodiment showed an example of optimizing the weights (α, β) of two modalities (text and image), the same method can be applied when there are three or more modalities. For example, when there are three modalities—text, image, and audio—the weights (α, β, γ, α+β+γ=1) of each can be optimized.

[0065] (3) Generation of evaluation data The above embodiment shows an example of automatically generating evaluation data using LLM, but the method of generating evaluation data is not limited to this. For example, evaluation data may be generated semi-automatically from the user's past search history and browsing history. Alternatively, if high-quality evaluation data created manually exists, it may be used, or used in combination with automatic generation by LLM.

[0066] (4) System Configuration In the above embodiment, the information processing device 100, the user terminal device 10, and the LLM server 200 are configured as separate devices, but these configurations can be changed as appropriate. For example, the information processing device 100 and the LLM server 200 may be integrated into a single device. Furthermore, the functions of this embodiment may be provided as a cloud service (SaaS) or built as an on-premise server system.

[0067] [summary] As described above, the information processing device 100 according to this embodiment includes a generation unit (evaluation data generation unit 120) that generates evaluation data 166 including an evaluation query and correct answer information corresponding to the evaluation query based on the content of target data (target document 162) including text and images, and an optimization unit 140 that optimizes search parameters (α, β) for controlling the search process on the target data so as to improve the search accuracy using the evaluation data 166.

[0068] This allows the information processing device 100 to automatically determine the optimal weighting balance between text search and image search according to the characteristics of the multimodal document to be searched (e.g., whether it is primarily text-based or primarily image-based). As a result, the accuracy of vector search for multimodal documents can be improved.

[0069] Furthermore, in this embodiment, evaluation data necessary for evaluating search accuracy is automatically generated using a large-scale language model. This significantly reduces the enormous amount of time and effort previously required for creating evaluation data and tuning search parameters, which were previously done manually by experts. Because the system automatically optimizes search accuracy, it is easy to maintain an optimal state even when search target data is added or updated.

[0070] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of symbols]

[0071] 10. User terminal device 100 Information Processing Devices 110 Input / output section 120 Evaluation Data Generation Unit 130 Vector Transformation Unit 140 Optimization Unit 150 Search Execution Unit 200 LLM servers

Claims

1. A generation unit generates evaluation data that includes an evaluation query, which is a query for the target data, and correct answer information that shows a part of the target data corresponding to the evaluation query, based on the content of the target data, including text and images. The system includes an optimization unit that optimizes search parameters for controlling the search process on the target data so as to improve the search accuracy using the aforementioned evaluation data, The search parameters include, in the search process, a first weight for adjusting the contribution of text similarity between the evaluation query and the ground truth information, and a second weight for adjusting the contribution of image similarity between the evaluation query and the ground truth information. Information processing device.

2. The generation unit analyzes the content of the target data using a large-scale language model, generates question sentences related to the content as evaluation queries, and generates answers to the question sentences as correct answer information. The information processing apparatus according to claim 1.

3. The generation unit generates location information from the content of the target data as an answer to the question. The information processing apparatus according to claim 2.

4. The aforementioned correct answer information is converted into a text vector corresponding to the text portion of the target data and an image vector corresponding to the image portion of the target data and stored. The text similarity is the similarity between the evaluation query vector and the text vector, and the image similarity is the similarity between the evaluation query vector and the image vector. The information processing apparatus according to claim 1.

5. The optimization unit calculates an integrated score by adding the value obtained by multiplying the text similarity by the first weight and the value obtained by multiplying the image similarity by the second weight, and optimizes the search parameters based on the integrated score. The information processing apparatus according to claim 4.

6. The optimization unit performs a search simulation on the evaluation data while changing the values ​​of the search parameters, and optimizes the search parameters so that the proportion of the correct answer information being included within the top predetermined number of search results ranked by the integrated score increases. The information processing apparatus according to claim 5.

7. The system further includes a search unit that receives a search query for the target data from a user's terminal device and, based on the search query, executes a search process for the target data according to optimized search parameters. The information processing apparatus according to claim 1.

8. Computers Based on the content of the target data, which includes text and images, evaluation data is generated that includes an evaluation query, which is a query for the target data, and correct answer information that represents a part of the target data corresponding to the evaluation query. To improve the search accuracy using the aforementioned evaluation data, the search parameters for controlling the search process on the target data are optimized. The search parameters include, in the search process, a first weight for adjusting the contribution of text similarity between the evaluation query and the ground truth information, and a second weight for adjusting the contribution of image similarity between the evaluation query and the ground truth information. Information processing methods.

9. On the computer, Based on the content of the target data, which includes text and images, evaluation data is generated that includes an evaluation query, which is a query for the target data, and correct answer information that represents a part of the target data corresponding to the evaluation query. To improve the search accuracy using the aforementioned evaluation data, the search parameters for controlling the search process on the target data are optimized. The search parameters include, in the search process, a first weight for adjusting the contribution of text similarity between the evaluation query and the ground truth information, and a second weight for adjusting the contribution of image similarity between the evaluation query and the ground truth information. program.

Images