Document information retrieval system and program

Abstract

This document information retrieval system provides the ability to generate answers to user questions from diverse perspectives and in appropriate formats. [Solution] The document information retrieval system 10 provides multiple summary databases 41, 42, and 43 of varying granularity. A database selection means 27A uses a large-scale language model (LLM) to select the appropriate summary database according to the content of the user's question. A search means 27B retrieves a summary of the appropriate granularity using the selected summary database. An answer generation means 28 uses the large-scale language model (LLM) to create an answer to the user's question using the retrieved summary.

Description

Technical Field

[0001] The present invention relates to a document information retrieval system and a program configured by a computer that executes a process of retrieving information included in a document. For example, it can be used when retrieving information included in a document in a format that can describe text, graphics, or images in addition to these, such as a slide-form document.

Background Art

[0002] There is known a system that creates a summary of a document stored in advance using a large language model (LLM: Large Language Models) (see Patent Document 1). In this system, for example, reports related to medicine, business, research, etc. are used as documents to be summarized.

[0003] Also, as a system that realizes Retrieval-Augmented Generation (RAG) that combines a search function with a large language model (LLM), there is known a browsing support system for accurately describing medical-related information (see Patent Document 2). In this browsing support system, reference information is read, and it includes a large language model (LLM) that summarizes the reference information based on a prompt consisting of text or an image. The knowledge search unit outputs a summary document created by the large language model (LLM) to a medical-related information management system.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

[0010] )

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] As mentioned above, RAG technology has been implemented that uses large-scale language models (LLMs) to create document summaries, stores the summaries created using the LLMs in a database, and uses this database to perform search processing and generate answers to user questions.

[0006] However, this conventional method of generating answers using LLM with search processing based on summary databases suffers from a lack of diversity, as it fails to reflect different perspectives and interpretations of the same information source, resulting in one-sided information provision and a limited understanding for the user.

[0007] Furthermore, there is a problem of inadequate information adaptability, as information is not provided in the most suitable format according to the user's needs and circumstances, making it difficult to provide information that meets the intended purpose.

[0008] The objective of the present invention is to provide a document information retrieval system and program that can generate answers to user questions that provide information from diverse perspectives and in an appropriate format. [Means for solving the problem]

[0009] The present invention is a document information retrieval system comprising a computer that performs a process of retrieving information contained in a document, Multiple summary databases store text data of summaries for each granularity level, created using segmented document information obtained by dividing the information contained in the same document into multiple different granularities, and A database selection means receives user question data from a user terminal, creates a database selection request data that includes the received question data, a list of multiple searchable summary databases, and instructions on which summary database to use to answer the user's question, inputs the created database selection request data into a large language model, and receives a search query that includes the summary database to be used and search question data for each summary database output from this large language model. A search means that sends the search query received by this database selection means to a database search system, and receives a summary from this database search system obtained as a result of the search process performed using the summary database to be used. This search means creates a summary, user question data, and response creation request data including instructions for creating answers to the user's questions; inputs the created response creation request data into a large-scale language model; receives the response data to the user's questions output from the large-scale language model; and transmits the received response data to the user terminal. It is characterized by having the following features.

[0010] Here, "document" includes not only documents in a format that can contain text, figures, and images, but also documents in a format that can contain text but only one of either figures or images, or documents in a format that can contain only text. In this description, when we say "a document in a format that can contain...", we mean that the specifications of the software used to create the document, or the format of the file that stores the created document, allow the inclusion (input, pasting) of any of the information represented by text, figures, or images. Therefore, for example, when we say "a document in a format that can contain text, figures, and images", we do not mean a document in which all of the following are included. Rather, the document to be summarized can be any of the following: a document containing only text, a document containing text and figures, a document containing text and images, a document containing only figures (however, text is inserted within the figures), or a document containing only images (however, text is included within the images).

[0011] Furthermore, "user question data" is received from the user's terminal, while "search question data for each summary database" is output from a large-scale language model. Therefore, the two are different, but the content of the data may match.

[0012] In the document information retrieval system of the present invention, multiple summary databases of varying granularity are provided, a database selection means selects the appropriate summary database according to the content of the user's question, a search means obtains a summary of the appropriate granularity using the selected summary database, and an answer generation means creates an answer to the user's question using the obtained summary.

[0013] Therefore, when creating answers to user questions, summaries at multiple levels of granularity are used. This allows for the use of fine-grained summaries for detailed questions and coarse-grained summaries for general questions, enabling the provision of information from diverse perspectives and in appropriate formats, thereby achieving the aforementioned objectives.

[0014] <Configuration equipped with means for determining whether or not an answer can be generated>

[0015] Furthermore, in the aforementioned document information retrieval system, It is desirable to have a configuration that includes a response generation feasibility determination means that creates response feasibility determination data, which includes a summary received by the search means, user question data, and an instruction sentence asking whether or not it is possible to answer the user's question using this information; inputs the created response feasibility determination data into a large-scale language model; receives response feasibility determination information output from the large-scale language model; and if the received response feasibility determination information indicates that the question cannot be answered, executes a process to return to the processing of the database selection means.

[0016] In this configuration, which includes a means for determining whether or not to generate an answer, the process by the database selection means can be repeated until the selection process of the summary database to be used by the database selection means reaches an appropriate state. This makes it possible to provide information from a wider range of perspectives and in an appropriate format.

[0017] <Configuration equipped with a means for generating summaries>

[0018] Furthermore, in the aforementioned document information retrieval system, The process involves creating segmented document information obtained by dividing the information contained in the document into the finest first level of granularity, and creating a first summary creation request data that includes instructions for creating a summary of the segmented document information. The created first summary creation request data is then input into a large-scale language model, and the first-level summary output from the large-scale language model is received. Furthermore, let N be an integer greater than or equal to 2, create Nth summary creation request data including an instruction to create summaries of the (N - 1)th granularity and a summary of the Nth granularity using these summaries of the (N - 1)th granularity, input the created Nth summary creation request data into a large language model, receive the summary of the Nth granularity output from the large language model, and repeat this process until N becomes the number of the largest granularity. It is desirable to adopt a configuration provided with summary creation means for executing a process of storing the created summaries of each granularity in a summary database corresponding to the granularity.

[0019] When adopting such a configuration provided with summary creation means, it becomes possible to use a large language model (LLM) to sequentially create summaries with coarser granularities from summaries with finer granularities, and it becomes possible to avoid or reduce manual work when preparing summary databases of multiple granularities.

[0020] <Configuration for creating a summary of the first granularity using the intention obtained from the arrangement of figures and the appearance information obtained by image analysis>

[0021] Also, in the configuration provided with the above-described summary creation means, Create figure intention acquisition request data including an instruction to extract and explain the information of figures containing characters extracted from the segmented document information segmented at the first granularity and the intention obtained from the arrangement of the figures, input the created figure intention acquisition request data into a large language model, and a figure intention acquisition means for receiving the intention output from the large language model; Create appearance information acquisition request data including an instruction to create image data obtained by imaging the segmented document information segmented at the first granularity and appearance information obtained by analyzing the image data, input the created appearance information acquisition request data into a large language model, and a multi-modal information processing means for receiving the appearance information output from the large language model. The summary creation means It is possible to adopt a configuration in which first summary creation request data is created, including the intention obtained by the graphic intention acquisition means, the appearance information obtained by the multimodal information processing means, and an instruction statement indicating that a summary is to be created by interpreting the content of the classified document information with these pieces of information.

[0022] Here, as a component of the "classified document information obtained by classifying at the first granularity", if there is text data outside a graphic (a graphic including characters), that text data is also included in the "first summary creation request data". Therefore, in that case, the "first summary creation request data" includes the intention obtained from the arrangement of the graphic, the text data outside the graphic, the appearance information obtained by image analysis, and the instruction statement.

[0023] When adopting a configuration in which a summary at the first granularity is created using the intention obtained from the arrangement of the graphic and the appearance information obtained by image analysis, it becomes possible to accurately grasp the information included in a document in a format capable of describing text and graphics and create a summary at the first granularity.

[0024] <Configuration for creating a summary at the first granularity using the intention obtained from the arrangement of the graphic, the appearance information obtained by image analysis, and the already created summary for the immediately preceding classified document information>

[0025] Furthermore, in a configuration provided with the above-described summary creation means, the summary creation means can be configured to create first summary creation request data including the intention obtained by the graphic intention acquisition means, the appearance information obtained by the multimodal information processing means, the already created summary for the immediately preceding classified document information, and an instruction statement indicating that a summary is to be created by interpreting the content of this classified document information with these pieces of information.

[0026] In this configuration, where the intention derived from the arrangement of shapes, the appearance information obtained from image analysis, and the previously created summary of the preceding sectioned document information are used to create the first-level summary, contextual information obtained from the previously created summary of the preceding sectioned document information is added. This makes it possible to more accurately grasp the information contained in a document in a format that allows for the inclusion of text and shapes, and to create the first-level summary.

[0027] <Configuration for analyzing graphic information together with image data using multimodal information processing>

[0028] Furthermore, in a configuration equipped with the aforementioned summarization means, The system includes a multimodal information processing means that creates integrated information acquisition request data which includes information on shapes containing characters extracted from segmented document information obtained by segmenting at a first level of granularity, image data obtained by converting the segmented document information obtained by segmenting at a first level of granularity into images, and an instruction statement requesting the creation of integrated information obtained by analyzing the shape information and image data; inputs the created integrated information acquisition request data into a large-scale language model; and receives integrated information output from the large-scale language model. The means of creating a summary is, The system may also be configured to create a first summary creation request data that includes integrated information obtained by a multimodal information processing means, a previously created summary of the immediately preceding segmented document information, and instructions requesting that the content of this segmented document information be interpreted using this information to create a summary.

[0029] Here, as a component of the "sectional document information obtained by sectionalizing at the first level of granularity," if there is text data outside of a shape (a shape containing text), that text data is also included in the "integrated information acquisition request data" or the "first summary creation request data."

[0030] In this configuration, where graphic information is analyzed together with image data using a multimodal information processing system, the multimodal information processing system analyzes the graphic information and image data to obtain integrated information. Using this integrated information and the previously created summary for the previously identified document information, a first-level summary of this document information is created. Therefore, in this case as well, contextual information obtained from the previously created summary for the previously identified document information is added, making it possible to more accurately grasp the information contained in a document in a format that allows for the inclusion of text and graphics, and to create a first-level summary.

[0031] <Invention of a program>

[0032] Furthermore, the program of the present invention is intended to enable a computer to function as the document information retrieval system described above.

[0033] Furthermore, the above program or any part thereof can be recorded and stored or distributed on recording media such as magneto-optical disks (MO), compact discs (CD), digital versatile disks (DVD), flexible disks (FD), magnetic tape, read-only memory (ROM), electrically erasable and rewritable read-only memory (EEPROM), flash memory, random access memory (RAM), hard disk drives (HDD), solid state drives (SSD), and flash disks. It can also be transmitted using transmission media such as wired networks like local area networks (LAN), metropolitan area networks (MAN), wide area networks (WAN), the Internet, intranets, extranets, wireless communication networks, or combinations thereof, and can also be transmitted on carrier waves. Moreover, the above program may be part of another program, or may be recorded on a recording media together with a separate program. [Effects of the Invention]

[0034] As described above, according to the present invention, multiple summary databases of varying granularity are prepared, a database selection means selects the appropriate summary database according to the content of the user's question, a search means obtains a summary of appropriate granularity using the selected summary database, and an answer generation means creates an answer to the user's question using the obtained summary. This has the effect of enabling the provision of information from diverse perspectives and in appropriate formats. [Brief explanation of the drawing]

[0035] [Figure 1] An overall configuration diagram of a document information retrieval system according to one embodiment of the present invention. [Figure 2] A diagram illustrating the usage scenarios of the large-scale language model (LLM) of the above embodiment. [Figure 3] A diagram illustrating the construction of multiple summary databases of different granularities according to the above embodiment. [Figure 4] A flowchart illustrating the document information retrieval process (part 1) of the above embodiment. [Figure 5] A flowchart illustrating the document information retrieval process (part 2) of the above embodiment. [Figure 6] This figure shows an example of the data for obtaining the graphic intent in the LLM usage scenario (1) of the above embodiment and the LLM response data. [Figure 7] This figure shows an example of the appearance information acquisition request data and the LLM response data in the LLM usage scenario (2) of the above embodiment. [Figure 8] This figure shows an example of the first summary creation request data and its LLM response data in the LLM usage scenario (3A) of the above embodiment. [Figure 9] This figure shows an example of the second summary creation request data and its LLM response data in the LLM usage scenario (3B) of the above embodiment. [Figure 10] This figure shows an example of the third summary creation request data and its LLM response data in the LLM usage scenario (3C) of the above embodiment. [Figure 11] This figure shows an example of the database selection request data and the LLM response data in the LLM usage scenario (4) of the above embodiment. [Figure 12] This figure shows an example of the data for determining whether a response is possible and the LLM response data in the LLM usage scenario (5) of the above embodiment. [Figure 13] A figure showing an example of the response creation request data and the LLM response data in the LLM usage scenario (6) of the above embodiment. [Modes for carrying out the invention]

[0036] One embodiment of the present invention will be described below with reference to the drawings. Figure 1 shows the overall configuration of the document information retrieval system 10 of this embodiment. Figure 2 is an explanatory diagram of the usage scenario of the Large-Scale Language Model (LLM), and Figure 3 is an explanatory diagram of the construction of multiple (in this embodiment, three as an example) summary databases 41, 42, and 43 of different granularities. Figures 4 and 5 show the flow of the document information retrieval process in flowcharts, and Figures 6 to 13 show examples of input data to the LLM and the LLM response data in each LLM usage scenario (1) to (6).

[0037] <Overall configuration of document information retrieval system 10>

[0038] In Figure 1, the document information retrieval system 10 includes a document information retrieval server 20 composed of one or more computers. This document information retrieval server 20 is connected via network 1 to a database search system 40, a service provision system 50 using large language models (LLMs), and a service provision system 51 that performs vectorization processing. Furthermore, the document information retrieval server 20 is connected via network 1 to a creator terminal 60 operated by the creator of the summary to be registered in the database search system 40, and to a user terminal 70 operated by the user (questioner).

[0039] Here, Network 1 is an external network primarily composed of the Internet, but it may also be a combination of the Internet and internal networks such as LANs or intranets, and it does not matter whether it is wired, wireless, or a hybrid of wired and wireless; in short, it is sufficient if it can transmit information at a reasonable speed between multiple locations (regardless of distance).

[0040] Furthermore, the database search system 40 is an external service provision system composed of one or more computers, which receives search queries containing question data (in this application, this is not the user's question data itself, but question data created by a Large-Scale Language System (LLM). Details will be described later) via the network 1, retrieves one or more summaries (text data) corresponding to (similar to) the received question data (text data) from the summary databases 41, 42, 43 (see Figure 3), and executes a process to return all retrieved summaries to the source of the search query via the network 1 as a response to the search query. In this embodiment, the number of summaries retrieved from one summary database is set in advance by the system (for example, 3), but it may also be set to a number specified by the user from the user terminal 70 (for example, the user specifies between 1 and 10). Furthermore, while Azure AI Search is used as the database search system 40 in this embodiment, it is not limited to this. In short, the database search system 40 can be any system that can obtain summaries corresponding to (similar to) the question data included in the search query using multiple summary databases 41, 42, 43 (see Figure 3) of varying granularity prepared by the creator. The summary databases 41, 42, 43 are composed of non-volatile memory such as hard disk drives (HDDs) and solid-state drives (SSDs). Details of the configuration of the summary databases 41, 42, 43 will be described later with reference to Figure 3.

[0041] Furthermore, the service provision system 50 using a Large-Scale Language Model (LLM) is a computer system that provides cloud API (Application Programming Interface) services. In this embodiment, the Chat Generative Pre-trained Transformer (ChatGPT, Azure OpenAI ChatCompletion API GPT-4o) is used, but it is not limited to this. For example, OpenAI's GPT-3.5, Google's Palm and Palm2, Amazon Web Services (AWS)'s Titan, Meta Platforms' Llama, etc. may also be used.

[0042] Furthermore, the service provision system 51 is an external service provision system via a cloud API connected through network 1, and is a computer system that performs vectorization processing of text data. In this embodiment, as an example, the Azure OpenAI Embeddings API service provision system (Azure OpenAI Embeddings API text-embedding-ada-002 version2) can be used, but it is not limited to this, and the embedding method may be Doc2Vec, BERT, Transformer, etc. The vector has a fixed length, but the number of dimensions is arbitrary.

[0043] Furthermore, the creator terminal 60 and the user terminal 70 are composed of computers and are equipped with display means such as an LCD display and input means such as a mouse and a keyboard. These creator terminal 60 and user terminal 70 may be notebook personal computers or portable devices such as smartphones and tablets.

[0044] <Configuration of Document Information Search Server 20>

[0045] The document information retrieval server 20 performs various processes related to retrieving information contained in documents and stores data necessary for these processes. It is composed of one or more computers and includes a document storage means 21, a document analysis means 22, a graphic intent acquisition means 23, an imaging means 24, a multimodal information processing means 25, a summary creation means 26, a search management means 27, a response generation means 28, and a document storage means 31. The search management means 27 performs processes to manage the search process and includes a database selection means 27A, a search means 27B, and a response generation feasibility determination means 27C.

[0046] Here, the document storage means 21, document analysis means 22, graphic intent acquisition means 23, image creation means 24, multimodal information processing means 25, summary creation means 26, search management means 27, and answer generation means 28 are implemented by a central processing unit (CPU) located inside the document information retrieval server 20, one or more programs that define the operating procedures of this CPU, and working memory such as main memory and cache memory. Details of each of these means 21 to 28 will be described later.

[0047] Furthermore, the document storage means 31 can be, for example, a non-volatile memory such as a hard disk drive (HDD) or a solid-state drive (SSD).

[0048] <Configuration of Document Information Search Server 20 / Document Storage Means 21>

[0049] The document storage means 21 receives data of documents to be summarized, which are transmitted via the network 1 from the creator terminal 60 by the creator of the summaries to be registered in the summary databases 41, 42, and 43, and executes the process of storing the data in the document storage means 31. The creator of the summaries operating the creator terminal 60 may be the same person as the user (the person performing the search) operating the user terminal 70, or they may be different people.

[0050] In this embodiment, the document to be summarized is described as a slide-format document such as a presentation (e.g., a document stored in a pptx file), but it is not limited to this. For example, it may be a Word or PDF document, in short, any document in a format that can contain text, figures, and images is acceptable. As already detailed in [Means for Solving the Problem], the document in the present invention is not limited to the "document in a format that can contain text, figures, and images" used in the description of this embodiment.

[0051] In slide-format documents stored in files with the extension .pptx (PowerPoint files), the information about shapes is the information of the shape object (hereinafter referred to as shape information), and shape information includes the type of shape, the text data within the shape, and the position (coordinates) of the shape. The text data within a shape refers to, for example, the text data written inside a rectangular box, so a shape is a shape into which text can be inserted (a shape into which text data can be inserted).

[0052] Furthermore, in slide-format documents, all text data is often contained within shapes, resulting in no text data outside the shapes. However, in documents such as Word or PDF, text data is more often present outside the shapes. This invention also covers documents where text data is present outside the shapes. In this application, tables are also included as shapes.

[0053] Furthermore, in this invention, multiple summaries of different granularities (three in this embodiment) are created from a single document. In this process, to create each summary of different granularities, it is necessary to divide (split) the information contained in the same document into multiple different granularities and prepare divided document information. In this embodiment, in order to create multiple (three) summaries of different granularities, the first granularity summary is a slide-based summary, the second granularity summary is a slide group-based summary, and the third granularity summary is a file-based summary (for example, a document unit such as presentation materials). Therefore, one slide (first granularity) and one file (third granularity) are determined without any special specification, but it is necessary to specify how many slides (from which slide to which slide) constitute a slide group (second granularity). This designation information may be pre-entered by the summary creator and stored in the document storage means 31 in association with the file containing the document, or the summary creator may input the designation information from the creator terminal 60 during processing by the summary creation means 26, or a system setting may be configured so that one slide group consists of a fixed number of slides. The same applies when multiple levels of granularity are different from slide units, slide group units, and file units (material units), and the designation information for the classification may be stored in the document storage means 31, or the summary creator may input it during processing by the summary creation means 26, or it may be fixed information.

[0054] <Configuration of Document Information Retrieval Server 20 / Document Analysis Means 22>

[0055] The document analysis means 22 extracts shape information (shape type, text data within the shape, shape position (coordinates)) from segmented document information (in this embodiment, information from any single slide, which is the first granularity) obtained by segmenting the information contained in the document at a first granularity. The number of extracted shape information items may be one, but is often multiple. This shape information extraction process is performed by a known document analysis program. As a document analysis program, for example, python-pptx, https: / / python-pptx.readthedocs.io / en / latest / index.html (python is a registered trademark), etc. can be used, but is not limited to these. In this application, the term "segmented document information" is used not only for information segmented at the first granularity, but also for information segmented at other granularities (second, third, ... granularities).

[0056] The processing by the document analysis means 22 is performed by receiving selection information of the documents to be summarized, which is transmitted from the creator terminal 60 via the network 1. The documents to be summarized are usually documents created by the author of the summary and stored in the document storage means 31 by the document storage means 21. However, if the author of the summary is, for example, a system administrator or someone in a position to operate documents created in their own department, then documents created by someone else and stored in the document storage means 31 may also be used.

[0057] <Configuration of Document Information Search Server 20 / Graphic Intent Acquisition Means 23>

[0058] The graphic intent acquisition means 23 creates graphic intent acquisition request data as a prompt, as shown in LLM usage scenario (1) in Figure 2, transmits the created graphic intent acquisition request data to the service provision system 50 using a large-scale language model (LLM) via the network 1, and performs the process of receiving LLM response data transmitted from the service provision system 50 via the network 1. Here, as shown in Figure 6, the graphic intent acquisition request data in LLM usage scenario (1) includes information on a graphic containing text extracted from the segmented document information obtained by segmenting at a first granularity (in this embodiment, shape information extracted by the document analysis means 22), and an instruction sentence asking the user to understand and explain the intent derived from the arrangement of the graphic. The LLM response data in LLM usage scenario (1) is text data that interprets the intent derived from the arrangement of the shapes (graphics containing text).

[0059] <Configuration of Document Information Search Server 20 / Image Processing Means 24>

[0060] The imaging means 24 performs the process of creating image data by imaging the segmented document information (in this embodiment, one slide) obtained by segmenting at the first granularity. The image data is, for example, data with the extension jpeg or gif.

[0061] Furthermore, the processing by the image conversion means 24 is performed by receiving selection information of the document to be summarized, which is transmitted from the creator terminal 60 via the network 1, similar to the processing by the document analysis means 22 described above.

[0062] <Configuration of Document Information Retrieval Server 20 / Multimodal Information Processing Means 25>

[0063] The multimodal information processing means 25 creates appearance information acquisition request data as a prompt, as shown in LLM usage scenario (2) in Figure 2, transmits the created appearance information acquisition request data to the service provision system 50 using a large-scale language model (LLM) via network 1, and performs the process of receiving LLM response data transmitted from the service provision system 50 via network 1. Here, as shown in Figure 7, the appearance information acquisition request data in LLM usage scenario (2) includes image data created by the imaging means 24 (image data obtained by imaging the segmented document information obtained by segmenting at a first granularity) and an instruction statement requesting that appearance information be created by analyzing the image data. The LLM response data in LLM usage scenario (2) is appearance information obtained by image analysis.

[0064] The multimodal information processing means 25 performs processing that utilizes the multimodal function of the generating AI (Artificial Intelligence), which is a function that performs processing that combines different types of data, such as text, images, audio, and video.

[0065] Therefore, this multimodal function can be used not only when you want to create appearance information obtained by analyzing image data as described above, but also when you want to create integrated information obtained by analyzing information other than image data. Specifically, the multimodal information processing means 25 may be configured to, for example, add shape information (type of shape, text data within the shape, position (coordinates) of the shape) extracted by the document analysis means 22 as a prompt, create integrated information acquisition request data that includes graphic information (shape information), image data created by the imaging means 24, and an instruction that it wants to create integrated information obtained by analyzing this graphic information and image data, transmit the created integrated information acquisition request data to the service provision system 50 using a large-scale language model (LLM) via the network 1, and execute a process to receive LLM response data (integrated information) transmitted from the service provision system 50 via the network 1.

[0066] Furthermore, since the multimodal function can analyze image data, it is possible to include not only the image data created by the imaging means 24 (sectional document information obtained by dividing it at the first granularity, i.e., image data obtained by imaging the entire slide), but also images that are originally written (attached) in the document (usually images that occupy only a part of a slide, not the entire slide) in the appearance information acquisition request data. In addition, the position information (coordinates) of the image on the slide may also be included in the appearance information acquisition request data.

[0067] <Configuration of Document Information Retrieval Server 20 / Summary Creation Means 26>

[0068] The summary creation means 26 creates summaries (text data) of multiple levels of granularity (three in this embodiment) for information contained in the same document, and stores each level of summary in a separate summary database. Specifically, in this embodiment, the first level of granularity (slide unit) summary is stored in summary database 41, the second level of granularity (slide group unit) summary is stored in summary database 42, and the third level of granularity (file unit) summary is stored in summary database 43. The summary creation means 26 also uses the service provision system 51 to create vector data for each level of summary created, and stores the vector data of the created summaries in summary databases 41, 42, and 43 (see Figure 3) in association with the summaries (text data). This vector data is used in the search process by the database search system 40.

[0069] More specifically, the summarization means 26 first creates a first summary creation request data as a prompt, as shown in the LLM usage scenario (3A) in Figure 2, transmits the created first summary creation request data to the service provision system 50 using a large-scale language model (LLM) via the network 1, and performs the process of receiving LLM response data transmitted from the service provision system 50 via the network 1. Here, as shown in Figure 8, the first summary creation request data in the LLM usage scenario (3A) includes segmented document information obtained by segmenting the information contained in the document at the finest first granularity (specifically, the intent obtained from the arrangement of figures obtained by the figure intent acquisition means 23, the appearance information obtained from image analysis obtained by the multimodal information processing means 25, and a previously created summary of the previous segmented document information (information of the previous slide)), and an instruction requesting that a summary be created for this segmented document information (this slide). The LLM response data in the LLM usage scenario (3A) is a summary (text data) at the first granularity, specifically a summary of the information on one slide. The process of creating this first level of summary (slide unit) is the process shown in step S5 of Figure 3.

[0070] Furthermore, if text data exists outside of a shape as part of the segmented document information obtained by segmenting at the finest first level of granularity, that text data will also be included in the first summary creation request data.

[0071] Next, the summarization means 26 creates a second summary creation request data as a prompt, as shown in the LLM usage scenario (3B) in Figure 2, and transmits the created second summary creation request data to the service provision system 50 using a large-scale language model (LLM) via the network 1. It then executes the process of receiving LLM response data transmitted from the service provision system 50 via the network 1. Here, as shown in Figure 9, the second summary creation request data in the LLM usage scenario (3B) includes multiple summaries (text data) of a first granularity (slide unit) and an instruction requesting that these multiple summaries of the first granularity be combined to create a summary (text data) of a second granularity (slide group unit). The LLM response data in the LLM usage scenario (3B) is a summary (text data) of the second granularity, and specifically, it is a summary of the information of one slide group.

[0072] Furthermore, the summarization means 26 creates a third summary creation request data as a prompt, as shown in the LLM usage scenario (3C) in Figure 2, transmits the created third summary creation request data to the service provision system 50 using a large-scale language model (LLM) via the network 1, and performs the process of receiving LLM response data transmitted from the service provision system 50 via the network 1. Here, as shown in Figure 10, the third summary creation request data in the LLM usage scenario (3C) includes summaries (text data) of multiple second granularities (slide group units) and an instruction requesting that these multiple second-granular summaries be combined to create a summary (text data) of a third granularity (file unit, i.e., document unit such as presentation materials). The LLM response data in the LLM usage scenario (3C) is a summary (text data) of the third granularity, specifically a summary of the information in one file (for example, one presentation material).

[0073] In this embodiment, as an example, summaries of three levels of granularity are created, but two levels of granularity or four or more levels of granularity may also be used. More generally, when creating summaries of the second and subsequent levels of granularity (second, third, fourth, ...), N is set to an integer of 2 or more, and a request data for creating the Nth summary is created, which includes multiple summaries of the (N-1)th level of granularity and an instruction to create a summary of the Nth level of granularity using these multiple summaries of the (N-1)th level of granularity. This request data for creating the Nth summary is then transmitted via network 1 to a service provision system 50 using a large-scale language model (LLM), and LLM response data (summaries of the Nth level of granularity) is received from the service provision system 50 via network 1. This process is repeated until N becomes the number of the largest level of granularity.

[0074] <Configuration of Document Information Search Server 20 / Search Management Means 27 / Database Selection Means 27A>

[0075] The database selection means 27A receives user question data transmitted from the user terminal 70 via the network 1, creates database selection request data as a prompt as shown in LLM usage scenario (4) in Figure 2, transmits the created database selection request data to the service provision system 50 using a large-scale language model (LLM) via the network 1, and performs the process of receiving LLM response data transmitted from the service provision system 50 via the network 1. Here, as shown in Figure 11, the database selection request data in LLM usage scenario (4) includes user question data received from the user terminal 70, a list of multiple searchable summary databases 41, 42, and 43, and instructions asking the user to select which summary database to use to answer the user's question and to create a search query. The LLM response data in LLM usage scenario (4) is a search query that includes the summary database to be used and the question data for searching each summary database. Note that the summary database to be used may be one summary database, multiple summary databases, or all of the summary databases 41, 42, and 43.

[0076] Furthermore, if multiple summary databases are to be used, search processing will be performed for each of those summary databases. Therefore, the Large-Scale Language Model (LLM) creates search queries for each summary database, and the question data included in these search queries will differ for each summary database (see Figure 11). Consequently, the question data included in the search queries is not the same as the question data entered by the user. In other words, the Large-Scale Language Model (LLM) creates multiple search queries, each containing question data for each summary database, so that appropriate search processing is performed for each of the selected summary databases. Note that the question data included in the response feasibility determination request data input from the response generation feasibility determination means 27C to the Large-Scale Language Model (LLM) in subsequent processing, and the response creation request data input from the response generation means 28 to the Large-Scale Language Model (LLM), is the same as the question data entered by the user.

[0077] <Configuration of Document Information Search Server 20 / Search Management Means 27 / Search Means 27B>

[0078] The search means 27B transmits the search query (including the question data) and the number of summaries to be retrieved received by the database selection means 27A to the database search system 40, and executes a process to receive summaries (text data) obtained as a result of the search process performed using the summary database to be used (specified in the search query) from the database search system 40. If there are multiple summary databases to be used, the summary search process in the database search system 40 is executed for each summary database, and the number of summaries to be retrieved, which is related to the system setting or user specification, indicates the number of summaries to be retrieved from each summary database.

[0079] <Configuration of Document Information Search Server 20 / Search Management Means 27 / Response Generation Feasibility Determination Means 27C>

[0080] The response generation feasibility determination means 27C receives the summary obtained from the search means 27B, creates response feasibility determination request data as a prompt as shown in LLM usage scenario (5) in Figure 2, transmits the created response feasibility determination request data to the service provision system 50 using a large-scale language model (LLM) via the network 1, and executes the process of receiving LLM response data transmitted from the service provision system 50 via the network 1. Here, as shown in Figure 12, the response feasibility determination request data in LLM usage scenario (5) includes the summary received by the search means 27B (a summary of the number of summaries to be obtained per summary database, based on system settings or user specifications), the user's question data, and an instruction statement requesting that the system determine whether or not it can answer the user's question based on this information. The LLM response data in LLM usage scenario (5) is response feasibility determination information (information on whether or not a response to the user's question can be created). Then, if the response generation feasibility determination means 27C receives information from the service provision system 50 using a large-scale language model (LLM) that a response cannot be provided, it executes a process to return to the processing of the database selection means 27A. At this time, the response generation feasibility determination means 27C sends the failure information (information indicating the circumstances under which a response could not be provided, i.e., information on which summary database was used for the search) to the database selection means 27A.

[0081] <Configuration of Document Information Search Server 20 / Answer Generation Means 28>

[0082] The response generation means 28, when the response feasibility determination information received by the response feasibility determination means 27C indicates that a response can be provided, creates response creation request data as a prompt, as shown in LLM usage scenario (6) in Figure 2, transmits the created response creation request data to the service provision system 50 using a large-scale language model (LLM) via the network 1, and executes the process of receiving LLM response data transmitted from the service provision system 50 via the network 1. Here, as shown in Figure 13, the response creation request data in LLM usage scenario (6) includes a summary received by the search means 27B (a summary of the number of summaries to be obtained per summary database, based on system settings or user specifications), the user's question data, and an instruction requesting that a response be created to the user's question. The LLM response data in LLM usage scenario (6) is the response data to the user's question. The response generation means 28 also executes the process of transmitting the response data (data for displaying the response data) received from the service provision system 50 using a large-scale language model (LLM) to the user terminal 70 via the network 1.

[0083] <Configuration of Document Information Retrieval Server 20 / Document Storage Means 31>

[0084] The document storage means 31 stores the documents to be summarized. Therefore, the documents to be summarized are selected by the summary creator from among the documents stored in the document storage means 31.

[0085] <Flowchart of the construction process for summary databases 41, 42, and 43, and the structure of summary databases 41, 42, and 43: Figure 3>

[0086] In Figure 3, a slide-format document (file) to be summarized is selected, and summaries of the first granularity (slide unit) are sequentially created for any one slide (slide number = i) that makes up that slide-format document, in the order i = 0, 1, 2, 3, ... The summaries are created in this order because, in the process of step S5 below, when creating the summary of slide number = i, the summary of the previous slide (slide number = i-1) is used.

[0087] First, the document analysis means 22 performs an analysis of the information of the slide to be processed (sectional document information obtained by dividing the entire document into a first granularity) (step S1), and extracts information about shapes, that is, information about shapes (shapes in which text can be placed). The shape information includes the type of shape, the text data within the shape, and the position (coordinates) of the shape. This process has already been described in detail in the description of the configuration of the document analysis means 22, so a detailed explanation is omitted here.

[0088] Next, the shape intent acquisition means 23 uses a large-scale language model (LLM) and the extracted shape information to acquire the intent (text data) derived from the arrangement of the shapes (step S2). This process is the same as the LLM usage scenario (1) shown in Figure 2, and has already been described in detail in the explanation of the configuration of the shape intent acquisition means 23, so a detailed explanation is omitted here.

[0089] Next, the imaging means 24 performs the process of converting the slide to be processed into an image (step S3).

[0090] Next, the multimodal information processing means 25 uses a large-scale language model (LLM) to obtain information from the appearance of the images using the slide image data (step S4). This process is the same as the LLM usage scenario (2) shown in Figure 2, and has already been described in detail in the explanation of the configuration of the multimodal information processing means 25, so a detailed explanation is omitted here.

[0091] Then, the summarization means 26 uses a large-scale language model (LLM) to create a summary (first-level summary) for slide number i, using text data that captures the intent behind the placement of shapes (figures containing text), text data as appearance information obtained from image data, and a previously created summary (text data) for the previous slide (slide number i-1) (step S5). This process is the LLM usage scenario (3A) shown in Figure 2, and has already been described in detail in the explanation of the configuration of the summarization means 26, so a detailed explanation is omitted here.

[0092] Furthermore, in step S4 described above, as shown by the dashed line in Figure 3, the multimodal information processing means 25 may use a large-scale language model (LLM) to obtain integrated information using shape information and slide image data. In this case, in step S5 described above, the summarization means 26 executes a process to create a summary for slide number i (a first-level summary) using the integrated information and the previously created summary for the previous slide (slide number i-1). If there is text data outside the shape as segmented document information obtained by segmenting at the finest first level of granularity, that text data is also added to create the summary for slide number i (a first-level summary).

[0093] When the processes described in steps S1 to S5 above are performed sequentially for slide numbers 0, 1, 2, 3, ..., summaries (first-level summaries) are created for each of the multiple slides. The summary creation means 26 then stores these first-level summaries in the first (slide-based) summary database 41. The summary creation means 26 also creates vector data for the first-level summaries and stores it in the summary database 41.

[0094] As shown in Figure 3, the summary database 41 stores the file name, slide number, first-level summary (text data), raw content (shape information used to create the first-level summary, and appearance information obtained from image data), and vector data of the first-level summary in association with each other.

[0095] Subsequently, the summarization means 26 uses a large-scale language model (LLM) to create a summary of the second granularity (slide group unit) using multiple summaries of the first granularity (slide unit) (step S6). This process is the LLM usage scenario (3B) shown in Figure 2, and has already been described in detail in the explanation of the configuration of the summarization means 26, so a detailed explanation is omitted here. In the example in Figure 3, a summary of one slide group (second granularity) is created from the summaries of three slides (first granularity) with slide numbers = 0, 1, and 2, a summary of one slide group (second granularity) is created from the summaries of three slides (first granularity) with slide numbers = 3, 4, and 5, and a summary of one slide group (second granularity) is created from the summaries of three slides (first granularity) with slide numbers = 6, 7, and 8. Note that the number of slides constituting one slide group is not limited to three, but can be any number. Then, the summary creation means 26 stores a summary (second-level summary) for each of the created slide groups in the second (slide group unit) summary database 42. The summary creation means 26 also creates vector data for the second-level summaries and stores it in the summary database 42.

[0096] As shown in Figure 3, the summary database 42 stores the file name, a set of slide numbers, a second-level summary (text data), raw content (multiple first-level summaries used to create the second-level summary), and vector data of the second-level summary, associating them with each other.

[0097] Furthermore, the summary creation means 26 uses a large-scale language model (LLM) to create a summary of a third level of granularity (file unit) using multiple summaries of a second level of granularity (slide group unit) (step S7). This process is the LLM usage scenario (3C) shown in Figure 2, and has already been described in detail in the explanation of the configuration of the summary creation means 26, so a detailed explanation is omitted here. The summary creation means 26 then stores the summary (third-level granularity summary) for the created file (for example, the entire document such as presentation materials) in the third (file unit) summary database 43. The summary creation means 26 also creates vector data of the third-level granularity summary and stores it in the summary database 43.

[0098] As shown in Figure 3, the summary database 43 stores the file name, a set of slide numbers (all slide numbers), a third-level summary (text data), raw content (multiple second-level summaries used to create the third-level summary), and vector data of the third-level summary, associating them with each other.

[0099] <Document information retrieval process flow: Figures 4 and 5>

[0100] In Figure 4, the summary creator first operates the creator terminal 60 to perform pre-processing for document information retrieval (step S21). Specifically, this involves registering the file of the document to be summarized (such as a slide document) in the document storage means 31, creating multiple summaries (text data) of different granularities (three in this embodiment as an example) from the same file (material) stored in the document storage means 31, vectorizing each summary, and registering the summaries of each granularity and their vector data in the summary databases 41, 42, and 43. This pre-processing has already been described in detail using Figure 3, so a detailed explanation is omitted here.

[0101] Next, the database selection means 27A receives user question data transmitted from the user terminal 70 via the network 1 (step S22).

[0102] Next, the LLM usage scenario (4) shown in Figure 2 is executed. Specifically, the database selection means 27A creates database selection request data (including question data, a list of searchable summary databases, and instructions to select which summary database to use to answer the user's question and to create a search query), sends the created database selection request data to the LLM service provision system 50, and receives LLM response data (the summary database to be used, and a search query including the question data) from the service provision system 50 (step S23). This process has already been described in detail in the explanation of the configuration of the database selection means 27A, so a detailed explanation is omitted here.

[0103] Then, the search means 27B searches each of the summary databases to be used (here, for example, two second and third summary databases 42 and 43) received from the LLM service provision system 50, and sends the received search query (including the query data for each summary database) and the number of summaries to be retrieved (here, for example, three) to the database search system 40 (step S24). This process has already been described in detail in the description of the configuration of the search means 27B, so a detailed explanation is omitted here.

[0104] Furthermore, the database search system 40 is provided with separate summary databases 41, 42, and 43 for each level of granularity (see Figure 3), and a separate database management system (DBMS) is also provided for each summary database 41, 42, and 43. The search process for summaries at each level of granularity using each summary database 41, 42, and 43 is executed independently.

[0105] Furthermore, when the database search system 40 uses vector similarity to retrieve a corresponding summary from the question data (search question data for each summary database) included in a search query using the summary databases 41, 42, and 43 of each granularity, it calculates the similarity (e.g., cosine similarity) between the vector of the search question data in the target summary database and the vector of each summary stored in that target summary database. In this case, since the summary databases 41, 42, and 43 already have vectors of summaries of each granularity calculated and stored (see Figure 3), the question data included in the search query must also be vectorized in the same way, and the number of dimensions of the vectors must match. Note that the method by which the database search system 40 retrieves a corresponding summary from the question data included in a search query is not necessarily limited to this method of using vector similarity; in short, the database search system 40 only needs to be able to retrieve a summary corresponding to the question data included in the search query using the pre-prepared summary databases 41, 42, and 43.

[0106] Subsequently, summaries obtained from the target summary databases are received (step S25). Here, for example, two summary databases 42 and 43 are designated as the summary databases to be used and are used for the search process. If the number of summaries to be obtained according to the system settings or user specifications is, for example, three, then three summaries of the second granularity (slide group unit) obtained from summary database 42 are received, as well as three summaries of the third granularity (file unit) obtained from summary database 43.

[0107] Next, the LLM usage scenario (5) shown in Figure 2 is executed. Specifically, the response generation feasibility determination means 27C creates response feasibility determination request data (including the user's question data, the number of summaries to be obtained, the obtained summaries (here, for example, 3 x 2 = 6 summaries), and an instruction that the system should determine whether or not it is possible to answer the user's question using this information), transmits the created response feasibility determination request data to the LLM service provision system 50, and receives LLM response data (response feasibility determination information indicating whether or not a response to the user can be created) from the service provision system 50 (step S26).

[0108] If the received information indicating that a response cannot be provided indicates that a response cannot be provided, the process returns to step S23 by the database selection means 27A (step S27). At this time, the response generation feasibility determination means 27C sends the failed information (information indicating the circumstances under which a response cannot be provided, i.e., information on which summary database was used for the search) to the database selection means 27A.

[0109] On the other hand, if the received information regarding the feasibility of answering in step S27 of Figure 4 indicates that an answer can be provided, the LLM usage scenario (6) is executed as shown in Figure 5. Specifically, the answer generation means 28 creates an answer creation request data (including the user's question data, the acquired summaries (for example, 3 x 2 = 6 summaries), and an instruction requesting that an answer be created to the user's question), transmits the created answer creation request data to the LLM service provision system 50, and receives LLM answer data (an answer to the user's question) from the service provision system 50 (step S28).

[0110] Then, the answer generation means 28 transmits the answer to the user's question (data for displaying the answer) to the user terminal 70 via the network 1 (step S29).

[0111] <Effects of this embodiment>

[0112] This embodiment provides the following advantages. Specifically, the document information retrieval system 10 provides multiple summary databases 41, 42, and 43 of varying granularity, and the database selection means 27A selects the appropriate summary database according to the content of the user's question. The search means 27B obtains a summary of the appropriate granularity using the selected summary database, and the answer generation means 28 creates an answer to the user's question using the obtained summary.

[0113] Therefore, when creating answers to user questions, summaries at multiple levels of granularity are used. For example, fine-grained summaries are used for detailed questions, and coarse-grained summaries are used for broader questions. Consequently, diverse perspectives and interpretations can be provided for the same information source, deepening the user's understanding.

[0114] Furthermore, because the most appropriate format of information can be selected according to the user's needs and circumstances, it is possible to provide information that meets the user's objectives.

[0115] Furthermore, providing diverse information using summaries at multiple levels of granularity can attract user interest, increase their attachment and loyalty to the system itself, as well as to the company, work, and other organizations providing the system (so-called engagement), and ultimately improve organizational strength.

[0116] Furthermore, since the document information retrieval system 10 is equipped with a means 27C for determining whether or not a response can be generated, the process by the database selection means 27A can be repeated until the selection process of the summary database to be used by the database selection means 27A reaches an appropriate state. As a result, it is possible to provide information from an even wider range of perspectives and in an appropriate format.

[0117] Furthermore, since the document information retrieval system 10 is equipped with a summarization means 26, it can utilize a large-scale language model (LLM) to sequentially create summaries starting from fine-grained to coarser-grained. Therefore, manual work can be avoided or reduced when preparing multiple summary databases 41, 42, and 43 with varying levels of granularity.

[0118] Furthermore, since the document information retrieval system 10 is equipped with a graphic intent acquisition means 23 and a multimodal information processing means 25, the summarization means 26 can create a first-level summary using the intent obtained from the arrangement of graphics and the appearance information obtained from image analysis. Therefore, it is possible to accurately grasp the information contained in a document in a format that can include text, graphics, and images, and create a first-level summary.

[0119] Furthermore, the summarization means 26 can create a first-level summary of the information on slide (slide number = i) by using the intent derived from the arrangement of the figures, the appearance information obtained from image analysis, and the previously created summary of the information on the preceding slide (slide number = i-1). As a result, contextual information is added, making it possible to grasp the information contained in a document in a format that can include text, figures, and images more accurately and create a first-level summary.

[0120] Furthermore, the multimodal information processing means 25 can analyze the graphic information (shape information) together with the image data to obtain integrated information. Therefore, the summarization means 26 can use the integrated information and the previously created summary of the information on the previous slide (slide number = i-1) to create a first-level summary of the information on the slide (slide number = i). Consequently, in this case as well, contextual information is added, making it possible to grasp the information contained in a document in a format that can include text, graphics, and images more accurately and create a first-level summary.

[0121] <Form of transformation>

[0122] It should be noted that the present invention is not limited to the embodiments described above, and any modifications that can achieve the objectives of the present invention are included within the scope of the present invention.

[0123] For example, in the above embodiment, the summaries of multiple granularities were defined as summaries at the slide level, slide group level, and file level (document level), but are not limited to these. For example, they could be summaries for each section of a document divided into chapters and sections, and also file-level summaries, or summaries for each section of a document divided into sections based on different character counts. [Industrial applicability]

[0124] As described above, the document information retrieval system and program of the present invention are suitable for use, for example, when retrieving information contained in a slide-format document. [Explanation of Symbols]

[0125] 1 Network 10. Document Information Retrieval System 22 Document Analysis Methods 23. Means for acquiring the intention of a graphic 24. Image acquisition means 25 Multimodal information processing means 26. Means of creating summaries 27A Database Selection Method 27B Search Method 27C Means for determining whether answer generation is possible 28 Answer generation means 40 Database Search System 50. Service delivery systems using Large-Scale Language Models (LLMs) 60 Creator's terminal 70 User terminals

Claims

1. A document information retrieval system comprising a computer that performs the process of searching for information contained in a document, Multiple summary databases that store text data of summaries for each granularity unit, created using segmented document information obtained by dividing the information contained in the same document into multiple different granularities, A database selection means that receives user question data from a user terminal, creates a database selection request data that includes the received question data, a list of multiple searchable summary databases, and instructions to select which summary database to use to answer the user's question, inputs the created database selection request data into a large language model, and receives a search query that includes the summary database to be used and search question data for each summary database output from the large language model. A search means that transmits the search query received by the database selection means to a database search system, and receives a summary from the database search system obtained as a result of a search process performed using the summary database to be used. This search means creates response creation request data including the summary received, the user's question data, and instructions requesting that answers be created to the user's questions; inputs the created response creation request data into the large-scale language model; receives the answer data to the user's questions output from the large-scale language model; and transmits the received answer data to the user terminal. A document information retrieval system characterized by having the following features.

2. The answer generation feasibility determination means creates an answer feasibility determination data that includes the summary received by the search means, the user's question data, and an instruction that it should be determined whether or not the user's question can be answered using this information; inputs the created answer feasibility determination data into the large-scale language model; receives answer feasibility determination information output from the large-scale language model; and if the received answer feasibility determination information indicates that the answer cannot be answered, executes a process to return to the processing of the database selection means. The document information retrieval system according to feature 1.

3. A first summary creation request data is created, which includes the segmented document information obtained by segmenting the information contained in the document at the finest first granularity, and an instruction to create a summary of the segmented document information; the created first summary creation request data is input into the large-scale language model; and the first granularity summary output from the large-scale language model is received. Furthermore, setting N to an integer of 2 or more, a request data for creating an Nth summary is created, which includes multiple summaries of the (N-1)th granularity and an instruction to create a summary of the Nth granularity using these multiple summaries of the (N-1)th granularity; the created Nth summary creation request data is input into the large-scale language model; the Nth granularity summary output from the large-scale language model is received; and this process is repeated until N becomes the number of the largest granularity. The system includes a summary creation means that performs a process of storing the summaries created for each granularity in the summary database of the corresponding granularity. The document information retrieval system according to feature 1.

4. A means for acquiring graphic intent, which creates graphic intent acquisition request data including information on graphic shapes containing characters extracted from the divided document information obtained by dividing it at the first granularity, and an instruction sentence requesting that the intention derived from the arrangement of the graphic be understood and explained, inputs the created graphic intent acquisition request data into the large-scale language model, and receives the intention output from the large-scale language model. The system comprises a multimodal information processing means that creates image data obtained by imaging the segmented document information obtained by segmenting it at the first granularity, and an appearance information acquisition request data including an instruction request to create appearance information obtained by analyzing the image data, inputs the created appearance information acquisition request data into the large-scale language model, and receives the appearance information output from the large-scale language model, The aforementioned summary creation means is The system is configured to create the first summary creation request data, which includes the intention acquired by the graphic intention acquisition means, the appearance information acquired by the multimodal information processing means, and an instruction that the content of the sectioned document information should be interpreted using this information to create the summary. The document information retrieval system according to feature 3.

5. The aforementioned summary creation means is The system is configured to create the first summary creation request data, which includes the intention obtained by the graphic intention acquisition means, the appearance information obtained by the multimodal information processing means, the previously created summary for the sectioned document information, and an instruction that the content of the sectioned document information should be interpreted using this information to create the summary. The document information retrieval system according to feature 4.

6. The system includes a multimodal information processing means that creates integrated information acquisition request data including information of a figure containing characters extracted from the segmented document information obtained by segmenting at the first granularity, image data obtained by imaging the segmented document information obtained by segmenting at the first granularity, and an instruction statement requesting the creation of integrated information obtained by analyzing the figure information and the image data; inputs the created integrated information acquisition request data into the large-scale language model; and receives the integrated information output from the large-scale language model. The aforementioned summary creation means is The system is configured to create a first summary creation request data which includes the integrated information acquired by the multimodal information processing means, the previously created summary for the sectioned document information, and an instruction that the content of the sectioned document information should be interpreted using this information to create the summary. The document information retrieval system according to feature 3.

7. A program for causing a computer to function as a document information retrieval system according to any one of claims 1 to 6.

Images