Computer program, information processing device and information processing method

Abstract

To provide a computer program, an information processing device and an information processing method related to a search for document data.SOLUTION: A computer program causes a computer to execute the processing to: acquire feature quantities of a plurality of pieces of document data stored in a storage and including photographs, figures, tables, graphs or texts, respectively; store the feature quantities in association with identification data for identifying the document data in a storage unit; receive an inquiry to the storage; and output an answer to the inquiry by giving a language model the received inquiry and document data extracted based upon the feature quantity.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a computer program, an information processing apparatus, and an information processing method relating to document data search. [Background technology]

[0002] Organizations such as companies and groups create and store various types of document data every day. This document data, excluding confidential information within the organization, is the organization's shared intellectual property, and by effectively utilizing it, it can lead to more efficient handovers and work efficiency based on tacit knowledge such as prior knowledge and know-how within the organization.

[0003] When document data becomes huge, it becomes difficult to find the data you need. Therefore, a search system has been proposed that stores document data in a database and extracts characteristic words from the stored document data to enable searching (Patent Document 1, etc.). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-174988 Summary of the Invention [Problem to be solved by the invention]

[0005] Document data that can be utilized as shared intellectual property within an organization is not limited to structured data such as spreadsheet-formatted data with defined attributes and item names. Document data can also include documents written in natural language, document data for presentations in which images, charts, and text are freely arranged, and image data. Document data and image data for presentations may not contain the content expressed using the document as text. In such cases, searching by text is not possible, and it is difficult to extract attributes and items as disclosed in Patent Document 1.

[0006] An object of the present disclosure is to provide a computer program, an information processing device, and an information processing method related to document data search. [Means for solving the problem]

[0007] A computer program according to an embodiment of the present disclosure causes a computer to acquire features of a plurality of document data stored in a storage unit, each of the document data including photographs, diagrams, tables, graphs, or text, associate the features with identification data that identifies the document data, store the features in a memory unit, accept a query to the storage unit, and output a response to the query by providing the accepted query and document data extracted based on the features to a language model. [Effects of the Invention]

[0008] According to the present disclosure, document data including unstructured data can be searched, and necessary data can be efficiently referenced from data shared by an organization or the like. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram of a search system. [Figure 2]FIG. 1 is a block diagram showing a configuration of an information processing device. [Figure 3] FIG. 1 is a schematic diagram of a model used in an information processing device. [Figure 4] FIG. 2 is a block diagram showing the configuration of a terminal device. [Figure 5] 10 is a flowchart illustrating an example of a procedure of pre-processing by an information processing device. [Figure 6] FIG. 10 is an explanatory diagram of a search table. [Figure 7] 10 is a flowchart illustrating an example of a procedure for accepting a search. [Figure 8] FIG. 10 is a diagram illustrating an example of a web page displayed on a terminal device. [Figure 9] FIG. 10 is a diagram illustrating an example of a web page displayed on a terminal device. [Figure 10] FIG. 10 is a diagram illustrating an example of a web page displayed on a terminal device. [Figure 11] FIG. 10 is a diagram illustrating an example of a web page displayed on a terminal device. [Figure 12] FIG. 10 is a diagram illustrating an example of a web page displayed on a terminal device. [Figure 13] FIG. 10 is a diagram illustrating an example of a web page displayed on a terminal device. [Figure 14] FIG. 10 is a diagram illustrating an example of a web page displayed on a terminal device. [Figure 15] 10 is a flowchart illustrating an example of a procedure for pre-processing by an information processing apparatus according to a second embodiment. [Figure 16] FIG. 10 is an explanatory diagram of a search table in the second embodiment. [Figure 17] 10 is a flowchart illustrating an example of a procedure for accepting a search in the second embodiment. [Figure 18] FIG. 10 is an explanatory diagram of a process in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] The present disclosure will be specifically described with reference to the drawings illustrating embodiments thereof. In the following embodiments, a search system including an information processing device that executes processing based on a computer program of the present disclosure will be described.

[0011] (First embodiment) FIG. 1 is a schematic diagram of a search system 100. The search system 100 is a system that enables the use of natural language to search for document data shared within an organization such as a company or an association. Searchable document data is not limited to structured data described with common attribute items and corresponding specific data, but also includes image data and unstructured data consisting of text and images. Examples of unstructured data include data used in presentations, PDF data used for distribution, and image data itself that may include video. The search system 100 disclosed herein also enables easy search of such unstructured data as intellectual property (knowledge) shared within an organization.

[0012] Document data is data used within an organization for internal and external presentations, or documents for sharing information between users within an organization. For an organization in the travel industry, document data may be design data for a travel brochure or web page data. For an organization in the fashion industry, document data may be design data for a product brochure or web page data.

[0013] The search system 100 includes an information processing device 1 connected to a local network LN within an organization, and a terminal device 2 that can communicate with the information processing device 1. The information processing device 1 functions as a server within the local network LN, and the terminal device 2 communicates with the information processing device 1 using a web browser program and can access each piece of data that can be shared via the local network LN.

[0014] The local network LN is a wired and / or wireless communication medium that enables communication connections and data transmission and reception from terminal devices 2 that are authenticated within the organization. The local network LN is connected to a public communication network, network N. The local network LN may include a VPN (Virtual Private Network) over the public communication network, network N, in which case it can accept communication connections from terminal devices 2 that are physically separated from other communication devices owned by the organization.

[0015] The search system 100 of the present disclosure stores document data in a storage 110 that is readable and writable by the information processing device 1. Each piece of document data is assigned a file path within the storage 110 and saved.

[0016] The information processing device 1 of the present disclosure uses a language model LM (Language Model) that has been trained to return appropriate response sentences in natural language to inquiries in natural language. The language model LM may be a large language model. The language model LM is a trained model that converts given natural language sentences into vector data and outputs response sentences in natural language having appropriate features for sentences having features represented by the vector data. The information processing device 1 crawls document data stored in the storage 110 and stores features (vector data) for the entire document and for each separable part such as a page or chapter in a search table 111. The features are acquired using the language model LM or a vectorization module corresponding to the language model LM. The information processing device 1 accepts inquiries (search requests) in natural language regarding document data in the storage 110 from a terminal device 2. The information processing device 1 extracts target document data or a part thereof that is similar to the natural sentence of the received inquiry using the search table 111, and provides the extracted document data or a part thereof and a sentence explaining that part to the language model LM to create an answer sentence and respond.

[0017] With the above-mentioned functions, by constructing the information processing device 1 as a server computer that can access each organization's storage 110, it becomes possible to search for intellectual property within each organization, including unstructured data, in natural language (conversational style) and present the results together with the rationale.

[0018] The following describes in detail the configuration and processing for realizing information processing that enables the information processing device 1 to search for various document data including unstructured data.

[0019] 2 is a block diagram showing the configuration of the information processing device 1. The information processing device 1 includes a processing unit 10, a storage unit 11, and a communication unit 12. The information processing device 1 is a server computer. In the following description, the information processing device 1 is described as being configured by one server computer, but it may also be configured such that a plurality of server computers are communicatively connected via a network N to perform distributed processing and function as one information processing device 1. The information processing device 1 may function as an on-premise server in the network N within an organization from a terminal device 2, or as a cloud server communicatively connected via the network N including a public network.

[0020] The processing unit 10 includes one or more processors such as a central processing unit (CPU), a micro-processing unit (MPU), a graphics processing unit (GPU), etc. The processing unit 10 also includes a memory that is a temporary storage medium such as a static random access memory (SRAM), a dynamic random access memory (DRAM), etc. The processing unit 10 may be configured as a single piece of hardware (SoC: System On a Chip) that integrates a processor, a memory, a storage unit 11, and a communication unit 12. The processing unit 10 reads an information processing program P1 stored in the storage unit 11 into the memory and executes it, thereby causing a general-purpose computer to perform various processes described below and function as the information processing device 1 of the present disclosure.

[0021] The storage unit 11 is a relatively large-capacity non-transitory storage medium such as a hard disk or flash memory. The storage unit 11 stores a program (program product) required for the processing unit 10 to execute processing, and reference setting data. The program product includes at least a Web server program P0, images and their designs (layout) for displaying a Web screen, and an information processing program P1. The storage unit 11 stores a model M1. Part or all of the model M1 may be stored in an external server.

[0022] The information processing program P1 and the model M1 stored in the storage unit 11 may be the information processing program P9 and the model M9 stored in a computer-readable non-transitory storage medium 9 that are read by the processing unit 10 and stored in the storage unit 11. The information processing program P1 and the model M1 may be the information processing program P9 and the model M9 that are downloaded by the processing unit 10 from a download server via the communication unit 12 and stored in the storage unit 11.

[0023] The memory unit 11 of the information processing device 1 functions as a storage 110 in which target document data of the search system 100 is stored. A partial area of ​​the memory unit 11 that is set to be sharable may function as the storage 110. Each memory unit 11 is a part of the storage 110, and may function as the storage 110 by storing data in a distributed or partially overlapping manner in the memory units 11 of multiple server computers that are communicatively connected.

[0024] The communication unit 12 realizes communication via the local network LN and the network N. The communication unit 12 may be a wired network card or a wireless communication device for WiFi. The communication unit 12 may also be a wireless communication device that connects to a carrier network. The processing unit 10 can send and receive data to and from the terminal device 2 via the communication unit 12.

[0025] 3 is a schematic diagram of a model M1 used in the information processing device 1. The model M1 includes a language model LM that has been trained to return an appropriate response sentence in natural language in response to an input query sentence (a sentence expressed in natural language and input for search, also called a prompt). The language model LM may be a large-scale language model.

[0026] In addition to the language model LM, model M1 includes an image recognition model IRM that uses a Vision Transformer or similar to output the features of input image data as text. The image recognition model IRM divides the input image data using filtering or other methods, converts each divided image into vector data corresponding to what the image represents, and performs processes such as updating the weights for words that should be noted from the vector data. The image recognition model IRM can be called independently from model M1 and used.

[0027] The model M1 includes an Embedding module EM that performs computations on the input prompt, such as tokenizing and embedding the input into vector data before providing it to the language model LM. The Embedding module EM outputs the features of the given natural sentence. This Embedding module EM can be called independently from the model M1 and used.

[0028] When a prompt is input, the model M1 functions to convert the text constituting the prompt into vector data using the embedding module EM and input the vector data to the language model LM. When an image is input together with the prompt, the model M1 is configured to concatenate the vector data output from the image recognition model IRM with the vectorized data of the prompt (text) and then input the vector data to the language model LM. The model M1 outputs the text output from the language model LM. The model M1 may be used to add original text, as described below, to a query sentence input by a user, or to provide an instruction at the beginning of the time series to ensure that the text to be output from the language model LM is appropriate for the functions of the search system 100, so that it is reflected in responses to subsequent queries.

[0029] The language model LM, the image recognition model IRM, and the embedding module EM that constitute the model M1 may be partly or entirely provided by an external server connected via the network N. The model M1 may employ a multimodal model such as LLaVA (Large Language-and Vision Assistant) and assume that image data is input. The model M1 may employ a model with another architecture.

[0030] The model M1 may use a natural language model trained using training data to output summaries of document data, instead of using an existing language model LM or a large-scale language model (LLM) provided from an external server. In this case, the training data may be textualized descriptions of document data and outline diagrams created by staff within the organization to which the search system 100 is applied.

[0031] 4 is a block diagram showing the configuration of the terminal device 2. The terminal device 2 is a personal computer. The terminal device 2 may be a smartphone or a tablet terminal. The terminal device 2 includes a processing unit 20, a storage unit 21, a communication unit 22, a display unit 23, an operation unit 24, and an audio input / output unit 25.

[0032] The processing unit 20 includes one or more processors such as a CPU, an MPU, a GPU, etc. The processing unit 20 includes a memory that is a temporary storage medium such as an SRAM or a DRAM. The processing unit 20 may be configured as a single piece of hardware (SoC: System On a Chip) that integrates the processor, the memory, and further the storage unit 21 and the communication unit 22.

[0033] The storage unit 21 is a relatively large-capacity non-volatile storage area such as an SSD or flash memory. The storage unit 21 stores programs (program products) required for the processing unit 20 to execute processing, and reference setting data. The program products include a web browser program P2 or a terminal program.

[0034] The communication unit 22 realizes communication with the information processing device 1 via the local network LN or communication with an external server via the network N. The communication unit 22 may be a wired network card or a wireless communication device for WiFi. The communication unit 12 may be a wireless communication device that connects to a carrier network. The processing unit 20 can send and receive data to and from the information processing device 1 via the communication unit 22.

[0035] The display unit 23 is a display such as a liquid crystal display or an organic EL (Electro Luminescence) display. The display unit 23 may be a display with a built-in touch panel. The processing unit 20 can display a Web screen obtained by the Web server function of the information processing device 1 on the display unit 23, and can display characters and images entered by the user.

[0036] The operation unit 24 is a user interface capable of inputting and outputting data to and from the processing unit 20, and includes, for example, a keyboard and a pointing device such as a mouse. The operation unit 24 may also include physical buttons, switches, and physical dials, or may be a touch panel built into the display unit 23.

[0037] The audio input / output unit 25 includes a speaker and a microphone. The processing unit 20 or the audio input / output unit 25 itself may have a voice recognition function that converts audio collected using the microphone into text data. The processing unit 20 can output sound effects and audio using the speaker.

[0038] The process executed by the search system 100 configured as described above will be described. In the search system 100, document data created or acquired by each terminal device 2 is stored in the storage 110 of the information processing device 1. The document data includes image data, video data, presentation data, text-based data in document editing software format, and spreadsheet software format data. The document data may be a mixture of images and text, spreadsheet software format data with image data or text data superimposed on it, or data containing diagrams and tables. The document data may be uploaded from the terminal device 2 using the web server function of the information processing device 1 and stored in the storage 110 by the information processing device 1, or may be stored in a shareable area of ​​the memory unit 11 of the information processing device 1 using a file system. The processing unit 10 of the information processing device 1 performs "pre-processing" on the data stored in the storage 110, making it possible for the terminal device 2 to search for the document data.

[0039] 5 is a flowchart showing an example of a procedure of pre-processing by the information processing device 1. The information processing device 1 crawls the storage 110 and executes the following processing. Whenever it is detected that document data has been added to the storage 110, the information processing device 1 may execute the following processing.

[0040] The processing unit 10 acquires document data that has been added to the storage 110 (step S101). In step S101, the processing unit 10 acquires unprocessed document data in the storage 110 by crawling.

[0041] The processing unit 10 identifies attribute data for the acquired document data (step S102). In step S102, the processing unit 10 acquires and identifies attribute data such as the name (file path), type, number of pages, creation date and time, and update date and time from the document data. The processing unit 10 stores the search table 111 storing the identified attribute data in association with data (e.g., a document ID) that identifies the document data acquired in step S101 in the storage unit 11 (step S103).

[0042] The processing unit 10 reads a portion of the document data acquired in step S101 in predetermined separable units (step S104). In step S104, the processing unit 10 may read data in page units of the document data, may read data according to the amount of tokens specified in the Embedding module of model M1, may read data separably by determining chapter or paragraph boundaries, or may read data by separating images and text portions.

[0043] The processing unit 10 determines whether each piece of data read in a predetermined separable unit contains a non-text portion such as a graph or an image (step S105). The processing of step S105 is not essential. If the processing unit 10 determines that the data contains a non-text portion (S105: YES), it extracts an image from the data read in the predetermined separable unit (e.g., one page) (step S106). In step S106, the processing unit 10 may convert a portion of the read document data (e.g., one page) directly into an image, or may convert each separable unit of data into an image, or may extract only the image portion. The processing unit 10 further extracts text from the portion of the read document data (step S107). In step S107, the processing unit 10 may extract the text data directly, or may read the text using an OCR function.

[0044] The processing unit 10 provides the extracted image and extracted text to the Embedding module EM of the model M1 to convert them into vector data (step S108). In step S108, the processing unit 10 inputs the text to the Embedding module EM of the model M1 and the extracted image to the image recognition model IRM, and acquires and concatenates the vector data output from each. The processing unit 10 may acquire the vector data of the image, the vector data of the text, and the vector data concatenating the image and the text. The processing unit 10 may add a phrase such as "Please explain the attached image" to the input text data (prompt), provide the image to the image recognition model IRM, and concatenate it with the vector data of the text.

[0045] In step S108, when converting each separable unit into vector data, the processing unit 10 may provide knowledge that the data is part of continuous data. The processing unit 10 may instruct the embedding module EM that the data is part of a single document, and then provide the data read out in separable units in a continuous manner to convert the data into vector data. This makes it possible to obtain vector data that takes advantage of the context surrounding the data.

[0046] The processing unit 10 associates the identification data of the document data acquired in step S101 with the data identifying each of the portions read in step S104, and adds and stores the vector data obtained in step S108 in the search table 111 (step S109). The processing unit 10 determines whether all of the separable units have been read from the document data (step S110). If it is determined that all of the separable units have been read (S110: YES), the processing unit 10 ends the processing. If it is determined that all of the data have not been read (S110: NO), the processing unit 10 returns the processing to step S104.

[0047] If it is determined in step S105 that the document data does not contain a non-text portion (S105: NO), text is extracted from a portion of the read document data (step S111), and the extracted text is provided to the Embedding module EM of the model M1 to be converted into vector data (step S112). The processing unit 10 associates the identification data of the document data acquired in step S101 with the data identifying the portion read in step S104, adds the vector data obtained in step S112 to the search table 111, and stores the data (step S113), and proceeds to step S110.

[0048] The processing unit 10 of the information processing device 1 executes the process shown in Fig. 5 on unprocessed document data that is not stored in the search table 111, among the document data stored in the storage 110. By the processing procedure shown in Fig. 5, the document data stored in the storage 110 can be searched for in natural language.

[0049] Fig. 6 is an explanatory diagram of the search table 111. The search table 111 stores the name of the document data, the type of document data, the number of pages, and the feature amount (vector) of the document data in association with the document ID. In the example of Fig. 6, the feature amount of the document data is stored on a page-by-page basis as a predetermined separable unit. If an image is included in the page, the image data is extracted separately, and the position within the page is identified and stored.

[0050] In the example of FIG. 6, for example, the name "*** / doc / file001.pdf," type "PDF," and number of pages "40" are stored as attribute data for document data with document ID "001." Furthermore, vector data obtained from the Embedding module is stored as the feature of the entire document data with document ID "001." A document ID "001_001" is assigned to page 1, and hierarchical information is represented by the attribute data name "*** / doc / file001.pdf_#1" and page number "1." It is determined that no image is extracted from the data of the first page with document ID "001_001," and the image data name field is filled with "-." It is also determined that an image is included on the second page of the same document data. The name automatically assigned to the extracted image and the position of the image on the page, "center," are associated with document ID "001_002." Image placement information may be represented by coordinates based on a reference point on the page.

[0051] 5, the search table 111 shown in Fig. 6 is stored in the memory unit 11 (or storage 110), thereby enabling searches using the following natural languages. The search table 111 shown in Fig. 6 may be stored in the memory unit 11 after being sorted into groups according to the storage area or type of the storage 110. Narrowing down of search targets may also be performed according to the group.

[0052] 7 is a flowchart showing an example of a search reception processing procedure. When a user uses a web browser program on terminal device 2 to access a web page for searching documents stored in storage 110 using a web server function provided by information processing device 1, the following processing begins.

[0053] The processing unit 20 of the terminal device 2 acquires a web page for search from the information processing device 1 (step S201). The processing unit 20 displays the acquired web page on the display unit 23 (step S202) and accepts a query statement written in natural language in an input field included in the web page (step S203). In step S203, the processing unit 20 may accept not only a query statement but also a data area divided in the storage 110 or a specification of document data (narrowing conditions).

[0054] The processing unit 20 transmits the query received in step S203 to the information processing device 1 via the local network LN (step S204).

[0055] The information processing device 1 receives a query statement via the communication unit 12 (step S121). The processing unit 10 provides the text of the received query statement to the Embedding module EM of the model M1 to convert it into vector data (step S122).

[0056] The processing unit 10 searches the search table 111 for vector data similar to the vectorized result (step S123), and identifies document data or parts thereof whose corresponding vector data have a similarity equal to or greater than a predetermined value (step S124). The processing unit 10 creates a list of document data including the identified parts (step S125), and transmits the created list together with thumbnail images of the identified parts to the terminal device 2 (step S126).

[0057] The processing unit 20 of the terminal device 2 receives a list (screen information) of a portion of the identified document data by a script in the search web page being displayed (step S205), and updates the list to display the identified portion and thumbnail images of that portion (step S206).

[0058] Meanwhile, the processing unit 10 of the information processing device 1 creates a prompt requesting an explanation of the identified portion along with the query received in step S121 (step S127). In step S127, the processing unit 10 may create a prompt requesting an explanation of all of the list of portions identified in steps S124 and S125. The processing unit 10 may also create a prompt requesting an explanation of each of the multiple portions of the identified document data. For example, in the former case, the processing unit 10 creates a prompt such as "Please respond to the following query by referring to all of the attached page data," and in the latter case, the processing unit 10 creates a prompt such as "Please respond to the following query by referring to the attached page."

[0059] The processing unit 10 provides the created prompt to the language model LM (step S128). The processing unit 10 acquires the text of the answer output from the language model LM (step S129) and transmits it to the terminal device 2 (step S130).

[0060] The processing unit 20 of the terminal device 2 receives the answer text (screen information) through a script in the currently displayed search web page (step S207), and updates the screen to display the received text (step S208).

[0061] The preliminary processing shown in Fig. 5 and the search reception processing shown in Fig. 7 will be described with reference to example screens displayed on the terminal device 2. Figs. 8 to 14 are diagrams showing examples of web pages displayed on the terminal device 2. Fig. 8 shows a state in which a web page provided by the information processing device 1 is displayed on a screen 230 based on a web browser program displayed on the display unit 23. As shown in Fig. 8, the search web page includes a selection unit 231 for selecting the memory area and type (group) to be searched within the storage 110, an input field 232, and a button 233 for starting an inquiry.

[0062] 8, multiple options can be selected using the selection unit 231, such as the root of the file path in the storage 110 where the document data is saved, or a name that distinguishes the server name. In the example shown in Fig. 8, the selection unit 231 is a check button, but other controls may also be used.

[0063] Text relating to document data to be searched for can be entered in natural language into the input field 232. The input field 232 may also allow image data to be entered.

[0064] When the button 233 is selected, the terminal device 2 accepts the search condition selected in the selection unit 231 to narrow down the search target and the text entered in the input field 232 as a query sentence (S203), and transmits them to the information processing device 1 (S204).

[0065] FIG. 9 shows an example of a display of a web page into which a query statement has been input. Similar to FIG. 8, FIG. 9 shows a state in which a web page provided by the information processing device 1 is displayed on a screen 230 based on a web browser program. In the example of FIG. 9, the query statement is input in an input field 232 on the web page in natural language, such as part of a conversation, such as "There must be a chart showing sales growth." When the user selects button 233, the query statement input in the input field 232 is transmitted to the information processing device 1. In the search system 100 of the present disclosure, a search is possible by inputting such natural language. The query statement is not limited to being received by inputting natural language by the user into the input field 232, but may also be created in the terminal device 2 or the information processing device 1 by selecting options for targets and conditions.

[0066] The processing unit 10 of the information processing device 1 provides the query statement "There should be a diagram showing sales growth" to the Embedding module EM of the model M1 to convert it into vector data (S122). The processing unit 10 then transmits to the terminal device 2 a list of some of the document data having similar vector data from the search table 111 for document data of the target storage area and type (group) selected by the selection unit 231 (S126).

[0067] FIG. 10 shows an example of a portion of document data searched using the search table 111. Similar to FIG. 8, FIG. 10 shows a state in which a web page updated by the information processing device 1 is displayed on a screen 230 based on a web browser program. Compared to FIG. 9, the web page shown in FIG. 10 displays a list 234 of a portion of the identified document data. The list 234 includes thumbnail images 235 of the images of the identified document data. All of the thumbnail images 235 in FIG. 10 are pages including a bar graph showing changes in sales. In this way, by using the search table 111, it is possible to extract target document data corresponding to a query statement.

[0068] FIG. 11 shows an example of a reply. Similar to FIG. 8, FIG. 11 shows a state in which a web page updated by the information processing device 1 is displayed on a screen 230 based on a web browser program. The web page shown in FIG. 11 displays reply 236 stating, "The extracted data includes charts showing sales growth. Each chart represents sales growth for a different item or for a different period." Reply 236 shown in FIG. 11 is created based on a prompt that requests a reply for all identified data.

[0069] 10 and the answer sentence 236 shown in FIG. 11, the user who entered the query sentence in FIG. 9 is likely to be able to recognize the document data they are looking for by looking at the list 234 of the section (page) containing the sales growth graph, the thumbnail image 235, and the name of the document data displayed together. The user can find the document data they are looking for by relying on the text included in the thumbnail image 235, such as the number of years of sales growth, the items of sales growth, and the number of years and title included in the name of the document data. In this way, the user can search for the document data they need by simply entering ambiguous natural language.

[0070] 12 shows an example of a display of a web page to which another query has been input. Similar to FIG. 9, FIG. 12 shows a state in which a web page provided by information processing device 1 is displayed on screen 230 based on a web browser program, and a query has been input. In the example of FIG. 12, a query has been input in input field 232 on the web page in natural language, such as the question "Do you have any photos of Luxembourg?" When the user selects button 233, the query input in input field 232 is sent to information processing device 1.

[0071] The processing unit 10 of the information processing device 1 provides the query "Do you have any photos of Luxembourg?" to the Embedding module EM of the model M1 to convert it into vector data (S122). The processing unit 10 then transmits to the terminal device 2 a list of some of the document data having similar vector data from the search table 111, which targets document data of the target storage area and type (group) selected by the selection unit 231 (S126).

[0072] FIG. 13 shows an example of a portion of document data searched using the search table 111. Similar to FIG. 12, FIG. 13 shows a state in which a web page updated by the information processing device 1 is displayed on a screen 230 based on a web browser program. Compared to FIG. 12, the web page shown in FIG. 13 displays a list 234 of a portion of identified document data. The list 234 includes thumbnail images 235 of images of the identified document data. Each of the thumbnail images 235 in FIG. 13 is an image of a page containing a description of Luxembourg and photographs of buildings in Luxembourg. By storing this document data in the storage 110, the information processing device 1 pre-processes and stores features extracted from image data within the page in the search table 111. Since these features are similar to those obtained from maps and buildings in Luxembourg, and furthermore, the word "Luxembourg" appears before and after the page containing the map and photographs of the building, it is highly likely that the weight of the word "Luxembourg" is high as a feature of this page. The page in question was identified by its similarity to the features of the query "Photos of Luxembourg."

[0073] FIG. 14 shows another example of a response sentence. Similar to FIG. 12, FIG. 14 shows a state in which a web page updated by the information processing device 1 is displayed on a screen 230 based on a web browser program. Following the display of the list 234 shown in FIG. 13, the web page shown in FIG. 14 displays a response sentence 236 to the query statement: "There is a photo of Luxembourg. The second document contains a photo of a building in Luxembourg. This photo contains..." This response sentence 236 is generated by adding image data extracted from one page of the identified document data and text requesting an explanation for this image data to the query statement, providing the result to the language model LM, and outputting the result.

[0074] In this way, the search system 100 of the present disclosure extracts target document data corresponding to a query statement by using the search table 111 obtained by preprocessing, and provides the extracted document data to the language model LM. This makes it possible to obtain an appropriate answer for the document data in the storage 110 by applying RAG (Retrieval-Augmented Generation) technology.

[0075] In particular, for document data that primarily consists of images, such as graphs rather than text, even if text can be obtained using OCR or character recognition from images, the context explained using the document may not match the obtained text. The processing procedure described in the first embodiment makes it possible to search for things explained in a target section by expressing them as vector data (e.g., a weighted word group in the context of that section) based on the recognition results of objects in the image data and their association with preceding and following separable units (page units). Therefore, even in document data that has a page where the graph itself is not titled "Sales Growth," if that page is intended to explain "Sales Growth," the feature values ​​corresponding to "Sales Growth" are stored, and the page can be searched for using natural language such as "There should have been a sales growth chart."

[0076] (Second embodiment) In the second embodiment, the information processing device 1 performs preprocessing by verbalizing as much of the image data as possible and then converting it into vector data. The configuration of the search system 100 in the second embodiment is the same as that of the first embodiment, except for the processing procedure described below. Therefore, among the configurations of the search system 100 in the second embodiment, the components common to the first embodiment are assigned the same reference numerals and detailed descriptions thereof will be omitted.

[0077] 15 is a flowchart showing an example of a procedure for pre-processing by the information processing device 1 of the second embodiment. The information processing device 1 crawls the storage and executes the following process. Whenever it is detected that document data has been added to the storage 110, the information processing device 1 may execute the following process. Among the process steps shown in FIG. 15, steps common to the process steps shown in FIG. 5 of the first embodiment are assigned the same step numbers, and detailed descriptions thereof will be omitted.

[0078] The processing unit 10 converts each of the data read in the predetermined separable unit in step S104 into an image (step S141). In step S141, the processing unit 10 converts each page, including pages consisting of text, into an image and temporarily stores it. In step S141, the processing unit 10 may convert each segment into an image.

[0079] The processing unit 10 generates a prompt requesting that text describing the imaged separable unit data be output (step S142), and provides the generated text together with the imaged data to the model M1 (step S143). In step S142, the processing unit 10 may instruct the Embedding module EM that a portion of one document data is continuous, and then provide the data continuously.

[0080] The processing unit 10 acquires explanatory text for the image output from the model M1 (step S144). The processing unit 10 provides the acquired explanatory text to the embedding module EM of the model M1 to convert it into vector data (step S145).

[0081] The processing unit 10 associates the identification data of the document data acquired in step S101 with the data identifying the portion read in step S104, adds the imaged data in step S141, the explanatory text acquired in step S144, and the vector data obtained in step S145 to the search table 111, and stores them (step S146), and then proceeds to step S110.

[0082] In the processing procedure shown in FIG. 15, the image is created regardless of whether it contains graphs, photographs, etc., and the meaning of the image is verbalized. The feature quantities (vector data) of the verbalized explanatory text are stored in a search table 111 and used for the search described below.

[0083] Fig. 16 is an explanatory diagram of a search table 111 in the second embodiment. Similar to the search table 111 shown in Fig. 6 of the first embodiment, the search table 111 shown in Fig. 16 stores the name of document data, the type of document data, the number of pages, and the feature amounts (vectors) of the document data in association with the document ID. In the example of Fig. 16 as well, the document data is read out in page units as a predetermined separable unit, and the feature amounts are stored.

[0084] In the search table 111 shown in Fig. 16, images are created on a page-by-page basis, so information about the position of the image is not stored, but image data of the imaged page itself is stored. Also, in the example of Fig. 16, the search table 111 stores the explanatory text acquired in step S144 in association with the document ID.

[0085] In the example of Fig. 16, for example, the attribute data for the document data of document ID "001" are stored as the name "*** / doc / file001.pdf", type "PDF", and number of pages "40", similar to the search table 111 shown in Fig. 6. The hierarchical information for the document data of document ID "001" stores that it is a "parent" with respect to other data. In the example of Fig. 16, each page of the name "*** / doc / file001.pdf" is imaged as a "child" and stored in the image data format "***.png".

[0086] For document data with document ID "001," the document ID "001_001" is assigned to the data (imaged data) of each page that is a "child," and hierarchical information is represented by the "child" information and the page number "1." The image data of the imaged page of this document ID "001_001" is given the name (file path) "*** / doc / file001_p001.png." Search table 111 stores explanatory text for the image data and vector data for the explanatory text, in association with document ID "001_001."

[0087] The second page of the same document data is also imaged and given the name (file path) "*** / doc / file001_p002.png." In the search table 111 shown in Fig. 16, for the second page, hierarchical information is represented by the information "child" and the page number "2" in association with the document ID "001_002," and the given name is stored. For the second page as well, the search table 111 stores explanatory text for the image data and vector data for the explanatory text in association with the document ID "001_002."

[0088] This makes it possible to perform searches using the following natural languages ​​in the second embodiment as well.

[0089] 17 is a flowchart showing an example of a search reception processing procedure in the second embodiment. The following processing is started when a user uses a web browser program on terminal device 2 to access a web page for searching documents stored in storage 110 using a web server function provided by information processing device 1. Among the processing procedures shown in FIG. 17, steps common to the processing procedures shown in FIG. 7 of the first embodiment are assigned the same step numbers and detailed descriptions thereof will be omitted.

[0090] In the second embodiment, when the information processing device 1 searches the search table 111 for the query statement in step S123, it identifies the imaged separable unit data as a part of the document data to which similar vector data is associated (S154).

[0091] After transmitting the list in step S126, the processing unit 10 of the information processing device 1 creates a prompt by combining the identified image data with the query sentence (step S157), and transmits the text output from the language model LM as an answer to the terminal device 2 (S130).

[0092] FIG. 18 is an explanatory diagram of processing in the second embodiment. In FIG. 18, the details of pre-processing are described at the top and the details of search processing are described at the bottom. The example in FIG. 18 will be described using document data consisting of two pages. First, as the pre-processing shown in FIG. 15, the information processing device 1 reads and images the document data acquired from the storage 110, page by page. The information processing device 1 then provides a prompt to the multimodal-compatible model M1 along with the image data to explain the imaged data (S143), and automatically creates explanatory text to verbalize the content of the page (S144). The information processing device 1 acquires features of the verbalized data using the Embedding module EM (S145) and stores them (S146, FIG. 16).

[0093] During a search, the information processing device 1 acquires the feature quantities of the query sentence using the Embedding module EM (S122), and searches the search table 111 for imaged pages associated with explanatory texts having similar feature quantities (S123). This may enable extraction of candidate document data that should be in the user's mind in the context of the query sentence.

[0094] The information processing device 1 then provides a prompt for the second page, which combines the image data, the query, and the text requesting an explanation, to the model M1 including the image recognition model IRM. This allows the document data that should be on the user's mind to be used as knowledge, and the accuracy of the answer from the language model LM to be improved and returned to the user in a manner similar to RAG technology.

[0095] In the second embodiment, as in the first embodiment, the above-described processing allows the necessary document data to be searched for in separable units in natural language from the storage 110 within the organization by inputting data on the web browser program P2 as shown in Figures 8-14.

[0096] In the second embodiment, the processing unit 10 of the information processing device 1 uniformly converts document data into images in separable units. However, the processing unit 10 may selectively use different methods depending on the determination, such as performing the process shown in Fig. 16 for document data that makes extensive use of images and the process shown in Fig. 5 for document data that is mainly text. In other words, the processing procedure of the first embodiment and the processing procedure of the second embodiment can be combined and used in parallel.

[0097] In the first and second embodiments, data searchable by the information processing device 1 is data that can be shared within an organization and is stored in the storage 110. The searchable data is not limited to document data within an organization, and may also be data uploaded to a cloud server within a service that can be shared when authentication is successful with an authorized terminal device 2.

[0098] The embodiments disclosed above are illustrative in all respects and are not restrictive. The scope of the present invention is defined by the claims, and includes all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0099] 1. Information processing equipment 10 Processing section 11 Storage section 110 Storage 111 Search Table P1 Information Processing Program M1 model EM Embedding Module LM Language Model IRM Image Recognition Model 2. Terminal Device 20 Processing section 22 Communications Department 23 Display section 234 List 235 thumbnail images 236 Answer text

Claims

1. On the computer, Acquire feature amounts of a plurality of document data, each of which includes a photograph, a diagram, a table, a graph, or text, stored in a storage device; The feature amount is stored in a storage unit in association with identification data for identifying the document data, Accepting an inquiry about the storage; The received query and document data extracted based on the feature amounts are provided to a language model, and an answer to the query is output. A computer program that executes a process.

2. The computer, converting each of a plurality of portions included in the document data into an image; Obtaining explanatory text that describes the content of the converted image using a language model; The explanatory text and / or vector data of the explanatory text are stored in the storage unit as the feature amount. The computer program product of claim 1 , which executes a process.

3. The computer, converting the document data into images on a page-by-page basis; obtaining descriptive text for each page from the language model by feeding the image to the language model; The vector data of the explanatory text is stored in the storage unit as the feature amount.

3. The computer program product according to claim 2, which executes a process.

4. The computer, extracting document data associated with features similar to the features of the received query; The received query and the extracted document data are provided to the language model.

4. A computer program according to claim 1, which causes a process to be executed.

5. The computer, Acquire feature values ​​for each of the multiple parts included in the acquired document data, the feature amount of each part is stored in the storage unit in association with the document data and identification data for identifying the plurality of parts; extracting document data or portions of document data associated with features similar to the features of the received query; The answer is output by providing the received inquiry and the extracted document data or part of the document data to the language model.

4. A computer program according to claim 1, which causes a process to be executed.

6. The computer, executing a process of storing a table that stores feature amounts of the document data and each of a plurality of portions included in the document data in association with identification data that identifies the document data and the plurality of portions; The table stores information indicating the location of the document data in the storage, information indicating the parent-child relationship between the plurality of parts and the document data, and information indicating the positions of the plurality of parts in the document data, together with the feature amounts. A computer program according to any one of claims 1 to 3.

7. the plurality of document data are classified into a plurality of groups according to storage areas in the storage or types of the document data, The computer, Accepting a group designation when accepting the inquiry; Extract the document data within the specified group 4. A computer program according to claim 1, which causes a process to be executed.

8. The computer, An answer to the query is output by providing a language model with the received query, document data extracted based on the feature amounts, a specification of a field related to the query, and text instructing an answer to the query.

4. A computer program according to claim 1, which causes a process to be executed.

9. The computer, In addition to the answer, screen information including a thumbnail image of the extracted document data or a portion of the document data and text related to the document data or the portion is output.

4. A computer program according to claim 1, which causes a process to be executed.

10. Acquire feature amounts of a plurality of document data, each of which includes a photograph, a diagram, a table, a graph, or text, stored in a storage device; The feature amount is stored in a storage unit in association with identification data for identifying the document data, Accepting an inquiry about the storage; The received query and document data extracted based on the feature amounts are provided to a language model, and an answer to the query is output. An information processing device including a processing unit that executes processing.

11. Acquire feature amounts of a plurality of document data, each of which includes a photograph, a diagram, a table, a graph, or text, stored in a storage device; The feature amount is stored in a storage unit in association with identification data for identifying the document data, Accepting an inquiry about the storage; The received query and document data extracted based on the feature amounts are provided to a language model, and an answer to the query is output. Information processing methods.

Images