Program, information processing method, and information processing device

Abstract

A novel program, information processing method, and information processing device are provided that improve the accuracy of answers from a generation AI compared to conventional methods. [Solution] In an information processing system, a program determines the format of a file, extracts non-text information contained in the file, and causes an information processing device, which is a management server 2, to perform the process of converting the extracted non-text information into a format that can be processed by a generation AI.

Description

[Technical Field]

[0001] The present invention relates to a program, an information processing method, and an information processing device. [Background technology]

[0002] Patent document 1 describes a similar image search device that searches a group of search target images for images with similar features to a search key image, the similar image search device including an image input unit that inputs a search key image, a feature extraction unit that extracts features from the search key image input by the image input unit, and a text conversion unit that converts the features into text information, the similar image search device having an image encoding unit that encodes the search key image into the text information, a similar image selection unit that selects similar images from the group of search target images based on the encoding result of the image encoding unit, and an image output unit that outputs the images selected by the similar image selection unit. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2002-318812 Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide a program, an information processing method, and an information processing device that solve the conventional problems.

[0005] One aspect of the present invention is a program that causes an information processing device to execute a process of determining the format of a file, extracting non-text information contained in the file, and converting the extracted non-text information into a format that can be processed by a generation AI.

[0006] According to the present invention, a novel program, information processing method, and information processing device are provided that improve the accuracy of answers from a generation AI compared to conventional methods. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a diagram illustrating an overall configuration of an information processing system according to an embodiment. [Figure 2] FIG. 2 is a block diagram of a user terminal 1 according to an embodiment. [Figure 3] FIG. 2 is a block diagram of a management server 2 and a large-scale language model server 3 in the embodiment. [Figure 4] 10 is a flowchart showing an operation in the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] First Embodiment (Information Processing System) 1 is a diagram showing the overall configuration of an information processing system according to an embodiment. The information processing system includes a user terminal 1, a management server 2, and a large-scale language model server 3, all of which are connected via a network NW. The network NW includes wired and wireless communications such as a mobile phone network and the Internet. Based on a request from the user terminal 1, the management server 2 requests processing from the large-scale language model server 3 and returns the obtained results to the user terminal 1. The management server 2 transmits predetermined information (described below) to the large-scale language model server 3, and the large-scale language model server 3 processes the received information. The extracted data is transferred to the management server 2 or the large-scale language model server 3, and data exchange between the servers uses protocols such as HTTP / HTTPS, RESTful API, and WebSocket.

[0009] (User terminal 1) 2 is a block diagram of a user terminal 1 according to an embodiment. The user terminal 1 includes a storage unit 11, a control unit 12, a communication unit 13, an input unit 14, and an output unit 15. The user terminal 1 is, for example, an information processing device such as a mobile phone, a smartphone, a desktop or notebook personal computer, a mobile terminal such as a tablet, a head-mounted display, or a wearable terminal such as a smartwatch or smart glasses.

[0010] The storage unit 11 stores various types of information about the user terminal 1. The storage unit 11 may also store programs that operate the user terminal 1. The storage unit 11 is configured with, for example, an HDD, SSD, eMMC, RAM, or ROM. The ROM may be a mask ROM, flash memory, PROM, EEPROM, or the like.

[0011] The control unit 12 controls the user terminal 1 based on a program. The control unit 12 is, for example, a processor such as a CPU, an MPU, or a GPU. The program may be stored in the storage unit 11, or may be stored outside the user terminal 1 and used by the user of the user terminal 1 through communication using the network NW.

[0012] The communication unit 13 includes an antenna and a communication circuit, and performs communication compatible with the network NW.

[0013] The input unit 14 inputs images, voice, characters, etc. The input unit 14 is a character input device such as a keyboard, mouse, touchpad or touch panel, a voice input device such as a microphone, or an image input device such as a camera.

[0014] The output unit 15 is a device that outputs images, sounds, characters, etc. The output unit 15 is, for example, a display panel such as a liquid crystal panel, a micro LED, or an organic EL panel, but may also be a device that projects images using a projector or the like. An audio output device such as a speaker, earphones, headphones, or a terminal thereof may also be included.

[0015] The memory unit 11, the control unit 12, the communication unit 13, the input unit 14, and the output unit 15 may each be provided within the housing of the user terminal 1, or may be provided outside the housing of the user terminal 1 and connected to the user terminal 1 via a wired or wireless connection.

[0016] (Management Server 2) FIG. 3 is a block diagram of the management server 2 and the large-scale language model server 3 according to an embodiment. The management server 2 includes a storage unit 21, a control unit 22, a communication unit 23, an input unit 24, and an output unit 25. Specific examples of the storage unit 21, the control unit 22, the communication unit 23, the input unit 24, and the output unit 25 are the same as those of the storage unit 11, the control unit 12, the communication unit 13, the input unit 14, and the output unit 15, respectively. The management server 2 is an information processing device, an example of which is a server capable of communicating with multiple terminals, including the user terminal 1. The management server 2 may include an authentication unit or a security unit that performs user authentication, access control, and data encryption using TLS / SSL, etc. A database (RDBMS or NoSQL) for recording user account information, setting information, history information, etc. may be provided in the storage unit 21 or in an external dedicated storage device. Multiple management servers 2 may be configured via a load balancer to improve load balancing and redundancy. The control unit 22 executes the processing of the management server 2 according to a control program. The program may be stored in the storage unit 21, or may be stored outside the management server 2 and used by a user of the management server 2 through communication using the network NW.

[0017] (Large-scale language model server 3) The large-scale language model server 3 includes a storage unit 31, a control unit 32, a communication unit 33, an input unit 34, an output unit 35, and an image recognition engine 36. Specific examples of the storage unit 31, the control unit 32, the communication unit 33, the input unit 34, and the output unit 35 are the same as those of the storage unit 11, the control unit 12, the communication unit 13, the input unit 14, and the output unit 15, respectively. The storage unit 31 includes a large-scale language model (LLM). Examples include, but are not limited to, information processing devices that process GPT-4, GPT-4o, Gemini, Claude3, Mistral, and LLaMA2 (all of which are registered trademarks). New models developed in the future, or tuned or fine-tuned models of existing models, may also be used. The storage unit 31 and the control unit 32 may be equipped with multiple computing devices, such as high-performance CPUs, GPUs, TPUs, and memories, and their configurations may be flexibly changed according to performance requirements. Furthermore, the storage unit 31 may link with external cloud providers or API services to outsource part of the processing or reference external data. Furthermore, a plurality of units may be configured via a load balancer, which may improve load distribution and redundancy. The control unit 32 executes processing using a large-scale language model according to a control program.

[0018] The output unit 35 provides an interface for outputting the processed information to an external device. The image recognition engine 36 has the function of extracting and analyzing non-text information using a large-scale language model. For example, using advanced image analysis technology utilizing a large-scale language model such as GPT-4o, it recognizes structured information such as text, tables, and flowcharts in images and extracts them as corresponding digital data.

[0019] (operation) 4 is a flowchart showing the operation in this embodiment. The user terminal 1 transmits a file to the management server 2 (step S101). The user terminal 1 may also transmit text such as a question or request together with the file. The management server 2 receives the file (step S201).

[0020] (File identification process) The management server 2 determines the format of the file received from the user terminal 1 (step S202). This file determination process may be executed by either the control unit 22 of the management server 2 or the large-scale language model server 3, or may be shared between them.

[0021] When the management server 2 receives a single file, it first automatically determines the format of that file based on the extension or file header information. If multiple files are received, it automatically determines the format for each file. To determine the file format, for example, the MIME type, extension, file magic number at the beginning of the bytes, or an identification process using a standard API is used.

[0022] The file format can be any, including a text file (.txt) that contains only text and no structure (no formatting, images, or charts); a JPEG file (.jpg) that contains only visual information such as a pixel image; an HTML file (.html) that contains a text-based web document structured with HTML tags and includes references to external images (e.g., IMG tags); or a PDF file (.pdf) that contains text data with layout, or whose content is entirely images, such as a scanned document.

[0023] It may also be a Word® file (.docx), PowerPoint® file (.pptx), or Excel® file (.xlsx), which contain not only text information but also visual structure data and embedded image information within a single file. It may also be an MP4 file (.mp4), which is an example of a video file consisting of consecutive frames of audio and video. In this case, the audio is transcribed and converted into text.

[0024] (Text extraction processing) The management server 2 analyzes the file (step S203). The management server 2 performs text extraction, image conversion, and large-scale language model processing on the received file. If a file contains text information, the management server 2 performs extraction processing on that text information. The text information is stored structured or sequentially, and character string data can be programmatically acquired without external image analysis. For example, the text information is stored as character string data based on encoding such as pure ASCII or UTF-8, and contains only control characters such as line break codes and spaces, but does not contain layout or visual structure. The text data contained in the file is extracted and directly converted into the Markdown format described below, while retaining structures such as paragraphs, headings, and bullet points.

[0025] The management server 2 can read such text data as is via the storage unit 21 or API, and directly acquire the document contents in the control unit 22. Since no image processing or structural analysis is involved, high-speed, low-load processing is achieved.

[0026] (Image conversion processing) Image conversion is a process in which, when an input file contains visual structure data, the file is rendered on a page-by-page or slide-by-slide basis to generate image information. Visual structure data refers to visually structured information such as diagrams, graphs, and flowcharts. This visual structure data is stored as structured data within the file, including properties such as position information, shape information, color attributes, character information, and font size for objects such as geometric shapes and text boxes. This allows structural elements such as diagrams to be extracted as recognizable information. Information stored in vector graphics format is also included in visual structure data.

[0027] Image information is information stored primarily as pixel-based images, such as PNG or JPEG. Both the visual structure data and the image information are non-textual information, and together they are referred to as non-textual information. This allows the management server 2 to convert the visual structure data into a format that can be processed by the image recognition engine 36. The management server 2 may also perform image conversion processing on not only the visual structure data but also some text information, such as information surrounding the visual structure data, or all text information in the file. This allows data processing to be performed on, for example, a diagram and text information, such as the diagram's description, together, thereby improving the accuracy of document searches using Retrieval-Augmented Generation (RAG). Part or all of this image conversion processing may be performed by the control unit 32 of the large-scale language model server 3.

[0028] (Large-scale language model processing) Large-scale language model processing is an information extraction process performed on image information (first image information that is an image at the time of input, or second image information that is an image generated by image conversion processing), recognizing visual information such as characters and diagrams contained in the image and extracting its content and structure as text data or digital data. The analysis targets the visual structure (diagrams, graphs, flowcharts, etc.) and characters contained in the image. The management server 2 performs this large-scale language model processing.

[0029] Alternatively, the image recognition engine 36 of the large-scale language model server 3 may perform part or all of this large-scale language model processing. When the image recognition engine 36 performs information extraction processing using a large-scale language model, the management server 2 transmits the image generated by the image conversion processing to the large-scale language model server 3. In this case, the image recognition engine 36 of the large-scale language model server 3 performs optical character recognition (OCR) processing on the characters in the image, converting the pixel data into the corresponding character code. The converted character strings are reconstructed into paragraphs or sentences, obtaining text information that retains natural language context. Next, visual structure data is analyzed for figures, tables, structures, etc., to extract, for example, the matrix structure of a table, the axes, values, and legends of a graph, and the process blocks and their connections (such as arrows) of a flowchart as digital data. This digital data obtained by the image recognition engine 36 is re-expressed as character string data in a subsequent Markdown format conversion process and is then subjected to processing by the generation AI. The large-scale language model server 3 transmits this digital data to the management server 2. Note that part or all of the large-scale language model processing may be performed by the management server 2.

[0030] (Markdown format conversion) The management server 2 converts the digital data obtained by the large-scale language model processing into character string data in Markdown format, a format that can be processed by the generation AI (step S204). Markdown format conversion here refers to the process of describing each component of the digital data according to predetermined syntax rules and converting it into character string data in a structured document format (Markdown format) that allows the generation AI to understand the semantic structure. In this conversion, for example, for text information, headings are expressed as "#heading", lists as "-item", and tables as "|cell1|cell2|", and paragraphs, hierarchical relationships, table structures, etc. are explicitly described. Markdown conversion is also performed on the digital data read after converting the text information into image information. Visual structure data is structurally extracted using image recognition in the previous stage and then reconstructed as Markdown syntactic units. Note that part or all of this process may be performed by the large-scale language model server 3.

[0031] The management server 2 sends the Markdown-converted file to the large-scale language model server 3 and sends a search request (step S205). The large-scale language model server 3 receives the search request (step S305) and performs a search based on the Markdown-converted file (step S306).

[0032] (Search and answer generation using large-scale language models) The search request received by the large-scale language model server 3 is used as input data for the search process of the large-scale language model executed by the large-scale language model server 3. In this embodiment, the information converted into Markdown format is configured to perform a RAG-type semantic search, which performs a semantic search on a vector database, and the large-scale language model comprehensively interprets the search results and the query information from the user to generate an optimal response sentence in natural language. Here, "search" refers to the process of converting document data based on the information converted into Markdown format into semantic vectors and extracting related information based on the semantic proximity to the query sentence.

[0033] The large-scale language model server 3 transmits the answer generated in this manner to the user terminal 1 via the management server 2 (step S307). The information transmitted by the management server 2 may be the same as the answer received from the large-scale language model server 3, or may be a partially modified answer based on the answer received from the large-scale language model server 3. The management server 2 receives the answer (step S207) and transmits the answer to the user terminal 1 (step S208). The user terminal 1 receives the answer, and the output unit 15 outputs the answer, for example by displaying it (step S108). The information received and displayed by the user terminal 1 is not limited to the answer, but may be any information obtained by determining the format of the transmitted file and then processing non-text information extracted from the file into Markdown format, which is a format processable by the generation AI.

[0034] Of the above, some or all of the configurations or functions described as being possessed by one of the user terminal 1, management server 2, and large-scale language model server 3 may be possessed by another one. For example, some of the configurations described as being possessed by the user terminal 1 may be possessed by the management server 2.

[0035] The management server 2 and the large-scale language model server 3 may be provided on a cloud service provider. Although the management server 2 and the large-scale language model server 3 are described as separate servers, they may be the same server. Each of the management server 2 and the large-scale language model server 3 may be composed of multiple separate servers.

[0036] According to this embodiment, the information processing device (administrative server 2) executes the process of determining the format of a file, extracting non-text information contained in the file, and converting the extracted non-text information into a format that can be processed by the generation AI. This structured the non-text information, making it easier for the generation AI to grasp the appropriate context. This improves the accuracy of document search using RAG, thereby improving the accuracy of the generation AI's answers. This also enables response generation based on advanced semantic understanding, including non-text information, which was difficult using conventional string search or keyword matching methods. It also enables specialized processing for different file formats.

[0037] In addition, since the format that the generation AI can process is Markdown format, the text is converted into a format that the generation AI can easily process, which can improve the accuracy of the generation AI's answers.

[0038] Furthermore, since non-text information is visually structured data, information can be obtained from sources other than text information.

[0039] In addition, the image information converted from the visual structure data is converted into a format that can be processed by the generation AI, so information can be obtained from the image.

[0040] The non-text information is visual structure data and first image information, and the visual structure data is converted and processed into second image information, and the first image information and second image information are converted into a format that can be processed by the generation AI.This makes it easier for the generation AI to grasp the appropriate context even when a single file contains visual structure data and image information, thereby improving the accuracy of the generation AI's answers.

[0041] Furthermore, since one file contains text information, document search using RAG is performed including the text information, improving accuracy.

[0042] In addition, the text information is extracted and directly converted to Markdown, and after converting the text information to image information, the read text data is converted to Markdown, thereby improving the accuracy of the answers provided by the generation AI.

[0043] Furthermore, since a single file contains both textual and non-textual information, a variety of information within a single file can be analyzed.

[0044] In addition, the text information is extracted and directly converted to Markdown, and after converting the text information to image information, the read text data is converted to Markdown, which further improves the accuracy of the text information reading when the generation AI responds.

[0045] The program also causes an information processing device, which is a user terminal 1, to execute a process of sending a single file and receiving a response based on the single file, and the response is obtained by determining the format of the single file and then processing the non-text information extracted from the single file into a format that can be processed by the generation AI, so it is possible to receive information based on the generation AI with improved accuracy of responses, etc. This also makes it possible to receive information based on advanced semantic understanding, including non-text information, which was difficult to achieve with conventional string search or keyword matching methods.

[0046] The present invention is not limited to the above-described embodiments, and it is clear that various modifications are included within the scope of the present invention as long as they do not deviate from the scope of the present disclosure. Furthermore, the means disclosed in each of the above-described embodiments may be combined as appropriate within the scope of feasibility. While the present disclosure has been described based on the embodiments, it is understood that the invention as claimed is not limited to these embodiments or structures. The present disclosure also encompasses various modifications and modifications within the scope of equivalents. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0047] The processes described using flowcharts in the above embodiments do not necessarily have to be performed in the order shown. Some process steps may be performed in parallel. In addition, additional process steps may be employed, and some process steps may be omitted. [Explanation of symbols]

[0048] 1. User terminal 11 Storage section 12 Control Unit 13 Communications Department 14 Input section 15 Output section 2 Management Server 21 Memory section 22 Control Unit 23 Communications Department 24 Input section 25 Output section 3 Large-scale language model server 31 Storage section 32 Control section 33 Communications Department 34 Input section 35 Output section 36 Image Recognition Engine NW Network

Claims

1. Identify the format of a single file, Analyzing the visual structure data, which is non-text information included in the one file, by image recognition, and extracting it as digital data including the matrix structure of a table, the axes and values ​​of a graph, or the connection relationships between processing blocks of a flowchart; The extracted digital data is converted into character string data in Markdown format, which is a format that can be processed by the generation AI. A program that causes an information processing device to execute a process.

2. The one file contains text information. The program according to claim 1.

3. Extracting the text information and directly converting it to Markdown; and After converting the text information into image information, the read data is converted into Markdown. The program according to claim 2.

4. A process of sending one file and receiving a response based on the one file sent A program to be executed by an information processing device, The answer is obtained by determining the format of the one file, then analyzing the visual structure data, which is non-text information contained in the one file, using image recognition to extract digital data including the matrix structure of a table, the axes and values ​​of a graph, or the connections between processing blocks in a flowchart, and converting the extracted digital data into character string data in Markdown format, which is a format that can be processed by the generation AI. program.

5. Identify the format of a single file, Analyzing the visual structure data, which is non-text information included in the one file, by image recognition, and extracting it as digital data including the matrix structure of a table, the axes and values ​​of a graph, or the connection relationships between processing blocks of a flowchart; The extracted digital data is converted into character string data in Markdown format, which is a format that can be processed by the generation AI. A method in which an information processing device performs an operation.

6. Identify the format of a single file, Analyzing the visual structure data, which is non-text information included in the one file, by image recognition, and extracting it as digital data including the matrix structure of a table, the axes and values ​​of a graph, or the connection relationships between processing blocks of a flowchart; The extracted digital data is converted into character string data in Markdown format, which is a format that can be processed by the generation AI. An information processing device having a control unit that performs control.

Images