Information processor, information processing method and information processing program

Abstract

To provide an information processor, an information processing method and an information processing program which output structured character strings.SOLUTION: An information processor includes: a storage part for storing knowledge data in which a first image and structured information of character strings recorded in the first image are associated with each other; an acquisition part for acquiring a second image specified by a user; an estimation part for estimating similarity between the second image acquired by the acquisition part, and the first image based on the knowledge data stored in the storage part; a presentation part for presenting a predetermined number of first images as candidate images to the user in the descending order of the similarity estimated by the estimation part; and an output control part for performing control to output the structured information corresponding to the selected candidate image, when one of the candidate images presented by the presentation part is selected.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, there is a device that performs OCR on a form to extract character strings and checks the character strings (see Patent Document 1). When checking the character strings, the device allows the user to distinguish between different workflow types and presents multiple options for correcting the character strings according to a priority order determined for each identified workflow type. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-056732 Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, when using a figure containing text (for example, a table or graph) with a generation AI, the generation AI must structure the text according to the structure of the figure (table, graph, etc.) in order to use the text in the figure in the same way as the figure's structure (in accordance with the meaning of the figure). The technology described in Patent Document 1 does not describe structuring character strings.

[0005] The present disclosure provides an information processing device, an information processing method, and an information processing program that output a structured character string. [Means for solving the problem]

[0006] An information processing device of one embodiment includes a memory unit that stores knowledge data that associates a first image with structured information of a character string recorded in the first image; an acquisition unit that acquires a second image specified by a user; an estimation unit that estimates the similarity between the second image acquired by the acquisition unit and the first image based on the knowledge data stored in the memory unit; a presentation unit that presents a predetermined number of first images to the user as candidate images in order of highest similarity estimated by the estimation unit; and an output control unit that, when one of the candidate images presented by the presentation unit is selected, controls the output of structured information corresponding to the selected candidate image. [Effects of the Invention]

[0007] The information processing device, information processing method, and information processing program of the present disclosure can output a structured character string. [Brief explanation of the drawings]

[0008] [Figure 1] 1A and 1B are diagrams illustrating conventional character information obtained by performing OCR on character strings written in bar graphs. (A) shows a bar graph on which character strings are written, and (B) shows character information obtained by OCR based on the bar graph shown in (A). [Figure 2] 1A and 1B are diagrams illustrating an example of structured information about a character string written in a first image (bar graph) according to the present embodiment, in which (A) shows an example of a first image (bar graph) in which a character string is written, and (B) shows an example of structured information about the character string based on the first image described in (A). [Figure 3] FIG. 10 is a diagram for explaining an example of a second image. [Figure 4] FIG. 10 is a diagram illustrating an example of a candidate image. [Figure 5] FIG. 1 is a block diagram illustrating an information processing device according to an embodiment. [Figure 6] 1 is a flowchart illustrating an information processing method according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] An embodiment will be described below.

[0010] [Overview of information processing device 100] First, an overview of an information processing device 100 according to an embodiment will be described. Fig. 1 is a diagram for explaining conventional character information obtained by performing OCR on character strings written on a bar graph. Fig. 1(A) shows a bar graph on which character strings are written, and Fig. 1(B) shows character information obtained by OCR based on the bar graph shown in (A). 2A and 2B are diagrams illustrating an example of structured information about a character string written in a first image (bar graph) according to this embodiment. Fig. 2A shows an example of a first image (bar graph) in which a character string is written, and Fig. 2B shows an example of structured information about the character string based on the first image described in (A).

[0011] The information processing device 100 may be configured as a presentation device or the like that, when a character string is written in a diagram (for example, a table and a graph), presents a character string structure (an example of a character string structure) based on another diagram that is similar to the structure of the diagram (the table and graph, and the position of the character string written in the table and the table graph). The information processing device 100 may also be configured as a conversion device or the like that automatically converts a character string written in a diagram into structured information of the character string so that it can be used by a generation AI. The information processing device 100 is not limited to the device of the example described above, and may be configured as various devices or the like. The information processing device 100 may be a computer such as a server, a desktop, a laptop, a tablet, or a smartphone.

[0012] As shown in Figure 1(A), when there are two bar graphs, multiple items (character strings) can be entered in one bar graph (stacked bar graph), and the time progression of the multiple items (character strings) can be seen in a total of two bar graphs (stacked bar graphs). Conventionally, when OCR is performed on such character strings, all character strings are displayed consecutively, regardless of the time progression of each item (character string) (see Figure 1(B)). Similarly, for example, if there is a table with two rows and two columns, each cell in the table contains a character string. Conventionally, when OCR is performed on an image containing such character strings, all of the character strings written in all of the cells may be displayed consecutively, regardless of the position of each cell in the table. A string of characters that is not structured according to such a diagram cannot be used as a meaningful string of characters by a generative AI.

[0013] Therefore, in this embodiment, the character string written in the figure is structured according to the figure so that the character string can be used by the generation AI. That is, when another figure (first image) is similar to the target figure (second image), the information processing device 100 presents a character string (structured information) that has been previously structured based on the other figure (first image). As an example, when there is a first image (e.g., a bar graph (stacked bar graph)) as illustrated in FIG. 2(A), the information processing device 100 presents structured information (see FIG. 2(B)) in which the character string written in the first image is structured. In the conventional text information of Fig. 1(B), multiple character strings are recorded consecutively. On the other hand, the structured information of this embodiment illustrated in Fig. 2(B) is information in which only character strings are structured according to the structure of the first image (bar graph) of Fig. 2(A) (department, fiscal year, numerical values ​​corresponding to the fiscal year, and the positions of increase / decrease rates, etc. (character string structure)). Note that the composition information of the first image and the character strings corresponding to the first image is not limited to that illustrated in Fig. 2, and may be various. As a result, the user or the information processing device 100 refers to the structural information of the character string based on the first image, and structures the character string written in the drawing (second image) so that it is similar to the structural information.

[0014] Specifically, the information processing device 100 stores, as knowledge data, a plurality of pieces of data (correspondence information) that associate a first image with structured information on character strings recorded in the first image. The structured information on character strings may be information that structures the structure of the first image, for example, character strings written according to each item in a table cell and a bar graph, so that the structure of the first image can be understood (for example, so that the relationship between the positions of each cell in a table and each cell in a bar graph corresponding to each character string can be understood) (see the example in FIG. 2).

[0015] FIG. 3 is a diagram illustrating an example of the second image. FIG. 4 is a diagram illustrating an example of the candidate image.

[0016] The information processing device 100 acquires a second image specified by a user. As an example, the information processing device 100 acquires a second image (bar graph) illustrated in FIG. 3. That is, when a second image to be subjected to character string structuring is specified by a user (for example, input via the input unit 121 (see FIG. 5) and a user terminal (not shown)), the information processing device 100 accepts the specified second image. The input unit 121 may be, for example, a keyboard and a mouse. Furthermore, the user terminal is a terminal used by a user, and may be, for example, a desktop, a laptop, a tablet, a smartphone, or the like.

[0017] The information processing device 100 estimates the similarity between the acquired second image and the first image based on the knowledge data. As an example, the information processing device 100 may estimate the similarity between the first image and the second image based on the distributed representation of the first image and the distributed representation of the second image. As a more specific example, the information processing device 100 may represent each of the first image and the second image (the meaning of the image) as a vector (numerical value) as the distributed representation, and estimate the similarity between the vector (numerical value) of the meaning of the first image and the vector (numerical value) of the meaning of the second image (based on the distance and positional relationship between the vectors). Various processes, including known natural language processing, may be used for such distributed representations (vectors).

[0018] The information processing device 100 presents a predetermined number of first images to the user as candidate images in descending order of the degree of similarity estimated as described above. The predetermined number may be various numbers (single or multiple), such as 3, 5, 8, 10, etc. As shown in an example in FIG. 4, the information processing device 100 presents a predetermined number of first images (candidate images) similar to the second image (see FIG. 3) in descending order of similarity (in the example shown in FIG. 4, candidate images from "1st place" with the highest similarity to "3rd place" in descending order of similarity).

[0019] When one of the candidate images presented as described above is selected by the user (for example, by input via the input unit 121 and the user terminal), the information processing device 100 refers to the knowledge data and outputs structured information corresponding to the selected candidate image (first image). As an example, when the first candidate image illustrated in Fig. 4 is selected, the information processing device 100 presents structured information (see Fig. 2(B)) of the character string corresponding to the selected candidate image (first image (see Fig. 2(A))). The structured information here is the structured information of the character strings described above, and may be information that structures the structure of the first image, for example, the character strings written according to each item in a table cell and a bar graph, so that the structure of the first image can be understood (for example, so that the relationship between the positions of each cell in the table and each cell in the bar graph corresponding to each character string can be understood).

[0020] [Details of the information processing device 100] Next, the information processing device 100 according to an embodiment will be described in detail. FIG. 5 is a block diagram illustrating an information processing device 100 according to an embodiment.

[0021] The information processing device 100 includes, for example, an input unit 121, a communication unit 131, a storage unit 132, a display unit 133, and a control unit 110. The communication unit 131, the storage unit 132, and the display unit 133 may be an embodiment of an output unit. The control unit 110 includes, for example, an acquisition unit 111, an estimation unit 112, a presentation unit 113, an output control unit 114, and a generation AI unit 115. The control unit 110 may be configured, for example, by an arithmetic processing unit of the information processing device 100. The control unit 110 (for example, an arithmetic processing unit) may realize the functions of each unit (for example, the acquisition unit 111, the estimation unit 112, the presentation unit 113, the output control unit 114, and the generation AI unit 115) by, for example, appropriately reading and executing various programs stored in the storage unit 132. In other words, the functions of each unit may be realized by computer implementation.

[0022] The input unit 121 may be, for example, an input interface such as a keyboard and a mouse, etc. The input unit 121 may be, for example, a graphical user interface (GUI).

[0023] The communication unit 131 is, for example, a communication interface that can transmit and receive various information to and from devices external to the information processing device 100 (external devices).

[0024] The storage unit 132 may store, for example, various information and programs. Examples of the storage unit 132 may be a memory, a solid state drive, a hard disk drive, etc. Note that the storage unit 132 may be, for example, a storage area or a server on a cloud.

[0025] The storage unit 132 stores knowledge data that associates a first image with structured information of a character string recorded in the first image. The storage unit 132 may store, as the knowledge data, a plurality of pieces of data (correspondence information) that associates a first image, a distributed representation of the first image, and structured information of a character string recorded in the first image.

[0026] The first image may be, for example, a plurality of types of diagrams, including a graph, a box diagram, a tree diagram, a table, a flow chart, etc. The first image may include a character string.

[0027] The structured information of the character string may be, for example, information that structures the character strings written according to the structure of the first image, e.g., each item such as a cell in a table and a bar graph, so that the structure of the first image can be understood (e.g., so that the relationship between the position of each cell in a table and each item in a bar graph corresponding to each character string can be understood) (see an example in Figure 2). The structured information of the character string is information that structures each of the multiple character strings recorded in the first image according to the position at which the character string is recorded in the first image, and is information that is structured by separating the multiple character strings (between character strings) with lines, symbols, spaces, tabs, etc. and writing the character strings over one or more lines. By structuring the character strings, the generation AI, etc. can refer to the character strings recorded in the first image as meaningful content as written in the first image.

[0028] The distributed representation of the first image may be, for example, a vector representation of the first image (the meaning of the first image), i.e., a vector of the meaning of the first image, etc. The distributed representation (vector) may be obtained by using various processes, including, for example, well-known natural language processing.

[0029] The knowledge data is, for example, information stored in relation to the first image and available to the information processing device 100, that is, information about so-called "knowledge" of the first image available to a computer.

[0030] The display unit 133 is a display capable of displaying, for example, various characters, symbols, images, and the like.

[0031] The acquisition unit 111 acquires a second image specified by the user. The acquisition unit 111 may acquire, as the second image, an image recording at least one of a graph, a box diagram, a tree diagram, a table, and a flowchart.

[0032] The second image is an image that is the target of structuring the character string so that it can be referenced by a generation AI, etc. (See an example in Figure 3.) The character string is recorded in the second image. As shown in Figure 3 as an example, when there are two graphs (for fiscal years 2023 and 2024), each graph can contain multiple items (character strings) (departments, values ​​for each department (W1, W2, X1, X2, Y1, Y2, Z1, Z2), and the rate of increase or decrease of each value, etc.), and the time trends of multiple items (character strings) can be seen on a total of two graphs. Similarly, as an example, if the second image is a table with multiple rows and multiple columns, a character string is written in each cell of the table.

[0033] The acquisition unit 111 acquires the second image from an external device (not shown), for example, via the communication unit 131. The external device may be, for example, a server, a user terminal, etc. The user terminal is a terminal used by a user of the information processing device 100, and may be, for example, a desktop, a laptop, a tablet, a smartphone, etc. Furthermore, for example, when the second image is recorded in an external memory (not shown) and the external memory is connected to an interface of the information processing device 100, the acquiring unit 111 may acquire the second image from the external memory. In addition, when there is text information (text file) containing a second image and the second image is selected (when an area in which the second image is to be written is specified) via the input unit 121 and a user terminal, etc., the acquisition unit 111 may acquire the selected second image (the specified area as the second image).

[0034] The estimation unit 112 estimates the similarity between the second image acquired by the acquisition unit 111 and the first image based on the knowledge data stored in the storage unit 132. The estimation unit 112 may, for example, acquire a distributed representation of the second image acquired by the acquisition unit 111, and estimate the similarity between the second image and the first image based on the distributed representation of the second image and the distributed representation of the first image stored in the storage unit 132. First, the estimation unit 112 acquires the distributed representation of the second image acquired by the acquisition unit 111. As an example, the estimation unit 112 may acquire the distributed representation of the second image by using various processes including known natural language processing. The distributed representation of the second image may be, for example, a vector representation of the second image (the meaning of the second image), i.e., a vector of the meaning of the second image. Next, the estimation unit 112 estimates the similarity between the first image and the second image, for example, based on the distributed representation of the first image (the vector of the meaning of the first image) and the distributed representation of the second image (the vector of the meaning of the second image), i.e., based on the relationship between the distance and position of the vectors of the first image and the second image.

[0035] The presentation unit 113 presents a predetermined number of first images to the user as candidate images in descending order of similarity estimated by the estimation unit 112 (see an example in FIG. 4). The presentation unit 113 identifies a first image that is more similar to the second image estimated by the estimation unit 112. That is, the presentation unit 113 identifies multiple first images in descending order of similarity to the second image estimated by the estimation unit 112. The presentation unit 113 presents a predetermined number of first images to the user as candidate images in descending order of similarity to the second image. The predetermined number may be various numbers (single or multiple), such as 3, 5, 8, 10, etc. In the example shown in FIG. 4, the presentation unit 113 presents the first to third most similar first images (candidate images) to the second image (see FIG. 3).

[0036] For example, the presentation unit 113 may control the display unit 133 to display one or more candidate images as presentation of the candidate images to the user. Furthermore, the presenting unit 113 may control the communication unit 131 to transmit information about one or more candidate images (candidate image information) to an external device (not shown) to present the candidate images to the user. The external device here may be, for example, a user terminal.

[0037] When one of the candidate images presented by the presentation unit 113 is selected, the output control unit 114 controls to output structured information corresponding to the selected candidate image. That is, when one of the candidate images presented by the presentation unit 113 is selected by a user (for example, by input via the input unit 121 or a user terminal), the output control unit 114 refers to the knowledge data stored in the storage unit 132 and outputs structured information corresponding to the selected candidate image (first image). When the candidate images (first images) illustrated in FIG. 4 are presented to the user and the top candidate image (first image (see FIG. 2(A))) is selected, the output control unit 114 outputs structured information (see FIG. 2(B)) of the character string corresponding to the candidate image (first image (see FIG. 2(A))).

[0038] The output control unit 114 may, for example, control the display unit 133 to display the structured information of the character string. The output control unit 114 may, for example, control the communication unit 131 to transmit the structured information of the character string to an external device (not shown). The external device here may be, for example, a user terminal or the like.

[0039] The structured information here is information that has been structured so that the character string recorded in the selected candidate image (first image) can be read meaningfully by a generation AI or the like. In other words, as shown in an example in Figure 2, when there are two graphs (first quarter (Q1) of fiscal year 2022 and first quarter (Q1) of fiscal year 2023), the values ​​for each period of multiple departments (department A, department B, and department C) are shown, i.e., in fiscal year 2022 Q1, department A is 150 billion, department B is 200 billion, and department C is 100 billion, and in fiscal year 2023 Q1, department A is 200 billion, department B is 250 billion, and department C is 150 billion, and the rate of increase or decrease in the values ​​of each department can be read (rather than simply listing the strings as in the conventional case in Figure 1 (B)). This structured information structures each string according to the structure (content) of the two graphs so that the rate of increase or decrease in the values ​​of each department can be read (rather than simply listing the strings as in the conventional case in Figure 1 (B)). Similarly, as an example, if the candidate image (first image) is a table with two rows and two columns, the structured information is information of a string of characters structured so that the positional relationship of each square in the top left, bottom left, top right, and bottom right of the table can be determined.

[0040] This enables the information processing device 100 to present structured information (see FIG. 2(B)) of the first image (candidate image) that is most similar to the second image, even when it is unclear how to structure multiple character strings recorded in the second image (see FIG. 3). Furthermore, by presenting structured information of the first image (candidate image) that is most similar to the second image, the information processing device 100 can structure the character strings recorded in the second image in the same way as the structure (composition information) of the character strings in the first image (candidate image), so that the meaning can be understood by a generation AI or the like (the character strings can be used by a generation AI or the like).

[0041] When structuring the character strings to be recorded in the second image (see FIG. 3) by referring to the composition information of the first image (see FIG. 2(B)), the output control unit 114 may automatically structure the character strings to be recorded in the second image. That is, the output control unit 114 may control the output of the structure information of the character strings to be recorded in the second image acquired by the acquisition unit 111 in accordance with the structure information of the character strings to be recorded in the first image corresponding to the selected candidate image. The output control unit 114 automatically structures the character strings to be recorded in the second image (generates the structure information of the second image) by arranging each of the multiple character strings to be recorded in the second image and separating the multiple character strings (between character strings) with lines, symbols, spaces, tabs, or the like, in the same manner as the structure (structure information) of the character strings of the selected candidate image (first image).

[0042] The output control unit 114 may control the output unit to output the automatically generated structured information of the character string of the second image. The output unit may be, for example, a communication unit 131, a storage unit 132, a display unit 133, etc. That is, the output control unit 114 may, for example, control the communication unit 131 to transmit the automatically generated structured information of the character string of the second image to an external device (not shown). The external device here may be, for example, a server, a user terminal, or the like. The output control unit 114 may, for example, control the storage unit 132 to store the automatically generated structured information of the character string of the second image. The output control unit 114 may, for example, control the display unit 133 to display the automatically generated structured information of the character string of the second image.

[0043] The generation AI unit 115 has an AI that generates content such as text, graphics, audio, still images, and videos. The AI ​​here may be, for example, a generation AI. The generation AI may include, for example, a Retrieval Augmented Generation (RAG) model or a large-scale language model (LLM) provided on various cloud services such as Microsoft Azure (registered trademark), AWS (registered trademark), and GCP (registered trademark). The generation AI unit 115 may refer to the structured information of the character string recorded in the second image output by the output control unit 114 to generate the content. Furthermore, for example, when one of the candidate images presented by the presentation unit 113 is selected, the generation AI unit 115 may generate content by referring to the structured information of the character string when the character string of the second image is structured (when structured information of the character string to be stored in the second image is generated) in response to output of structured information corresponding to the selected candidate image by the output control unit. As an example, when an instruction (prompt) to generate content is input, the generation AI unit 115 may refer to the structured information of the character string recorded in the second image and generate content such as text, figures, audio, still images, and videos in accordance with the instruction (prompt).

[0044] [Information processing method] Next, an information processing method according to an embodiment will be described. FIG. 6 is a flowchart illustrating an information processing method according to an embodiment.

[0045] In step ST101, the storage unit 132 stores knowledge data that associates a first image with structured information of a character string recorded in the first image. The storage unit 132 may store, as the knowledge data, a plurality of pieces of data (correspondence information) that associates the first image, the distributed representation of the first image, and structured information of a character string recorded in the first image.

[0046] In step ST102, the acquiring unit 111 acquires a second image specified by the user. The acquiring unit 111 may acquire, as the second image, an image recording at least one of a graph, a box diagram, a tree diagram, a table, and a flowchart.

[0047] In step ST103, the estimation unit 112 estimates the similarity between the second image acquired in step ST102 and the first image based on the knowledge data stored in step ST101. The estimation unit 112 may, for example, acquire a distributed representation of the second image acquired in step ST102, and estimate the similarity between the second image and the first image based on the distributed representation of the second image and the distributed representation of the first image stored in step ST101.

[0048] In step ST104, the presentation unit 113 presents a predetermined number of first images to the user as candidate images in descending order of the similarity estimated in step ST103.

[0049] In step ST105, when one of the candidate images presented in step ST104 is selected, the output control section 114 controls so as to output the structured information corresponding to the selected candidate image. The output control unit 114 may control the generation and output of structural information of the character string to be recorded in the second image acquired in step ST102 based on the structural information of the character string to be recorded in the first image corresponding to the selected candidate image. After the process of step ST105, the generation AI unit 115 may generate content by referring to the structured information of the character string recorded in the second image output in step ST105.

[0050] [Functions and circuits] Next, the functions and circuits of the information processing device 100 will be described. Each unit of the information processing device 100 may be realized as a function of a computer's arithmetic processing unit, etc. That is, the acquisition unit 111, estimation unit 112, presentation unit 113, output control unit 114, and generation AI unit 115 (control unit 110) of the information processing device 100 may be realized as an acquisition function, an estimation function, a presentation function, an output control function, and a generation AI function (control function), respectively, by a computer's arithmetic processing unit, etc. The information processing program can cause a computer to realize each of the above-mentioned functions. The information processing program may be recorded on a computer-readable non-transitory storage medium, such as a memory, a solid-state drive, a hard disk drive, or an optical disk. The storage medium may also be referred to as a non-transitory computer-readable medium that stores the information processing program. The information processing program may also be transmitted online. Furthermore, as described above, each unit of the information processing device 100 may be realized by a computer's arithmetic processing unit or the like. The arithmetic processing unit or the like is configured, for example, by an integrated circuit or the like. Therefore, each unit of the information processing device 100 may be realized as a circuit that constitutes the arithmetic processing unit or the like. That is, the acquisition unit 111, estimation unit 112, presentation unit 113, output control unit 114, and generation AI unit 115 (control unit 110) of the information processing device 100 may be realized as an acquisition circuit, an estimation circuit, a presentation circuit, an output control circuit, and a generation AI circuit (control circuit) that constitute the computer's arithmetic processing unit or the like. The input unit 121, communication unit 131, storage unit 132, and display unit 133 (output unit) of the information processing device 100 may be realized as, for example, an input function including the functions of an arithmetic processing device, and a communication function, storage function, and display function (output function). The input unit 121, communication unit 131, storage unit 132, and display unit 133 (output unit) of the information processing device 100 may be realized as, for example, an input circuit, a communication circuit, a storage circuit, and a display circuit (output circuit) by being configured using integrated circuits, etc. The input unit 121, communication unit 131, storage unit 132, and display unit 133 (output unit) of the information processing device 100 may be realized as, for example, an input device, a communication device, a storage device, and a display device (output device) by being configured using a plurality of devices.

[0051] The information processing device 100 can combine one or any combination of the above-mentioned multiple units. In this disclosure, the term "information" is used, but the term "information" can be replaced with "data" and the term "data" can be replaced with "information."

[0052] [Aspects and Effects of the Present Embodiment] Next, one aspect of this embodiment and the effects of each aspect will be described. Note that each aspect described below is an example at the time of filing, and this embodiment is not limited to the aspects described below. In other words, this embodiment is not limited to the aspects described below, and may be realized by appropriately combining the above-mentioned parts. Furthermore, a lower-level aspect may be able to cite any of the higher-level aspects. The effects of the present embodiment described below are merely examples, and the effects of each aspect are not limited to those described below. Each aspect may, for example, achieve at least one of the effects described below.

[0053] (Aspect 1) An information processing device of one embodiment includes a memory unit that stores knowledge data that associates a first image with structured information of a character string recorded in the first image; an acquisition unit that acquires a second image specified by a user; an estimation unit that estimates the similarity between the second image acquired by the acquisition unit and the first image based on the knowledge data stored in the memory unit; a presentation unit that presents a predetermined number of first images to the user as candidate images in order of highest similarity estimated by the estimation unit; and an output control unit that, when one of the candidate images presented by the presentation unit is selected, controls the output of structured information corresponding to the selected candidate image. This allows the information processing device to present, for example, a data structure of a string (structured information of a string based on a first image) optimized for a generative AI and a search augmentation generation (RAG) model. For example, by presenting the data structure of a string, the information processing device can provide information creation support (support for creating structured information of a string based on a second image) for making a second image (non-public information) used only within an organization such as a company understandable for a generative AI and a search augmentation generation (RAG) model. Even when it is unclear how to structure multiple character strings recorded in a second image, the information processing device can present structured information of a first image (candidate image) that is most similar to the second image. That is, the information processing device can search for a first image similar to the second image and present to the user a template (structured information of character strings based on the first image) in which the character strings described in the first image are formatted. In other words, when transcribing character strings recorded in a second image so that they can be used by a generation AI, the information processing device can present a reference first image and a character string (character string structure) transcribed based on the first image. Furthermore, by presenting structured information for the first image (candidate image) that is most similar to the second image, the information processing device can structure the character string recorded in the second image in the same way as the character string structure (composition information) of the first image (candidate image), making it possible for the meaning to be understood by a generation AI, etc. (to be usable by a generation AI). When the information processing device structures the multiple character strings described in the second image in the same manner as the first image (candidate image) (when it generates structured information), it can store the structured information of the character strings based on the second image as private information in a data storage of information used by the generation AI. The generation AI, search expansion generation model, etc. can refer to the structured information of the character strings based on the second image as private information of the search target stored in the data storage, and answer the character string. Even when a user of an information processing device manually transcribes a character string recorded in a second image into a character string structure that is easy for a generation AI to interpret, the user can easily transcribe the character string (character string structure) recorded in the second image by referring to the character string structure of the first image, which is more similar to the second image. That is, even if the user has relatively little expertise in the generation AI, the user can transcribe the character string recorded in the second image so that it has a character string structure that can be interpreted by the generation AI. In other words, the user can reduce the labor and time, i.e., the cost, required to transcribe the character string recorded in the second image according to the data structure of the character string compared to when the structured information of the first image is not presented.

[0054] (Aspect 2) In the information processing device of one aspect, the acquisition unit may acquire, as the second image, an image recording at least one of a graph, a box diagram, a tree diagram, a table, and a flowchart. This allows the information processing device to present to the user first images (candidate images) that are more similar to the second images, even if the second images are various types of diagrams and tables.

[0055] (Aspect 3) In one embodiment of the information processing device, the memory unit stores, as knowledge data, data that associates a first image, a distributed representation of the first image, and structured information of a character string recorded in the first image, and the estimation unit acquires the distributed representation of the second image acquired by the acquisition unit, and estimates the similarity between the second image and the first image based on the distributed representation of the second image and the distributed representation of the first image stored in the memory unit. This allows the information processing device to estimate the similarity between the first image and the second image by using natural language processing, and to identify the first image that is more similar to the second image.

[0056] (Aspect 4) In one embodiment of the information processing device, the output control unit may be configured to control the output of structural information of a character string recorded in a second image acquired by the acquisition unit in accordance with structural information of a character string recorded in a first image corresponding to a selected candidate image. This allows the information processing device to automatically structure the character string recorded in the second image (generate structured information for the second image). That is, the information processing device can arrange the character string recorded in the second image into a data structure (for example, text such as CSV) that is easy for the generation AI to interpret.

[0057] (Aspect 5) An information processing device of one embodiment includes a memory unit that stores knowledge data that associates a first image with structural information of a character string recorded in the first image; an acquisition unit that acquires a second image specified by a user; an estimation unit that estimates a similarity between the second image acquired by the acquisition unit and the first image based on the knowledge data stored in the memory unit; a presentation unit that presents a predetermined number of first images to the user as candidate images in descending order of the similarity estimated by the estimation unit; an output control unit that, when one of the candidate images presented by the presentation unit is selected, outputs structural information of the character string to be recorded in the second image acquired by the acquisition unit in accordance with structural information of the character string recorded in the first image corresponding to the selected candidate image; and a generation AI unit that generates content by referring to the structural information of the character string output by the output control unit. As a result, the information processing device has, for example, a generation AI and a search expansion generation (RAG) model, and when an instruction (prompt) is input, it can refer to the structured information of the character string and generate content (such as a sentence, for example) according to the instruction (prompt). In other words, the information processing device can provide a generation AI, etc.

[0058] (Aspect 6) In one aspect of the information processing method, a computer having a memory unit that stores knowledge data that associates a first image with structured information of a character string recorded in the first image executes the following steps: an acquisition step of acquiring a second image specified by a user; an estimation step of estimating the similarity between the second image acquired by the acquisition step and the first image based on the knowledge data stored in the memory unit; a presentation step of presenting a predetermined number of first images to the user as candidate images in descending order of the similarity estimated by the estimation step; and an output control step of controlling the output of structured information corresponding to the selected candidate image when one of the candidate images presented by the presentation step is selected. As a result, the information processing method can achieve the same effects as the information processing device of the above-described aspect.

[0059] (Aspect 7) An information processing program of one embodiment implements, in a computer having a memory unit that stores knowledge data that associates a first image with structured information of a character string recorded in the first image, an acquisition function that acquires a second image specified by a user, an estimation function that estimates the similarity between the second image acquired by the acquisition function and the first image based on the knowledge data stored in the memory unit, a presentation function that presents a predetermined number of first images to the user as candidate images in descending order of the similarity estimated by the estimation function, and an output control function that, when one of the candidate images presented by the presentation function is selected, controls the output of structured information corresponding to the selected candidate image. As a result, the information processing program can achieve the same effects as the information processing device of the above-described aspect. [Explanation of symbols]

[0060] 100 Information processing device 110 control section 111 Acquisition Department 112 Estimation Department 113 Presentation section 114 Output control section 115 Generation AI Department 121 Input section 131 Communications Department 132 Storage section 133 Display section

Claims

1. a storage unit that stores knowledge data that associates a first image with structural information of a character string recorded in the first image; an acquisition unit that acquires a second image specified by a user; an estimation unit that estimates a similarity between the second image acquired by the acquisition unit and the first image based on knowledge data stored in the storage unit; a presentation unit that presents a predetermined number of the first images to a user as candidate images in descending order of the similarity estimated by the estimation unit; an output control unit that, when one of the candidate images presented by the presentation unit is selected, controls to output structured information corresponding to the selected candidate image; An information processing device comprising:

2. The acquisition unit acquires, as the second image, an image recording at least one of a graph, a box diagram, a tree diagram, a table, and a flowchart. The information processing device according to claim 1 .

3. the storage unit stores, as knowledge data, data in which the first image, a distributed representation of the first image, and structured information of a character string recorded in the first image are associated with each other; The estimation unit acquires a distributed representation of the second image acquired by the acquisition unit, and estimates a similarity between the second image and the first image based on the distributed representation of the second image and the distributed representation of the first image stored in the storage unit. The information processing device according to claim 1 .

4. The output control unit controls to output structured information of a character string recorded in the second image acquired by the acquisition unit in accordance with structured information of a character string recorded in a first image corresponding to the selected candidate image. The information processing device according to claim 1 .

5. a storage unit that stores knowledge data that associates a first image with structural information of a character string recorded in the first image; an acquisition unit that acquires a second image specified by a user; an estimation unit that estimates a similarity between the second image acquired by the acquisition unit and the first image based on knowledge data stored in the storage unit; a presentation unit that presents a predetermined number of the first images to a user as candidate images in descending order of the similarity estimated by the estimation unit; an output control unit that, when one of the candidate images presented by the presentation unit is selected, outputs structural information of a character string recorded in the second image acquired by the acquisition unit in accordance with structural information of a character string recorded in a first image corresponding to the selected candidate image; a generation AI unit that generates content by referring to the structured information of the character string output by the output control unit; An information processing device comprising:

6. a computer including a storage unit that stores knowledge data that associates a first image with structured information of a character string recorded in the first image, an acquisition step of acquiring a second image specified by a user; an estimation step of estimating a similarity between the second image acquired by the acquisition step and the first image based on knowledge data stored in the storage unit; a presenting step of presenting a predetermined number of the first images to a user as candidate images in descending order of the similarity estimated by the estimating step; an output control step of controlling, when one of the candidate images presented by the presentation step is selected, to output structured information corresponding to the selected candidate image; An information processing method that performs the above.

7. a computer including a storage unit that stores knowledge data that associates a first image with structured information of a character string recorded in the first image; an acquisition function for acquiring a second image specified by a user; an estimation function for estimating a similarity between the second image acquired by the acquisition function and the first image based on knowledge data stored in the storage unit; a presentation function that presents a predetermined number of the first images to a user as candidate images in descending order of the similarity estimated by the estimation function; an output control function that, when one of the candidate images presented by the presentation function is selected, controls to output structured information corresponding to the selected candidate image; An information processing program that makes this possible.

Images