Information processing device, information processing method, and information processing program

JP7588260B1Active Publication Date: 2024-11-21SOFTBANK CORPORATION
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2024035204
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2024-03-07
Publication Date
2024-11-21
Estimated Expiration
2044-03-07

AI Technical Summary

Benefits of technology

【0007】 本開示の情報処理装置、情報処理方法及び情報処理プログラムは、構造化された文字列を出力することができる。

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007588260000001_ABST
    Figure 0007588260000001_ABST
Patent Text Reader

Abstract

An information processing device, an information processing method, and an information processing program for outputting a structured character string are provided. [Solution] The information processing device includes a memory unit that stores knowledge data that corresponds a first image with structured information of a character string recorded in the first image, an acquisition unit that acquires a second image identified by a user, an estimation unit that estimates a similarity between the second image acquired by the acquisition unit and the first image based on the knowledge data stored in the memory unit, a presentation unit that presents a predetermined number of first images to the user as candidate images in order of highest similarity estimated by the estimation unit, and an output control unit that controls the output of structured information corresponding to the selected candidate image when one of the candidate images presented by the presentation unit is selected.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, there is a device that performs OCR on a form to extract character strings and checks the character strings (see Patent Document 1). When checking character strings, the device allows the user to distinguish between different workflow types and presents multiple selection candidates for correcting the character string according to a priority order determined for each workflow type. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-056732 Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, when using a generation AI with a figure (such as a table or graph) that contains text, the generation AI must structure the text according to the structure of the figure (such as the table or graph) in order to use the text in the figure according to the structure of the figure (according to the meaning of the figure). The technology described in Patent Document 1 does not describe structuring of character strings.

[0005] The present disclosure provides an information processing device, an information processing method, and an information processing program that output a structured character string. [Means for solving the problem]

[0006] An information processing device of one embodiment includes a memory unit that stores knowledge data that associates a first image with structured information of a character string recorded in the first image, an acquisition unit that acquires a second image specified by a user, an estimation unit that estimates a similarity between the second image acquired by the acquisition unit and the first image based on the knowledge data stored in the memory unit, a presentation unit that presents a predetermined number of first images to a user as candidate images in order of highest similarity estimated by the estimation unit, and an output control unit that controls the output of structured information corresponding to the selected candidate image when one of the candidate images presented by the presentation unit is selected. Effect of the Invention

[0007] The information processing device, information processing method, and information processing program disclosed herein can output a structured character string. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram for explaining conventional character information obtained by performing OCR on character strings written in a bar graph, in which (A) shows a bar graph on which character strings are written, and (B) shows character information obtained by OCR based on the bar graph shown in (A). [Diagram 2] 1 is a diagram for explaining an example of structured information about a character string written in a first image (bar graph) according to an embodiment of the present invention, in which (A) shows an example of a first image (bar graph) in which a character string is written, and (B) shows an example of structured information about the character string based on the first image described in (A). [Diagram 3] FIG. 11 is a diagram for explaining an example of a second image. [Figure 4] FIG. 11 is a diagram for explaining an example of a candidate image. [Diagram 5] FIG. 1 is a block diagram illustrating an information processing device according to an embodiment. [Figure 6] 1 is a flowchart illustrating an information processing method according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] An embodiment will be described below.

[0010] [Overview of information processing device 100] First, an overview of an information processing device 100 according to an embodiment will be described. Fig. 1 is a diagram for explaining conventional character information obtained by performing OCR on character strings written in a bar graph. Fig. 1(A) shows a bar graph on which character strings are written, and Fig. 1(B) shows character information obtained by OCR based on the bar graph shown in (A). 2 is a diagram for explaining an example of structured information about a character string written in a first image (bar graph) according to the present embodiment. Fig. 2(A) shows an example of a first image (bar graph) in which a character string is written, and Fig. 2(B) shows an example of structured information about a character string based on the first image described in (A).

[0011] The information processing device 100 may be configured as a presentation device or the like that presents a structure of a string (an example of a string structure) based on another figure similar to the structure of the figure (the table and graph, and the position of the string written in the table and the table graph) when a string is written in the figure (for example, a table and a graph). The information processing device 100 may also be configured as a conversion device or the like that automatically converts a string written in a figure into structured information of the string so that it can be used by a generation AI. The information processing device 100 is not limited to the example device described above, and may constitute various devices. The information processing device 100 may be a computer such as a server, a desktop, a laptop, a tablet, or a smartphone.

[0012] As shown in Figure 1(A), when there are two bar graphs, multiple items (character strings) are written in one bar graph (stacked bar graph), and the time progression of the multiple items (character strings) can be seen in a total of two bar graphs (stacked bar graphs). Conventionally, when OCR was performed on such character strings, all the character strings were displayed consecutively, regardless of the time progression of each item (character string) (see Figure 1(B)). Similarly, for example, in a two-row, two-column table, each cell in the table contains a character string. Conventionally, when OCR is performed on an image containing such character strings, all of the character strings in each cell may be displayed consecutively, regardless of the position of each cell in the table. A string of characters that is not structured according to such a diagram cannot be used as a meaningful string by a generative AI.

[0013] Therefore, in this embodiment, the character string described in the figure is structured according to the figure so that the description of the character string can be used by the generation AI. That is, when the target figure (second image) is similar to another figure (first image), the information processing device 100 presents a character string (structured information) that has been previously structured based on the other figure (first image). As an example, when there is a first image (for example, a bar graph (stacked bar graph), etc.) as illustrated in FIG. 2(A), the information processing device 100 presents structured information (see FIG. 2(B)) in which the character string described in the first image is structured. In the conventional character information of Fig. 1(B), multiple character strings are recorded consecutively. On the other hand, the structured information of this embodiment illustrated in Fig. 2(B) is information in which only character strings are structured according to the structure of the first image (bar graph) of Fig. 2(A) (department, fiscal year, numerical values ​​according to the fiscal year, and the positions of increase / decrease rates, etc. (character string structure)). Note that the composition information of the first image and the character strings corresponding to the first image is not limited to the example illustrated in Fig. 2, and may be various. As a result, the user or the information processing device 100 refers to the structuring information of the character string based on the first image, and structures the character string written in the drawing (second image) so that it becomes similar to the structuring information.

[0014] Specifically, the information processing device 100 stores, as knowledge data, a plurality of data (correspondence information) that associates the first image with structuring information of character strings recorded in the first image. The structuring information of character strings may be information that structures the structure of the first image, for example, character strings written according to each item of a cell in a table and a bar graph, so that the structure of the first image can be understood (for example, so that the relationship between the position of each cell in a table and each cell in a bar graph corresponding to each character string can be understood) (see the example in FIG. 2).

[0015] FIG. 3 is a diagram for explaining an example of the second image. FIG. 4 is a diagram for explaining an example of the candidate image.

[0016] The information processing device 100 acquires a second image specified by a user. As an example, the information processing device 100 acquires a second image (bar graph) illustrated in FIG. 3. That is, when a second image to be subjected to structuring of a character string is specified by a user (for example, input via the input unit 121 (see FIG. 5) and a user terminal (not shown)), the information processing device 100 accepts the specified second image. The input unit 121 may be, for example, a keyboard and a mouse. Furthermore, the user terminal is a terminal used by a user, and may be, for example, a desktop, a laptop, a tablet, a smartphone, or the like.

[0017] The information processing device 100 estimates the similarity between the acquired second image and the first image based on the knowledge data. As an example, the information processing device 100 may estimate the similarity between the first image and the second image based on the distributed representation of the first image and the distributed representation of the second image as the estimation of the similarity. As a more specific example, the information processing device 100 may express each image (meaning of the image) of the first image and the second image as a vector (numerical value) as the distributed representation, and estimate the similarity between the vector (numerical value) of the meaning of the first image and the vector (numerical value) of the meaning of the second image (based on the distance and positional relationship between the vectors). For such distributed representation (vector), various processes including, for example, known natural language processing, etc. may be used.

[0018] The information processing device 100 presents a predetermined number of first images as candidate images to the user in order of the degree of similarity estimated as described above. The predetermined number may be various numbers (single or multiple), such as 3, 5, 8, 10, etc. As shown in an example in Fig. 4, the information processing device 100 presents a predetermined number of first images (candidate images) similar to the second image (see Fig. 3) in order of the degree of similarity (in the example shown in Fig. 4, candidate images from "1st place" with the highest degree of similarity to "3rd place" in order).

[0019] When one of the candidate images presented as described above is selected by the user (e.g., input via the input unit 121 and a user terminal), the information processing device 100 refers to the knowledge data and outputs structured information corresponding to the selected candidate image (first image). As an example, when the first candidate image illustrated in Fig. 4 is selected, the information processing device 100 presents structured information (see Fig. 2(B)) of a character string corresponding to the selected candidate image (first image (see Fig. 2(A))). The structured information here is the structured information of the character strings described above, and may be information that structures the structure of the first image, for example, the character strings written corresponding to each item of the cells in a table and a bar graph, so that the structure of the first image can be understood (for example, so that the relationship between the position of each cell in the table and each cell in the bar graph corresponding to each character string can be understood).

[0020] [Details of the information processing device 100] Next, the information processing device 100 according to an embodiment will be described in detail. FIG. 5 is a block diagram illustrating an information processing device 100 according to an embodiment.

[0021] The information processing device 100 includes, for example, an input unit 121, a communication unit 131, a storage unit 132, a display unit 133, and a control unit 110. The communication unit 131, the storage unit 132, and the display unit 133 may be an embodiment of the output unit. The control unit 110 includes, for example, an acquisition unit 111, an estimation unit 112, a presentation unit 113, an output control unit 114, and a generation AI unit 115. The control unit 110 may be configured, for example, by an arithmetic processing device of the information processing device 100. The control unit 110 (for example, an arithmetic processing device) may realize the functions of each unit (for example, the acquisition unit 111, the estimation unit 112, the presentation unit 113, the output control unit 114, and the generation AI unit 115) by, for example, appropriately reading and executing various programs stored in the storage unit 132. That is, the functions of each unit may be realized by computer implementation.

[0022] The input unit 121 may be, for example, an input interface such as a keyboard and a mouse, etc. The input unit 121 may be, for example, a graphical user interface (GUI).

[0023] The communication unit 131 is, for example, a communication interface capable of transmitting and receiving various information to and from a device external to the information processing device 100 (external device).

[0024] The storage unit 132 may store, for example, various information and programs. Examples of the storage unit 132 may be a memory, a solid state drive, a hard disk drive, etc. Note that the storage unit 132 may be, for example, a storage area and a server on a cloud.

[0025] The storage unit 132 stores knowledge data that associates the first image with structured information of a character string recorded in the first image. The storage unit 132 may store, as the knowledge data, a plurality of pieces of data (correspondence information) that associates the first image, the distributed representation of the first image, and structured information of a character string recorded in the first image.

[0026] The first image may be, for example, a number of types of diagrams, including a graph, a box diagram, a tree diagram, a table, a flow chart, etc. The first image may include a character string.

[0027] The structured information of the character string may be, for example, information in which the character strings written according to the structure of the first image, for example, each item such as a square in a table and a bar graph, are structured so that the structure of the first image can be understood (for example, so that the relationship between the positions of each square in the table and each item in a bar graph corresponding to each character string can be understood) (see an example in FIG. 2). The structured information of the character string is information in which each of the multiple character strings recorded in the first image is structured according to the position at which the character string is recorded in the first image, and is information structured by separating the multiple character strings (between character strings) with lines, symbols, spaces, tabs, or the like and writing the character strings over one or more lines. By structuring the character string, the generation AI, etc. can refer to the character strings recorded in the first image as meaningful content as described in the first image.

[0028] The distributed representation of the first image may be, for example, a vector representation of the first image (the meaning of the first image), that is, a vector of the meaning of the first image, etc. The distributed representation (vector) may be obtained by using various processes including, for example, known natural language processing.

[0029] The knowledge data is, for example, information stored about the first image and available to the information processing device 100, ie, information about so-called "knowledge" of the first image that is computer-available.

[0030] The display unit 133 is, for example, a display capable of displaying various characters, symbols, images, and the like.

[0031] The acquisition unit 111 acquires a second image specified by the user. The acquisition unit 111 may acquire, as the second image, an image recording at least one of a graph, a box diagram, a tree diagram, a table, and a flow chart.

[0032] The second image is an image on which a character string is structured so that it can be referenced by a generation AI or the like (see an example in FIG. 3). A character string is recorded in the second image. As shown in an example in Figure 3, when there are two graphs (for fiscal years 2023 and 2024), multiple items (character strings) (departments, values ​​for each department (W1, W2, X1, X2, Y1, Y2, Z1, Z2), and the rate of increase or decrease of each value, etc.) can be listed on each graph, and the temporal trends of multiple items (character strings) can be seen on a total of two graphs. Similarly, as an example, if the second image is a table with multiple rows and multiple columns, a character string is written in each cell of the table.

[0033] The acquisition unit 111 acquires the second image from an external device (not shown), for example, via the communication unit 131. The external device may be, for example, a server, a user terminal, etc. The user terminal is a terminal used by a user of the information processing device 100, and may be, for example, a desktop, a laptop, a tablet, a smartphone, etc. Furthermore, for example, when the second image is recorded in an external memory (not shown) and the external memory is connected to an interface of the information processing device 100, the acquiring unit 111 may acquire the second image from the external memory. In addition, when there is text information (text file) including a second image and a second image is selected (when an area in which the second image is to be written is specified) via the input unit 121 and a user terminal, the acquisition unit 111 may acquire the selected second image (the specified area as the second image).

[0034] The estimation unit 112 estimates a similarity between the second image acquired by the acquisition unit 111 and the first image based on the knowledge data stored in the storage unit 132. The estimation unit 112 may, for example, acquire a distributed representation of the second image acquired by the acquisition unit 111, and estimate a similarity between the second image and the first image based on the distributed representation of the second image and the distributed representation of the first image stored in the storage unit 132. First, the estimation unit 112 acquires the distributed representation of the second image acquired by the acquisition unit 111. As an example, the estimation unit 112 may acquire the distributed representation of the second image by using various processes including known natural language processing. The distributed representation of the second image may be, for example, a vector representation of the second image (the meaning of the second image), that is, a vector of the meaning of the second image. Next, the estimation unit 112 estimates the similarity between the first image and the second image, for example, based on the distributed representation of the first image (vector of the meaning of the first image) and the distributed representation of the second image (vector of the meaning of the second image), i.e., based on the relationship between the distance and position of the vectors of the first image and the second image.

[0035] The presentation unit 113 presents a predetermined number of first images to the user as candidate images in descending order of similarity estimated by the estimation unit 112 (see an example in FIG. 4). The presentation unit 113 identifies a first image that is more similar to the second image estimated by the estimation unit 112. That is, the presentation unit 113 identifies a plurality of first images in descending order of similarity to the second image estimated by the estimation unit 112. The presentation unit 113 presents a predetermined number of first images to the user as candidate images in descending order of similarity to the second image. The predetermined number may be various numbers (single or multiple), such as, for example, 3, 5, 8, and 10. In the case of the example shown in FIG. 4, the presentation unit 113 presents the first to third first images (candidate images) having the highest similarity to the second image (see FIG. 3).

[0036] The presentation unit 113 may, for example, control the display unit 133 to display one or more candidate images as presentation of the candidate images to the user. Furthermore, the presentation unit 113 may control the communication unit 131 to transmit information on one or more candidate images (candidate image information) to an external device (not shown) to present the candidate images to the user. The external device here may be, for example, a user terminal or the like.

[0037] When one of the candidate images presented by the presentation unit 113 is selected, the output control unit 114 controls to output structured information corresponding to the selected candidate image. That is, when one of the candidate images presented by the presentation unit 113 is selected by a user (for example, input via the input unit 121 and a user terminal), the output control unit 114 refers to the knowledge data stored in the storage unit 132 and outputs structured information corresponding to the selected candidate image (first image). When the candidate images (first images) illustrated in FIG. 4 are presented to the user and the first candidate image (first image (see FIG. 2(A))) is selected, the output control unit 114 outputs structured information (see FIG. 2(B)) of the character string corresponding to the candidate image (first image (see FIG. 2(A))).

[0038] The output control unit 114 may, for example, control the display unit 133 to display structuring information of a character string. The output control unit 114 may, for example, control the communication unit 131 to transmit the structuring information of the character string to an external device (not shown). The external device here may be, for example, a user terminal or the like.

[0039] The structured information here is information that has been structured so that the character string recorded in the selected candidate image (first image) can be read meaningfully by a generation AI, etc. In other words, as shown in an example in Figure 2, when there are two graphs (Q1 FY2022 and Q1 FY2023), the values ​​for each period of multiple departments (Department A, Department B, and Department C) are shown, i.e., in Q1 FY2022, Department A has 150 billion, Department B has 200 billion, and Department C has 100 billion, and in Q1 FY2023, Department A has 200 billion, Department B has 250 billion, and Department C has 150 billion, and the rate of increase or decrease of the values ​​for each department can be read (rather than a simple list of strings as in the conventional case in Figure 1 (B)). The structured information is information in which each string is structured according to the structure (content) of the two graphs so that the rate of increase or decrease of the values ​​for each department can be read (rather than a simple list of strings as in the conventional case in Figure 1 (B)). Similarly, as an example, the structured information is information of a string of characters structured so that, when the candidate image (first image) is a table with two rows and two columns, the positional relationship of each of the upper left, lower left, upper right, and lower right squares of the table can be determined.

[0040] This enables the information processing device 100 to present structured information (see FIG. 2(B)) of the first image (candidate image) that is most similar to the second image even when it is unclear how to structure a plurality of character strings recorded in the second image (see FIG. 3). Furthermore, by presenting structured information of the first image (candidate image) that is most similar to the second image, the information processing device 100 can structure the character strings recorded in the second image in the same manner as the structure (composition information) of the character strings in the first image (candidate image), so that the meaning can be understood by a generation AI or the like (can be used by a generation AI or the like).

[0041] When structuring the character string recorded in the second image (see FIG. 3) by referring to the composition information (see FIG. 2(B)) of the first image described above, the output control unit 114 may automatically structure the character string recorded in the second image. That is, the output control unit 114 may control to output the structure information of the character string recorded in the second image acquired by the acquisition unit 111 according to the structure information of the character string recorded in the first image corresponding to the selected candidate image. The output control unit 114 automatically structures the character string recorded in the second image (generates the structure information of the second image) by arranging each of the multiple character strings recorded in the second image and separating the multiple character strings (between character strings) with a line, a symbol, a space, a tab, or the like, in the same manner as the structure (structure information) of the character string in the selected candidate image (first image).

[0042] The output control unit 114 may control the output unit to output the automatically generated structured information of the character string of the second image. The output unit may be, for example, a communication unit 131, a storage unit 132, a display unit 133, etc. That is, the output control unit 114 may control the communication unit 131 to transmit the automatically generated structured information of the character string of the second image to an external device (not shown). The external device here may be, for example, a server, a user terminal, or the like. The output control unit 114 may, for example, control the storage unit 132 to store the automatically generated structuring information of the character string of the second image. The output control unit 114 may, for example, control the display unit 133 to display the automatically generated structured information of the character string of the second image.

[0043] The generation AI unit 115 has an AI that generates content such as text, figures, audio, still images, and videos. The AI ​​here may be, for example, a generation AI. The generation AI may include, for example, a Retrieval Augmented Generation (RAG) model, a large-scale language model (LLM) provided on various cloud services such as Microsoft Azure (registered trademark), AWS (registered trademark), and GCP (registered trademark), and the like. The generation AI unit 115 may refer to the structural information of the character string recorded in the second image output by the output control unit 114 to generate the content. Furthermore, for example, when one of the candidate images presented by the presentation unit 113 is selected, the generation AI unit 115 may output structure information corresponding to the selected candidate image by the output control unit, and when the character string in the second image is structured (when structure information of the character string to be stored in the second image is generated), generate content by referring to the structure information of the character string. As an example, when an instruction (prompt) for generating content is input, the generation AI unit 115 may refer to structuring information of the character string recorded in the second image and generate content such as text, figures, audio, still images, and videos in accordance with the instruction (prompt).

[0044] [Information processing method] Next, an information processing method according to an embodiment will be described. FIG. 6 is a flowchart illustrating an information processing method according to an embodiment.

[0045] In step ST101, the storage unit 132 stores knowledge data in which a first image is associated with structured information of a character string recorded in the first image. The storage unit 132 may store, as the knowledge data, a plurality of pieces of data (correspondence information) in which the first image, the distributed representation of the first image, and structured information of a character string recorded in the first image are associated with each other.

[0046] In step ST102, the acquisition unit 111 acquires a second image specified by the user. The acquisition unit 111 may acquire, as the second image, an image recording at least one of a graph, a box diagram, a tree diagram, a table, and a flowchart.

[0047] In step ST103, the estimation unit 112 estimates the similarity between the second image acquired in step ST102 and the first image based on the knowledge data stored in step ST101. The estimation unit 112 may, for example, acquire a distributed representation of the second image acquired in step ST102, and estimate the similarity between the second image and the first image based on the distributed representation of the second image and the distributed representation of the first image stored in step ST101.

[0048] In step ST104, the presentation unit 113 presents a predetermined number of first images to the user as candidate images in descending order of the similarity estimated in step ST103.

[0049] In step ST105, when one of the candidate images presented in step ST104 is selected, the output control section 114 performs control so as to output the structured information corresponding to the selected candidate image. The output control unit 114 may control the generation and output of structuring information of the character string recorded in the second image acquired in step ST102 based on the structuring information of the character string recorded in the first image corresponding to the selected candidate image. After the process of step ST105, the generation AI unit 115 may generate content by referring to the structured information of the character string recorded in the second image output in step ST105.

[0050] [Functions and circuits] Next, the functions and circuits of the above-mentioned information processing device 100 will be described. Each unit of the information processing device 100 may be realized as a function of a computer's arithmetic processing device, etc. That is, the acquisition unit 111, the estimation unit 112, the presentation unit 113, the output control unit 114, and the generation AI unit 115 (control unit 110) of the information processing device 100 may be realized as an acquisition function, an estimation function, a presentation function, an output control function, and a generation AI function (control function) by the computer's arithmetic processing device, etc. The information processing program can cause a computer to realize each of the above-mentioned functions. The information processing program may be recorded in a non-transitory computer-readable storage medium, such as a memory, a solid state drive, a hard disk drive, or an optical disk. The storage medium may be rephrased as a non-transitory computer-readable medium that stores the information processing program. The information processing program may also be transmitted online. As described above, each unit of the information processing device 100 may be realized by a computer arithmetic processing device or the like. The arithmetic processing device or the like is configured by, for example, an integrated circuit or the like. Therefore, each unit of the information processing device 100 may be realized as a circuit constituting the arithmetic processing device or the like. That is, the acquisition unit 111, the estimation unit 112, the presentation unit 113, the output control unit 114, and the generation AI unit 115 (control unit 110) of the information processing device 100 may be realized as an acquisition circuit, an estimation circuit, a presentation circuit, an output control circuit, and a generation AI circuit (control circuit) constituting the arithmetic processing device or the like of a computer. The input unit 121, the communication unit 131, the storage unit 132, and the display unit 133 (output unit) of the information processing device 100 may be realized as, for example, an input function including the functions of an arithmetic processing device, and a communication function, a storage function, and a display function (output function). The input unit 121, the communication unit 131, the storage unit 132, and the display unit 133 (output unit) of the information processing device 100 may be realized as, for example, an input circuit, a communication circuit, a storage circuit, and a display circuit (output circuit) by being configured by an integrated circuit, etc. The input unit 121, the communication unit 131, the storage unit 132, and the display unit 133 (output unit) of the information processing device 100 may be realized as, for example, an input device, a communication device, a storage device, and a display device (output device) by being configured by a plurality of devices.

[0051] The information processing device 100 can combine one or any two or more of the above-mentioned units. In this disclosure, the term "information" is used, but the term "information" can be replaced with "data" and the term "data" can be replaced with "information."

[0052] [Aspects and Effects of the Present Embodiment] Next, one aspect of this embodiment and the effects of each aspect will be described. Note that each aspect described below is an example at the time of filing, and this embodiment is not limited to the aspects described below. In other words, this embodiment is not limited to each aspect described below, and may be realized by appropriately combining each of the above-mentioned parts. In addition, a lower aspect may be able to cite any of the higher aspects. The effects of the present embodiment described below are merely examples, and the effects of each aspect are not limited to those described below. Each aspect may, for example, have at least one of the effects described below.

[0053] (Aspect 1) An information processing device of one embodiment includes a memory unit that stores knowledge data that corresponds a first image with structured information of a character string recorded in the first image, an acquisition unit that acquires a second image specified by a user, an estimation unit that estimates a similarity between the second image acquired by the acquisition unit and the first image based on the knowledge data stored in the memory unit, a presentation unit that presents a predetermined number of first images to a user as candidate images in order of highest similarity estimated by the estimation unit, and an output control unit that controls the output of structured information corresponding to the selected candidate image when one of the candidate images presented by the presentation unit is selected. This allows the information processing device to present, for example, a data structure of a character string (structured information of a character string based on a first image) optimized for a generative AI and a search augmentation generation (RAG) model. For example, by presenting the data structure of a character string, the information processing device can provide information creation support (support for creating structured information of a character string based on a second image) for making a generative AI and a search augmentation generation (RAG) model understand a second image (private information) used only within an organization such as a company. Even when it is unclear how to structure a plurality of character strings recorded in a second image, the information processing device can present structured information of a first image (candidate image) that is most similar to the second image. That is, the information processing device can search for a first image similar to the second image and present to the user a template (structured information of a character string based on the first image) in which the character string described in the first image is shaped. In other words, when transcribing a character string recorded in a second image so that it can be used by a generation AI, the information processing device can present a first image that serves as a reference and a character string (structure of a character string) transcribed based on the first image. Furthermore, by presenting structured information of a first image (candidate image) that is most similar to the second image, the information processing device can structure the character string recorded in the second image in a similar manner to the character string structure (composition information) of the first image (candidate image), making it possible for a generation AI, etc. to understand the meaning (to be usable by the generation AI). When the information processing device structures the multiple character strings described in the second image in the same manner as the first image (candidate image) (when generating structured information), the information processing device can store the structured information of the character strings based on the second image as private information in a data storage of information used by the generation AI. The generation AI and the search extension generation model, etc. can refer to the structured information of the character strings based on the second image as private information of the search target stored in the data storage, and answer the character string. Even when a user of an information processing device manually transcribes a character string recorded in a second image into a character string structure that is easy for a generation AI to interpret, the user can easily transcribe the character string (character string structure) recorded in the second image by referring to the character string structure of the first image that is more similar to the second image. That is, even if the user has a relatively low level of expertise in the generation AI, the user can transcribe the character string recorded in the second image into a character string structure that can be interpreted by the generation AI. In other words, the user can reduce the labor and time, i.e., the cost, required to transcribe the character string recorded in the second image according to the data structure of the character string, compared to the case where the structured information of the first image is not presented.

[0054] (Aspect 2) In the information processing device of one aspect, the acquisition unit may acquire, as the second image, an image recording at least one of a graph, a box diagram, a tree diagram, a table, and a flowchart. This allows the information processing device to present to the user first images (candidate images) that are more similar to the second images, even if the second images are various types of figures and tables.

[0055] (Aspect 3) In one embodiment of the information processing device, the memory unit stores, as knowledge data, data that associates a first image, a distributed representation of the first image, and structured information of a character string recorded in the first image, and the estimation unit acquires the distributed representation of the second image acquired by the acquisition unit, and estimates the similarity between the second image and the first image based on the distributed representation of the second image and the distributed representation of the first image stored in the memory unit. This enables the information processing device to estimate the similarity between the first image and the second image by using natural language processing, and to identify the first image that is more similar to the second image.

[0056] (Aspect 4) In one embodiment of the information processing device, the output control unit may control the output of structuring information of a character string recorded in a second image acquired by the acquisition unit in accordance with structuring information of a character string recorded in a first image corresponding to the selected candidate image. This allows the information processing device to automatically structure the character string recorded in the second image (generate structured information for the second image). That is, the information processing device can arrange the character string recorded in the second image into a data structure (for example, text such as CSV) that is easy for the generation AI to interpret.

[0057] (Aspect 5) An information processing device of one embodiment includes a memory unit that stores knowledge data that associates a first image with character string structuring information recorded in the first image, an acquisition unit that acquires a second image specified by a user, an estimation unit that estimates a similarity between the second image acquired by the acquisition unit and the first image based on the knowledge data stored in the memory unit, a presentation unit that presents a predetermined number of first images to a user as candidate images in order of highest similarity estimated by the estimation unit, an output control unit that, when one of the candidate images presented by the presentation unit is selected, outputs character string structuring information recorded in the second image acquired by the acquisition unit in accordance with character string structuring information recorded in the first image corresponding to the selected candidate image, and a generation AI unit that generates content by referring to the character string structuring information output by the output control unit. As a result, the information processing device has, for example, a generation AI and a search expansion generation (RAG) model, and when an instruction (prompt) is input, it can generate content (such as a sentence, for example) according to the instruction (prompt) by referring to the structured information of the character string. That is, the information processing device can provide a generation AI, etc.

[0058] (Aspect 6) In one aspect of the information processing method, a computer having a memory unit that stores knowledge data that associates a first image with structured information of a character string recorded in the first image executes the following steps: an acquisition step of acquiring a second image specified by a user; an estimation step of estimating a similarity between the second image acquired by the acquisition step and the first image based on the knowledge data stored in the memory unit; a presentation step of presenting a predetermined number of first images to the user as candidate images in order of highest similarity estimated by the estimation step; and an output control step of controlling the output of structured information corresponding to the selected candidate image when one of the candidate images presented by the presentation step is selected. As a result, the information processing method can achieve the same effects as the information processing device according to the above-described aspect.

[0059] (Aspect 7) An information processing program of one embodiment implements, in a computer having a memory unit that stores knowledge data that associates a first image with structured information of a character string recorded in the first image, an acquisition function that acquires a second image specified by a user, an estimation function that estimates a similarity between the second image acquired by the acquisition function and the first image based on the knowledge data stored in the memory unit, a presentation function that presents a predetermined number of first images to a user as candidate images in order of highest similarity estimated by the estimation function, and an output control function that, when one of the candidate images presented by the presentation function is selected, controls the output of structured information corresponding to the selected candidate image. As a result, the information processing program can achieve the same effects as the information processing device according to the above-described aspect. [Explanation of symbols]

[0060] 100 Information processing device 110 Control section 111 Acquisition Department 112 Estimation Department 113 Presentation section 114 Output control section 115 Generation AI Department 121 Input section 131 Communications Department 132 Storage section 133 Display section

Claims

1. a storage unit that stores knowledge data in which a first image is associated with structured information of a character string recorded in the first image; an acquisition unit that acquires a second image specified by a user; an estimation unit that estimates a similarity between the second image acquired by the acquisition unit and the first image based on knowledge data stored in the storage unit; a presentation unit that presents a predetermined number of the first images as candidate images to a user in order of the degree of similarity estimated by the estimation unit; an output control unit that, when one of the candidate images presented by the presentation unit is selected, controls so as to output structured information corresponding to the selected candidate image; An information processing device comprising:

2. The acquisition unit acquires, as the second image, an image recording at least one of a graph, a box diagram, a tree diagram, a table, and a flowchart. The information processing device according to claim 1 .

3. the storage unit stores, as knowledge data, data in which the first image, a distributed representation of the first image, and structured information of a character string recorded in the first image are associated with each other; The estimation unit acquires a distributed representation of the second image acquired by the acquisition unit, and estimates a similarity between the second image and the first image based on the distributed representation of the second image and the distributed representation of the first image stored in the storage unit. The information processing device according to claim 1 .

4. The output control unit controls to output structure information of a character string recorded in the second image acquired by the acquisition unit in accordance with structure information of a character string recorded in a first image corresponding to the selected candidate image. The information processing device according to claim 1 .

5. a storage unit that stores knowledge data in which a first image is associated with structured information of a character string recorded in the first image; an acquisition unit that acquires a second image specified by a user; an estimation unit that estimates a similarity between the second image acquired by the acquisition unit and the first image based on knowledge data stored in the storage unit; a presentation unit that presents a predetermined number of the first images as candidate images to a user in order of the degree of similarity estimated by the estimation unit; an output control unit that, when one of the candidate images presented by the presentation unit is selected, outputs structured information of a character string recorded in the second image acquired by the acquisition unit in accordance with structured information of a character string recorded in a first image corresponding to the selected candidate image; a generation AI unit that generates content by referring to the structured information of the character string output by the output control unit; An information processing device comprising:

6. a computer including a storage unit that stores knowledge data in which a first image is associated with structured information of a character string recorded in the first image, acquiring a second image specified by a user; an estimation step of estimating a similarity between the second image acquired by the acquisition step and the first image based on knowledge data stored in the storage unit; a presenting step of presenting a predetermined number of the first images as candidate images to a user in order of the degree of similarity estimated by the estimating step; an output control step of controlling, when one of the candidate images presented by the presentation step is selected, to output structured information corresponding to the selected candidate image; An information processing method for performing the above.

7. A computer including a storage unit that stores knowledge data in which a first image is associated with structured information of a character string recorded in the first image, an acquisition function for acquiring a second image specified by a user; an estimation function for estimating a similarity between the second image acquired by the acquisition function and the first image based on knowledge data stored in the storage unit; a presentation function of presenting a predetermined number of the first images as candidate images to a user in order of the degree of similarity estimated by the estimation function; an output control function that, when one of the candidate images presented by the presentation function is selected, controls to output structured information corresponding to the selected candidate image; An information processing program that realizes this.

Citation Information

Patent Citations

  • Form document registration method and program

    JP2022128766A

  • Data processing system, data processing method, and program

    JP2021056732A