Information processing device, information processing method, and computer program

Abstract

To provide a technique which allows for automatically generating information indicating the presence of some relevance between a figure and a character string in an image.SOLUTION: An information processing device provided herein comprises a controller configured to acquire an image containing a figure and a character string, and generate relevance information indicative of relevance between the figure and the character string on the basis of a positional relationship between the figure and the character string in the image.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a computer program. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there is known a technique for identifying an object contained in an image and recognizing the position of the object in the image (see, for example, Patent Document 1).

[0003] Also known is an OCR (Optical Character Recognition) technology that analyzes an image representing a character string that has been generated by, for example, reading the image into a scanner, and converts the image into character string data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2013-114596 A Summary of the Invention [Problem to be solved by the invention]

[0005] When an analog drawing (referring to a drawing printed on paper or a drawing drawn by hand, etc.) containing figures and character strings is created, the figures and character strings may be related to each other. For example, when a system design drawing contains figures showing components of the system and character strings explaining the figures, the figures and character strings can be said to be related to each other. When such an analog drawing is converted into an image by a scanner and used on a computer, it is useful to be able to use information showing the relationship between the figures and character strings from the viewpoints of reusing the drawings by searching them and reducing the data of the drawings.

[0006] However, while conventional configurations were capable of recognizing specific objects contained in an image and converting character strings into character string data, they were unable to identify whether there was any correlation between the figures contained in the image and the character strings.

[0007] The present disclosure has been made in consideration of the above-mentioned points, and aims to provide a technology capable of automatically generating information indicating that a graphic and a character string included in an image are associated with each other. [Means for solving the problem]

[0008] According to some embodiments, an information processing device includes a control unit that acquires an image including a figure and a character string, and generates association information indicating an association between the figure and the character string based on a positional relationship between the figure and the character string in the image. This makes it possible to automatically generate information indicating an association between the figure and the character string.

[0009] In an information processing device according to an embodiment, the control unit may generate the association information based on at least one of a distance between the graphic and the character string and a direction of the character string relative to the graphic, as the positional relationship, thereby making it possible to generate information that accurately indicates the association between the graphic and the character string.

[0010] In an information processing device according to an embodiment, the control unit may obtain a prediction model of relevance that is machine-learned using training data including a position of a sample figure as a sample, a position of a sample string as a sample, and a relevance between the sample figure and the sample string, and when generating the relevance information, may use the prediction model of relevance to predict the relevance between the figure and the string based on the positional relationship between the figure and the string. This makes it possible to generate information indicating the relevance between the figure and the string without manually setting a criterion for whether or not the figure and the string are relevant in advance.

[0011] In one embodiment of the information processing device, the control unit may detect a predetermined type of figure from the image, and generate the association information based on a positional relationship between the detected figure and the character string. This makes it possible to generate association information by distinguishing between types of figures.

[0012] In an information processing device according to an embodiment, the control unit may obtain a prediction model for figure detection that is machine-learned using the type and a sample image showing a figure of the type as training data, and use the prediction model for figure detection when detecting the type of figure from the image. This makes it possible to distinguish between types of figures and generate association information without manually setting criteria for distinguishing between types of figures in advance.

[0013] In one embodiment of the information processing device, the control unit may analyze the image to obtain a character string contained in the image, and generate the association information based on a positional relationship between the detected figure and the obtained character string. This makes it possible to automatically obtain a character string from an image and generate association information.

[0014] In an information processing device according to an embodiment, the control unit may acquire an image including a plurality of figures and a plurality of character strings as the image, and generate, for each of the plurality of figures, information indicating a character string included in the plurality of character strings that is associated with the figure as the association information. In this way, even when a plurality of figures and a plurality of character strings exist in an image, it is possible to identify a character string that is associated with each figure and generate association information.

[0015] In an information processing method according to some embodiments, a control unit acquires an image including a figure and a character string, and generates association information indicating an association between the figure and the character string based on a positional relationship between the figure and the character string in the image. This makes it possible to automatically generate information indicating an association between the figure and the character string.

[0016] According to some embodiments, a computer program causes a computer to execute a process of acquiring an image including a figure and a character string, and a process of generating association information indicating an association between the figure and the character string based on a positional relationship between the figure and the character string in the image, thereby making it possible to automatically generate information indicating an association between the figure and the character string. Effect of the Invention

[0017] According to the present disclosure, a technique is provided that can automatically generate information indicating that a graphic and a character string included in an image have some kind of association. [Brief description of the drawings]

[0018] [Figure 1] 10A to 10C are diagrams illustrating a procedure for associating a graphic with a character string from an image of a drawing including the graphic and the character string by a device according to a comparative example. [Diagram 2] 10A and 10B are diagrams illustrating a procedure for detecting an object by a device according to a comparative example. [Diagram 3] 1 is a block diagram showing a configuration of an information processing device according to an embodiment of the present disclosure. [Figure 4] 10 is a flowchart illustrating an operation of an information processing device according to an embodiment of the present disclosure. [Diagram 5] FIG. 1 is a diagram showing an example of a handwritten drawing. [Figure 6] FIG. 6 is a diagram showing an example of data in which object and label information is linked to the drawing in FIG. 5. [Figure 7] FIG. 13 is a diagram showing an example in which a predictive model for figure detection is generated. [Figure 8] FIG. 11 is a diagram illustrating an example of an object detection process performed by an information processing device according to an embodiment of the present disclosure. [Figure 9] 11 is a flowchart illustrating a relevance acquisition process performed by an information processing device according to an embodiment of the present disclosure. [Figure 10] FIG. 11 is a diagram illustrating an example of a relationship acquisition process performed by an information processing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] (Comparative Example) Fig. 1 is a diagram showing a schematic diagram of a procedure for associating figures and character strings from an image of an analog drawing including the figures and character strings by a device according to a comparative example. Fig. 1 shows an example of processing a drawing on which a piping diagram is handwritten by a device such as a PC. PC is an abbreviation for Personal Computer.

[0020] The PC scans the drawing and obtains a scanned image (step S81). In step S81, first, a scanner connected to the PC photographs the drawing. The PC converts the photographed image into bitmap data and corrects the bitmap data according to the state of the image. Specifically, the PC corrects the tilt, distortion, and parts showing through from the back side of the image. The PC stores the corrected image in memory as bitmap data. Hereinafter, the image stored in memory is referred to as a "scanned image."

[0021] Next, the PC extracts each of the predetermined figures from the scanned image as pixel data (step S82). At that time, the PC acquires the position of each of the extracted figures on the scanned image. The PC may extract each figure using, for example, the method described in Patent Document 1.

[0022] Next, the PC recognizes each character string depicted in the scanned image and acquires character string data (step S83). At that time, the PC acquires the position of each extracted character string on the scanned image. The PC may recognize the character string by OCR.

[0023] Next, the user operates the PC to associate each figure extracted from the scanned image with each character string drawn in the scanned image (step S84). FIG. 2 is a diagram that shows a schematic diagram of a procedure for associating figures and character strings by a device according to a comparative example. That is, the user visually checks the scanned image 91 on a monitor connected to the PC, and compares and associates each figure and each character string in the scanned image 91 (92). In the example of FIG. 2, the user inputs the result of the comparison to the PC. In the scanned image 91, there is a handwritten figure showing a valve, and a handwritten character string "for valve" near the figure. Therefore, the user performs an operation to associate the figure showing the valve with the character string "for valve". In response to this, the PC generates information 93 that shows the association between the type and position of the figure and the character string associated with the figure. In the example of FIG. 2, in the information 93, the type of the figure is shown by "object name: valve". The position of the figure in the scanned image is shown by "position: (200, 200)". The character string associated with the figure is indicated by “Label: for valve.” Hereinafter, the character string that is written next to a figure and that explains the figure will be referred to as a “label.”

[0024] In the comparative example, in order to obtain information indicating the association between figures and character strings, the user had to visually check the scanned image and perform an operation to match and associate each figure in the scanned image with each character string. Therefore, in the comparative example, manual work by the user was required, and for example, when there were a large number of drawings to be processed, the processing required a huge amount of work time. In addition, since the comparative example required manual work by the user, there was a risk of human error being introduced when associating each figure in the scanned image with each character string.

[0025] (Embodiments of the present disclosure) Hereinafter, the configuration and operation of an information processing device according to an embodiment of the present disclosure will be mainly described with reference to the accompanying drawings. Fig. 3 is a block diagram showing the configuration of an information processing device 10 according to an embodiment of the present disclosure.

[0026] The information processing device 10 generates association information indicating the association between a figure and a character string based on the positional relationship between the figure and the character string included in an image. Therefore, the information processing device 10 according to an embodiment of the present disclosure can automatically generate information indicating the association between a figure and a character string without requiring human intervention. Note that in this embodiment, the character string includes not only a character string consisting of multiple characters, but also a character string consisting of one character.

[0027] The information processing device 10 is one or more server devices capable of communicating with each other. The information processing device 10 is not limited to these, and may be any general-purpose electronic device such as a PC, or may be another dedicated electronic device. As shown in FIG. 3, the information processing device 10 includes a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, and an output unit 15.

[0028] The control unit 11 includes one or more processors. In one embodiment, the "processor" is a general-purpose processor or a dedicated processor specialized for a specific process, but is not limited thereto. The control unit 11 is communicably connected to each component of the information processing device 10, and controls the operation of the information processing device 10 as a whole.

[0029] The storage unit 12 includes any storage module including an HDD, an SSD, an EEPROM, a ROM, and a RAM. The storage unit 12 may function as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 12 stores any information used in the operation of the information processing device 10. For example, the storage unit 12 may store a system program, an application program, and various information received by the communication unit 13. The storage unit 12 is not limited to one built into the information processing device 10, and may be an external database or an external storage module connected by a digital input / output port such as a USB. HDD is an abbreviation for Hard Disk Drive. SSD is an abbreviation for Solid State Drive. EEPROM is an abbreviation for Electrically Erasable Programmable Read-Only Memory. ROM is an abbreviation for Read-Only Memory. RAM is an abbreviation for Random Access Memory. USB is an abbreviation for Universal Serial Bus.

[0030] The communication unit 13 includes any communication module that can communicate with other devices such as a scanner using any communication technology. The communication unit 13 may further include a communication control module for controlling communication with other devices, and a storage module for storing communication data such as identification information required for communication with other devices.

[0031] The input unit 14 includes one or more input interfaces that receive an input operation from a user and obtain input information based on the user operation. For example, the input unit 14 is, but is not limited to, a physical key, a capacitive key, a pointing device, a touch screen that is integrated with the display of the output unit 15, or a microphone that receives voice input.

[0032] The output unit 15 includes one or more output interfaces that output information to the user and notify the user. For example, the output unit 15 is a display that outputs information as an image, or a speaker that outputs information as sound, but is not limited to these. At least one of the input unit 14 and the output unit 15 may be integral with the information processing device 10 or may be provided separately.

[0033] The functions of the information processing device 10 are realized by executing a computer program (program) according to this embodiment on a processor included in the control unit 11. That is, the functions of the information processing device 10 are realized by software. The computer program causes a computer to execute processing of steps included in the operation of the information processing device 10, thereby causing the computer to realize functions corresponding to the processing of the steps. That is, the computer program is a program for causing a computer to function as the information processing device 10 according to this embodiment.

[0034] The computer program may be recorded on a computer-readable recording medium. Examples of the computer-readable recording medium include a magnetic recording device, an optical disk, a magneto-optical recording medium, or a semiconductor memory. The program may be distributed, for example, by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM on which the program is recorded. "DVD" is an abbreviation for Digital Versatile Disc. "CD-ROM" is an abbreviation for Compact Disc Read Only Memory. The program may be distributed by storing the program in the storage of a server and transferring the program from the server to another computer via a network. The program may be provided as a program product.

[0035] A computer temporarily stores a program recorded on a portable recording medium or transferred from a server in a main storage device. Then, the computer reads the program stored in the main storage device with a processor and executes processing according to the read program with the processor. The computer may read the program directly from a portable recording medium and execute processing according to the program. The computer may execute processing according to the received program each time a program is transferred from the server to the computer. Such processing may be executed by a so-called ASP-type service that realizes functions only by issuing execution instructions and obtaining results without transferring a program from the server to the computer. "ASP" is an abbreviation for Application Service Provider. Programs include information used for processing by a computer and equivalent to a program. For example, data that is not a direct command to a computer but has the property of defining computer processing falls under "substance equivalent to a program."

[0036] A part or all of the functions of the information processing device 10 may be realized by a dedicated circuit included in the control unit 11. That is, a part or all of the functions of the information processing device 10 may be realized by hardware. Furthermore, the information processing device 10 may be realized by a single information processing device, or may be realized by cooperation of multiple information processing devices.

[0037] The operation of the information processing device 10 will be described with reference to Fig. 4. The operation of the information processing device 10 described with reference to Fig. 4 corresponds to the information processing method according to the present embodiment, and the operation of each step is executed under the control of the control unit 11.

[0038] 4, the control unit 11 acquires an image of an analog drawing on which a figure and a character string are written. Specifically, the control unit 11 acquires image data of the drawing acquired by a scanner directly from the scanner or from an external storage device that stores image data, via the communication unit 13. Alternatively, the control unit 11 may acquire image data that has been stored in advance in the storage unit 12.

[0039] FIG. 5 is a diagram showing an example of an image including handwritten figures and character strings, which are shown by image data. In image 50 in FIG. 5, six figures 51-56 are written. Character strings 57-60 correspond to the above-mentioned labels. Character string 57, "For tank", is written near figure 51. Character string 58, "For boiler", is written near figure 52. Character string 59, "Water tank", is written near figure 53. Character string 60, "Boiler A", is written near figure 54. No character strings are written near figures 55 and 56.

[0040] As will be described in detail later, control unit 11 generates relevance information indicating the relevance between figures and character strings based on the positional relationship between each of figures 51-56 included in image 50 and character strings 57-60, and generates a structured image by linking the relevance information to image 50. Fig. 6 is a diagram showing an example of a structured image generated based on image 50. In Fig. 6, structured image 65 includes figures 51-56 maintaining the same positional relationship as image 50. Information 61-64 indicating the type and label of the figure is attached to figures 51-54.

[0041] In this embodiment, an example in which the information 61 to 64 is written in XML format will be described, but the format of this information is not limited to XML. For example, this information may be written in CSV or JSON format. XML is an abbreviation for Extensible Markup Language. CSV is an abbreviation for Comma-Separated Values. JSON is an abbreviation for JavaScript Object Notation. In the information 61 to 64, the "figure" tag indicates a figure. The attribute "name" indicates the type of figure. The attribute "value" indicates a character string that corresponds to the content of a label that is associated with the figure. For example, in the information 61, the type of figure 51 is "valve". The character string that corresponds to the content of a label that is associated with figure 51 is "for tank".

[0042] For ease of understanding, in Fig. 6, corresponding information 61-64 is shown near figures 51-54, but when image 65 is displayed on the display of output unit 15, information 61-64 is not displayed. Information 61-64 is used when a computer uses image 65 for searching or the like. However, information 61-64 may be displayed near corresponding figures 51-54 so that a user can confirm the type and label of each figure. Also, in Fig. 6, information on the type and label of figure is not provided for figures 55 and 56 which do not have corresponding labels, but such information may be provided.

[0043] In step S12 of Fig. 4, the control unit 11 converts the image data acquired in step S11 into bitmap data, and corrects the bitmap data according to the state of the image. Specifically, the control unit 11 corrects the inclination, distortion, and parts showing through from the back side of the image. The control unit 11 stores the corrected image data as bitmap data in the memory of the storage unit 12. Hereinafter, the image stored in the memory is referred to as a "scanned image."

[0044] In step S13 of FIG. 4, the control unit 11 detects an object from the scanned image stored in the memory in step S12. Hereinafter, each figure included in the scanned image is abstracted and classified as a predefined figure detection model, and is referred to as an "object." The object is, for example, information indicating the type and position of the figure. The control unit 11 detects a predetermined type of figure from the scanned image to detect the object. Specifically, the control unit 11 acquires a prediction model for figure detection that is machine-learned using the type of figure and a sample image indicating that type of figure as training data, and detects the object using this prediction model. In this way, the control unit 11 performs the object detection process using the prediction model acquired by machine learning, but the object detection process may also be performed based on a criterion set by the user.

[0045] Fig. 7 is a diagram showing an example of performing machine learning using a type of figure and a sample image showing that type of figure as training data to generate a prediction model for figure detection. The prediction model shown in Fig. 7 is generated before the control unit 11 performs a process of generating association information showing an association between a figure and a character string. The prediction model is generated by the information processing device 10, but a prediction model generated by another computer may be used.

[0046] In the example of Fig. 7, a model generation unit 20, which is a functional element realized by a computer program in information processing device 10, generates a prediction model 22. In the example of Fig. 7, a plurality of sample images 23-25 ​​showing a figure of the type "valve" are input to model generation unit 20. Model generation unit 20 learns (21) these sample images 23-25 ​​corresponding to figures of the type "valve" as training data, and generates a prediction model 22 for images showing the figures of "valve".

[0047] The control unit 11 detects objects from the scanned image using the prediction model generated in this manner. FIG. 8 is a diagram showing an example of the object detection process. The control unit 11 performs an object detection process (32) for detecting objects in the scanned image 31 using the above-mentioned prediction model. The control unit 11 outputs the detected objects as information 33 in XML format. In the information 33, the name of the scanned image "Piping Diagram 1" is described by the attribute "name" of the "Drawing" tag. The "figure" tag indicates information on each figure included in the scanned image of "Piping Diagram 1". The attribute "name" in the "figure" tag indicates the name of the type of figure. The attributes "positionX" and "positionY" indicate the XY coordinates corresponding to the position of the figure. For example, for a figure in the scanned image 31 with the handwritten character string "Water Tank" written on it, <figure name=""タンク”" positionx=""400”" positiony=""50” / ">This information indicates that the type of shape is "tank" and the position of the shape is (400,50). Note that the format of the object information is not limited to XML and may be, for example, CSV or JSON.

[0048] In step S14 of FIG. 4, the control unit 11 analyzes the scanned image and acquires character strings contained in the scanned image as labels. This process may be performed using existing OCR technology. The information acquired in step S14 includes, for each character string, character string data indicating the content of the character string and information indicating the position of the character string in the scanned image. Furthermore, using existing morphological analysis technology, the character string may be divided into words or phrases, and the part of speech may be identified for each phrase. By identifying grammatical information of such character strings for each character string and recording it as a label, it becomes possible to perform a more accurate image search using such grammatical information.

[0049] 4, the control unit 11 acquires the association between the object detected in step S13 and the label acquired in step S14. Then, in step S16, the control unit 11 adds the associated label information to the object information by linking them.

[0050] In this way, the control unit 11 generates relevance information indicating the relevance between the figure and the character string based on the positional relationship between the figure and the character string in the image. Specifically, the control unit 11 generates the relevance information based on at least one of the distance between the figure and the character string and the direction of the character string relative to the figure as the positional relationship between the figure and the character string. For example, the control unit 11 may acquire a character string that exists in the vicinity of the figure within a predetermined distance as a character string that has a relevance to the figure. Alternatively, the control unit 11 may acquire a character string that exists in a specific direction as seen from the figure as a character string that has a relevance to the figure. Generally, a character string that has a relevance to a figure is written in the vicinity of the character string of the figure, so that by determining the presence or absence of the relevance based on the positional relationship between the figure and the character string, information indicating that the figure and the character string included in the image have some relevance can be generated automatically with high accuracy.

[0051] In addition, in this embodiment, the control unit 11 acquires in advance a prediction model acquired by machine learning for determining whether or not there is a correlation between such a figure and a character string. That is, the control unit 11 acquires a prediction model for the correlation acquired by machine learning using the positions of the sample figures as samples, the positions of the sample character strings as samples, and the correlation between the sample figures and the sample character strings as teacher data. Furthermore, when generating correlation information, the control unit 11 predicts the correlation between the figure and the character string based on the positional relationship between the figure and the character string using the prediction model for the correlation. This makes it possible to generate information indicating the correlation between the figure and the character string without manually setting in advance a criterion for whether or not there is a correlation between the figure and the character string.

[0052] In this embodiment, when a scanned image contains multiple figures and multiple character strings, the control unit 11 generates, for each of the multiple figures, information indicating which character strings among the multiple character strings are related to the figure as relevance information. Therefore, even when multiple figures and multiple character strings exist in an image, it is possible to identify the character strings related to each figure and generate relevance information.

[0053] 9 is a flowchart showing the procedure of a correlation acquisition process for generating correlation information when a scanned image contains multiple figures. In the correlation acquisition process, the control unit 11 determines whether or not there is a label to be associated with each object related to the figures contained in the scanned image, and if there is, performs a process of adding information about the label to information about the object.

[0054] In step S21 of Fig. 9, the control unit 11 judges whether or not a label having a relationship with the graphic of the object of interest exists. This judgment is made based on the positional relationship between the graphic and the character string, as described above. If it exists (YES in step S21), the control unit 11 performs the process of step S22. If it does not exist (NO in step S21), the control unit 11 performs the process of step S23.

[0055] 9, the control unit 11 adds relevant label information to the information of the object of interest. Then, the control unit 11 performs the process of step S23.

[0056] 9, the control unit 11 determines whether or not an unprocessed object exists in the scanned image. If an unprocessed object exists (YES in step S23), the control unit 11 performs the process of step S21. If an unprocessed object does not exist (NO in step S23), the control unit 11 ends the relevance acquisition process.

[0057] FIG. 10 is a diagram showing an example of the relevance acquisition process. The control unit 11 performs the above-mentioned relevance acquisition process (42) on the object information in the scanned image 41. The control unit 11 outputs the object information 33 to which a label has been added as information 43 in XML format. As is clear from a comparison of the information 33 in FIG. 8 with the information 43 in FIG. 10, the two shapes of type "valve", the shape of type "tank", and the shape of type "boiler" have labels that have relevance, and therefore information corresponding to the labels has been added. For example, " <figure name=""タンク”" positionx=""400”" positiony=""50” / ">" has label information "value="Boiler A"" added as an attribute. This information indicates that the shape of type "Tank" present at position (400,50) is associated with the label "Boiler A". Note that the format of the association information is not limited to XML and may be, for example, CSV or JSON.

[0058] In step S17 of Fig. 4, the control unit 11 links the relevance information generated up to step S16 to the scanned image 41 to generate the structured image described above with reference to Fig. 6. Then, the control unit 11 ends the process.

[0059] As described above, the information processing device 10 generates relevance information indicating the relevance between a figure and a character string based on the positional relationship between the figure and the character string in the scanned image. Specifically, the relevance information is generated based on at least one of the distance between the figure and the character string and the direction of the character string relative to the figure as the positional relationship. Therefore, it is possible to automatically generate information indicating that a figure and a character string included in an image have some kind of relevance without human intervention. With this configuration, it is possible to easily add information about a figure included in a handwritten drawing and its associated character string in a form that can be used by a computer. Therefore, it is possible to expand the use of handwritten drawings in computers, such as searching for handwritten drawings.

[0060] Although the embodiments according to the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can easily make various modifications or corrections based on the present disclosure. Therefore, it should be noted that these modifications or corrections are included in the scope of the present disclosure. For example, the functions included in each component or step can be rearranged so as not to be logically inconsistent, and multiple components or steps can be combined or divided into one. [Explanation of symbols]

[0061] 10. Information processing device 11 Control section 12 Storage section 13. Communications Department 14 Input section 15 Output section< / figure> < / figure>

Claims

1. Acquire an input image including a plurality of figures and character strings, Detecting the plurality of figures and the character string from the input image; for each of the plurality of figures detected from the input image, determining whether or not the character string related to the figure exists based on a positional relationship between the figure and the character string in the input image, and if it is determined that the character string related to the figure exists, generating relationship information for the plurality of figures by associating coordinates indicating a position of the figure and a type of the figure with text information indicating the character string related to the figure; A display image is displayed on a display unit, in which text information indicating the type of the plurality of figures and the character string associated with the figures is arranged near the plurality of figures based on the relevance information. A control unit is provided, The control unit is obtaining a first prediction model of association that is machine-learned using training data including a position of a sample graphic as a sample, a position of a sample character string as a sample, and an association between the sample graphic and the sample character string; A second prediction model for figure detection is obtained by machine learning using a predetermined type of figure and a sample image showing the predetermined type of figure as training data; when generating the relevance information, the first prediction model is used to predict a relevance between the graphic and the character string based on a positional relationship between the graphic and the character string, and the second prediction model is used to detect types of the plurality of graphic elements detected from the input image; the positional relationship between the graphic and the character string includes at least one of a distance between the graphic and the character string and a direction of the character string relative to the graphic; Information processing device.

2. The control unit: As the input image, an image including a plurality of handwritten figures and character strings is acquired; Detecting the handwritten figures and the character string from the input image; for each of the plurality of figures detected from the input image including the plurality of handwritten figures and the character strings, determining whether or not the character string related to the figure exists based on a positional relationship between the figure and the character string in the input image, and if it is determined that the character string related to the figure exists, generating relationship information for the plurality of figures by associating coordinates indicating a position of the figure and a type of the figure with text information indicating the character string related to the figure; An image in which text information indicating the type of the plurality of handwritten figures and the character string associated with the plurality of handwritten figures is arranged near the plurality of handwritten figures based on the relevance information is displayed on the display unit as the display image. The information processing device according to claim 1 .

3. An information processing device as described in claim 1 or 2, wherein the control unit causes the display unit to display a display image in which, for any of the figures included in the input image that does not have a corresponding string, text information indicating only the type of the figure is placed near the figure.

4. The control unit is Detecting a predetermined type of figure from the input image; The relevance information is generated based on a positional relationship between the detected figure and the character string. The information processing device according to claim 1 .

5. The control unit is As the input image, an image including a plurality of figures and a plurality of character strings is obtained, For each of the plurality of figures, information indicating a character string included in the plurality of character strings that is related to the figure is generated as the related information. The information processing device according to claim 1 .

6. The control unit: acquiring an input image including a plurality of figures and character strings; detecting the plurality of figures and the character string from the input image; determining whether or not the character string related to each of the plurality of figures detected from the input image exists based on a positional relationship between the figure and the character string in the input image, and if it is determined that the character string related to the figure exists, generating correlation information for the plurality of figures by associating coordinates indicating a position of the figure and a type of the figure with text information indicating the character string related to the figure; displaying, on a display unit, a display image in which text information indicating a type of the plurality of figures and the character string associated with the figures is arranged near the plurality of figures based on the relevance information; Including, The control unit is obtaining a first prediction model of association that is machine-learned using training data including a position of a sample graphic as a sample, a position of a sample character string as a sample, and an association between the sample graphic and the sample character string; A second prediction model for figure detection is obtained by machine learning using a predetermined type of figure and a sample image showing the predetermined type of figure as training data; when generating the relevance information, the first prediction model is used to predict a relevance between the graphic and the character string based on a positional relationship between the graphic and the character string, and the second prediction model is used to detect types of the plurality of graphic elements detected from the input image; the positional relationship between the graphic and the character string includes at least one of a distance between the graphic and the character string and a direction of the character string relative to the graphic; Information processing methods.

7. A process for acquiring an input image including a plurality of figures and character strings; A process of detecting the plurality of figures and the character string from the input image; a process of determining whether or not a character string related to each of the plurality of figures detected from the input image exists based on a positional relationship between the figure and the character string in the input image, and if it is determined that a character string related to the figure exists, generating correlation information for the plurality of figures by associating coordinates indicating a position of the figure and a type of the figure with text information indicating the character string related to the figure; displaying, on a display unit, a display image in which text information indicating the type of the plurality of figures and the character string associated with the figures is arranged near the plurality of figures based on the relevance information; Run the following on your computer: moreover, A process of acquiring a first prediction model of association obtained by machine learning using training data including a position of a sample graphic as a sample, a position of a sample character string as a sample, and an association between the sample graphic and the sample character string; A process of acquiring a second prediction model for figure detection that is machine-learned using a predetermined type of figure and a sample image showing the predetermined type of figure as training data; a process of predicting a relationship between the graphic and the character string based on a positional relationship between the graphic and the character string using the first prediction model, and detecting types of the plurality of graphic elements detected from the input image using the second prediction model, in the process of generating the relationship information; causing the computer to execute the positional relationship between the graphic and the character string includes at least one of a distance between the graphic and the character string and a direction of the character string relative to the graphic; Computer program.

Images