Text extraction method

Abstract

To provide a character extraction method capable of suppressing an increase in time required to find a portion of an image where characters indicating contents of predetermined information are written.SOLUTION: A character extraction method causes a computer to execute steps of: segmenting an image into a plurality of sections (S13); selecting, on the basis of a predetermined information, from among the plurality of sections, a section that is most likely to contain characters indicating contents of the predetermined information (S17, S19); and extracting characters indicating the contents of the predetermined information from the selected section (S21, S23).SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a character extraction method for extracting characters from an image. [Background technology]

[0002] Patent Document 1 discloses an example of a method for converting a text image into a text format using optical character recognition technology. Specifically, in this method, a bounding box of a text region is determined from a picture to be identified, and a text region picture is extracted from the picture to be identified based on the bounding box. Next, a bounding box of a text line is determined from the text region picture, and a text line picture is extracted from the text region picture based on the bounding box. Then, text sequence recognition is performed on the text line picture to obtain a recognition result.

[0003] The classification results are obtained using a lightweight text sequence classification model, which is a trained model that has undergone machine learning. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-197190 Summary of the Invention [Problem to be solved by the invention]

[0005] In the above method, it may take a long time to find a portion of the picture to be identified that contains characters indicating the content of the desired information as a text line picture. [Means for solving the problem]

[0006] A character extraction method for solving the above problem is a method for extracting characters indicating the content of predetermined information from an image represented by image data. The character extraction method causes a computer to execute the following steps: divide the image into a plurality of sections, select from the plurality of sections a section that is most likely to contain characters indicating the content of the predetermined information based on the predetermined information, and extract the characters indicating the content of the predetermined information from the selected section.

[0007] In the character extraction method, an image is divided into a plurality of sections, and a section that is most likely to contain characters related to predetermined information is selected from the plurality of sections. At this time, the section is selected based on the predetermined information. Then, characters indicating the content of the predetermined text information are extracted from the section. Therefore, the character extraction method can prevent the time required to find a portion of an image containing characters indicating the content of the predetermined information from increasing.

[0008] In addition, when selecting the section that is most likely to contain characters indicating the content of specified information from among multiple sections, a trained model that has been subjected to machine learning may be used. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an outline of an extraction device for realizing a character extraction method according to an embodiment. [Figure 2] FIG. 2 is a schematic diagram showing an image of a design drawing. [Figure 3] FIG. 3 is a schematic diagram showing an image of a report. [Figure 4] FIG. 4 is a flowchart showing a character extraction method according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] An embodiment of the character extraction method will be described below with reference to FIGS. The character extraction method of this embodiment is a method for extracting characters written in an image represented by image data from the image. More specifically, the character extraction method is a method for extracting characters indicating the content of predetermined information from an image.

[0011] <Extraction device> Referring to FIG. 1, an extraction device 10 for implementing the character extraction method will be described. The extraction device 10 includes a user interface 11 and a computer 20 .

[0012] The user interface 11 has an operation unit 12 operated by the worker and a display unit 13 that displays the contents of information received from the computer 20. The operation unit 12 has at least one of physical buttons or switches and buttons displayed on a display screen having a touch panel. The user interface 11 outputs requests to the computer 20 in response to the operation of the operation unit 12 by the worker.

[0013] The computer 20 is, for example, an electronic control device. In this case, the computer 20 has a CPU 21, a first storage device 22, and a second storage device 23. The first storage device 22 stores a control program CP executed by the CPU 21. When the CPU 21 executes the control program CP, the computer 20 performs the series of processes shown in FIG. 4. That is, the computer 20 divides an image represented by image data into multiple sections. The computer 20 ranks the multiple sections in order of likelihood that characters indicating the content of predetermined information are written therein. The computer 20 then performs a process of extracting characters indicating the content of the predetermined information from the section with the highest ranking among the multiple sections using OCR. Note that OCR stands for optical character recognition.

[0014] The second storage device 23 stores a trained model LM and a plurality of texts TX. The trained model LM is a learning model that has undergone machine learning to rank multiple sections into which an image is divided in order of the likelihood that characters indicating the content of predetermined information are written. For example, an example of the trained model LM is an approximator of a multidimensional polynomial. For example, the trained model LM is configured by a fully connected forward propagation type neural network with one hidden layer.

[0015] When predetermined information, image data, and the type of document are input as input variables, the trained model LM outputs, as output variables, values ​​indicating the likelihood that characters indicating the content of the predetermined information are written in each of the multiple sections. If the number of sections in the image is four, the trained model LM outputs an output variable for each of the four sections.

[0016] For example, the trained model LM was generated by applying the following machine learning to the training model. FIG. 2 illustrates an image IMGA of a design drawing. In the example shown in FIG. 2, the image IMGA is divided into four sections SC1, SC2, SC3, and SC4. In the image IMGA of a design drawing, the document type is "design drawing," and an example of the predetermined information is a general tolerance. In the image IMGA shown in FIG. 2, the characters "JISG100," which indicate the content of the general tolerance (predetermined information), are present in section SC4. Therefore, when the general tolerance is used as the predetermined information, the design drawing is used as the document type, and the image data of the image IMGA are used as input variables of a learning model, machine learning is performed on the learning model so that the output variable of section SC4 is larger than the output variables of the other sections SC1, SC2, and SC3.

[0017] FIG. 3 illustrates a report image IMGB. The report illustrated in FIG. 3 has six pages. Therefore, in the example illustrated in FIG. 3, the image IMGB is divided into six sections SC1, SC2, SC3, SC4, SC5, and SC6. In the report image IMGB, the document type is "report," and an example of the predetermined information is the publication date. In the image IMGB illustrated in FIG. 3, characters indicating the content of the publication date (predetermined information) are present in section SC1. Therefore, when the publication date is the predetermined information, the report is the document type, and the image data of the image IMGB are used as input variables of a learning model, machine learning is performed on the learning model so that the output variable of section SC1 is larger than the output variables of the other sections SC2, SC3, SC4, SC5, and SC6.

[0018] The text TX is used to identify characters extracted from an image. The multiple text TX include text TX for design drawings and text TX for reports. For example, the text TX for the general tolerance of a design drawing is "JISG100".

[0019] <Character extraction method> 4 is a flowchart showing the execution procedure of a plurality of processes that constitute the character extraction method. When the computer 20 acquires image data, the computer 20 starts the series of processes shown in FIG.

[0020] In step S11, the computer 20 acquires the document type and predetermined information in the image of the acquired image data. The computer 20 acquires the document type and predetermined information based on, for example, an input operation by the operator on the operation unit 12 of the user interface 11.

[0021] In step S13, the computer 20 divides the image data into a plurality of sections. At this time, the computer 20 may divide the image into a number of sections according to the type of document acquired in step S11. For example, if the image is an image IMGA of a design drawing as shown in FIG. 2, the computer 20 divides the image IMGA into four sections SC1 to SC4. Alternatively, if the image is an image IMGB of a report as shown in FIG. 3, the computer 20 divides the image IMGB into sections equal in number to the number of pages.

[0022] In step S15, the computer 20 inputs the document type, the specified information, and the image data as input variables into the trained model LM. In the following step S17, the computer 20 ranks the multiple sections based on the output variables of the trained model LM in order of likelihood that the section contains characters indicating the content of the specified information. For example, suppose an image is divided into four sections SC1 to SC4, and the output variable of section SC4 is the highest, the output variable of section SC2 is the second highest, the output variable of section SC3 is the third highest, and the output variable of section SC1 is the fourth highest. In this case, the computer 20 ranks section SC1 first, section SC2 second, section SC3 third, and section SC1 fourth. Then, the computer 20 selects the section with the highest likelihood of containing characters indicating the content of the specified information from the multiple sections.

[0023] In step S19, the computer 20 sets the number N to 1. Then, the computer 20 shifts the process to step S21. In step S21, the computer 20 extracts characters from the Nth-ranked section among the multiple sections using OCR. If the number N is set to 1, the computer 20 performs a process of extracting characters from the first-ranked section, i.e., the section that is most likely to contain characters indicating the content of predetermined information. After completing this process, the computer 20 proceeds to step S23.

[0024] In step S23, the computer 20 calculates the similarity X between the characters extracted in step S21 and the characters of the text TX. Specifically, the computer 20 reads out the text TX corresponding to the predetermined information and document type from the second storage device 23. Then, the computer 20 calculates the similarity X by comparing the characters extracted in step S21 with the characters of the text TX read out from the second storage device 23.

[0025] Here, an example will be described in which the characters extracted in step S21 are "JISG106" and the characters in text TX used for comparison are "JISG100." In this case, among the characters extracted in step S21, the first "J," the second "I," the third "S," the fourth "G," the fifth "1," and the sixth "0" match the characters in text TX. On the other hand, the seventh "6" does not match the characters in text TX. Therefore, the computer 20 can determine that the number of characters that match the text TX is six. Then, the computer 20 calculates the similarity X using, for example, the following relational expression (F1). In relational expression (F1), "Z1" is the number of characters that match the text TX, "Y1" is the number of characters extracted by OCR, and "Y2" is the number of characters in text TX.

[0026] X = (2 × Z1) / (Y1 + Y2) (F1) In the example described here, the number of characters extracted by OCR is 7, and the number of characters in the text TX is 7. Therefore, the computer 20 assigns 7 to "Y1" and 7 to "Y2." Then, the computer 20 assigns 6 to "Z1," thereby calculating the similarity X as "0.857."

[0027] If the characters extracted in step S21 are "JISG100", the computer 20 substitutes 7 for "Z1" in the relational expression (F1), and therefore calculates the similarity X as "1".

[0028] After calculating the similarity X, the computer 20 proceeds to step S25. In step S25, the computer 20 determines whether the similarity X calculated in step S23 is greater than a threshold value Xth. The threshold value Xth is a criterion for determining whether characters indicating the content of predetermined information are included in the image. Character extraction using OCR may result in extraction errors. On the other hand, if the similarity X is less than 0.5, it is difficult to determine whether the characters extracted by OCR match the text TX. Therefore, it is preferable to set the threshold value Xth to a value greater than 0.5 and less than 1. If the similarity X is greater than the threshold value Xth, it is determined that characters indicating the content of predetermined information are included in the image. On the other hand, if the similarity X is equal to or less than the threshold value Xth, it is determined that characters indicating the content of predetermined information are not included in the image. If the similarity X is greater than the threshold value Xth (S25: YES), the computer 20 proceeds to step S27. On the other hand, if the similarity X is equal to or less than the judgment value Xth (S25: NO), the computer 20 shifts the process to step S31.

[0029] In step S27, computer 20 displays on display unit 13 of user interface 11 that the image contains text indicating the content of the predetermined information. Computer 20 also writes the "text indicating the content of the predetermined information" contained in the image to a predetermined work file. Computer 20 then terminates the series of processes.

[0030] In step S31, the computer 20 determines whether the character extraction process has been performed on all sections. For example, if the number N matches the number of sections in the image, the computer 20 determines that the character extraction process has been performed on all sections (S31: YES). The computer 20 then proceeds to step S35. On the other hand, if the number N is smaller than the number of sections in the image, the computer 20 determines that there is a section among the multiple sections for which the character extraction process has not yet been performed (S31: NO). The computer 20 then proceeds to step S33.

[0031] In step S33, the computer 20 increments the number N by 1. After updating the number N in this way, the computer 20 shifts the process to step S21.

[0032] In step S35, computer 20 notifies the worker that the image does not contain characters indicating the content of the predetermined information. In this case, computer 20 displays a message on display 13 of user interface 11 indicating that the image does not contain characters indicating the content of the predetermined information. Then, computer 20 ends the series of processes.

[0033] <Action and effect> When image data is input to computer 20, the series of processes shown in Figure 4 is executed. When computer 20 acquires the document type and predetermined information of the image represented by the image data, computer 20 processes the image and divides the image into multiple sections based on the document type. For example, if the document type is a design drawing, the image is divided into four sections SC1 to SC4. Then, from the multiple sections, the section that is most likely to contain characters indicating the content of the predetermined information is selected. Then, the characters indicating the content of the predetermined information are extracted from the selected section.

[0034] That is, in this embodiment, based on the predetermined information, the part of the image that is most likely to contain the text indicating the content of the predetermined information is searched for, and the text extraction process is performed on that part with priority over other parts, thereby preventing the time required to search for the part of the image that contains the text indicating the content of the predetermined information from being too long.

[0035] In this embodiment, the following effects can be further obtained. (1) In this embodiment, a trained model LM is used to rank multiple sections into which an image is divided in order of likelihood that characters indicating the content of predetermined information are written. There are cases where the first-ranked section does not contain characters indicating the content of the predetermined information. In this case, a process is executed to determine whether the second-ranked section contains characters indicating the content of the predetermined information. Then, if the second-ranked section does not contain characters indicating the content of the predetermined information, a process is executed to determine whether the third-ranked section contains characters indicating the content of the predetermined information. Therefore, if any of the images contains characters indicating the content of the predetermined information, the characters can be extracted.

[0036] (2) The input variables of the trained model LM include the document type in addition to the specified information and image data. Increasing the number of input variables in this way improves the accuracy of ranking the multiple sections into which an image is divided in order of the likelihood that characters indicating the content of the specified information are written.

[0037] <Example of change> The above embodiment can be modified as follows: The above embodiment and the following modifications can be combined with each other within the scope of technical compatibility.

[0038] When extracting characters from an image using OCR, misreading of characters can occur. For example, the following characters are prone to misreading: The capital letter "O" and the number "0 (zero)."

[0039] The capital letter "I" and the number "1". The capital letter "Z" and the lowercase letter "z". The capital letter "G" and the number "6".

[0040] It is advisable to register characters that are prone to misreading as confirmation characters in advance. For example, if the character actually written in the image is "I (uppercase alphabet)," OCR may extract "I (uppercase alphabet)" or "1 (number one)." Therefore, if the character extracted by OCR is the confirmation character and the character in text TX is "I (uppercase alphabet)," it is advisable to determine that the extracted character matches the character in text TX, regardless of whether the character extracted by OCR is "I" or "1." This prevents the similarity X from being calculated too low due to the accuracy of character extraction by OCR.

[0041] The input variables of the trained model LM do not need to include the document type as long as they contain the specified information and image data. Neural networks are not limited to feedforward networks with one hidden layer. For example, neural networks can have two or more hidden layers, or they can be convolutional neural networks or recurrent neural networks.

[0042] The trained model based on machine learning does not have to be a neural network. For example, a support vector machine can be used as the trained model. When ranking multiple sections into which an image is divided in order of the likelihood that characters indicating the content of predetermined information are written, it is not necessary to use the trained model LM. For example, multiple sections may be ranked based on a rule base created in advance.

[0043] The computer 20 is not limited to a computer having a CPU and ROM and executing software processing. In other words, the computer 20 may have any of the following configurations (a) to (c):

[0044] (a) The computer 20 has one or more processors that execute various processes according to a computer program. The processor includes a CPU and memory such as RAM and ROM. The memory stores program code or instructions configured to cause the CPU to execute processes. Memory, i.e., computer-readable media, includes any available media that can be accessed by a general-purpose or special-purpose computer.

[0045] (b) The computer 20 includes one or more dedicated hardware circuits for performing various processes. Examples of dedicated hardware circuits include application-specific integrated circuits (ASIC) and FPGA. ASIC is an abbreviation for "Application Specific Integrated Circuit," and FPGA is an abbreviation for "Field Programmable Gate Array."

[0046] (c) The computer 20 includes a processor that executes some of the various processes in accordance with a computer program, and dedicated hardware circuits that execute the remaining processes among the various processes.

[0047] The expression "at least one" used herein means "one or more" of the desired options. As an example, the expression "at least one" used herein means "only one option" or "both of two options" if the number of options is two. As another example, the expression "at least one" used herein means "only one option" or "any combination of two or more options" if the number of options is three or more. [Explanation of symbols]

[0048] 20...Computer, 21...CPU, 23...Second storage device, IMGA, IMGB...Images, LM...Trained model, SC1 to SC6...Partitions.

Claims

1. A character extraction method for extracting characters indicating content of predetermined information from an image represented by image data, comprising: Segmenting the image into a plurality of sections; selecting, from the plurality of sections, the section that is most likely to contain characters indicating the content of the predetermined information based on the predetermined information; extracting characters indicating the content of the predetermined information from the selected section; The computer includes a storage device that stores a trained model that has been subjected to machine learning, and the trained model uses the predetermined information and the image data as input variables, and outputs, as an output variable, a value that indicates the likelihood that characters indicating the content of the predetermined information are written in the plurality of sections into which the image is divided; When selecting the section that is most likely to contain characters indicating the content of the predetermined information from the plurality of sections, the computer ranks the plurality of sections in order of likelihood that characters indicating the content of the predetermined information are written based on output variables of the trained model when the predetermined information and the image data are input as input variables to the trained model, and selects the section with the highest rank; selecting the section with the second highest ranking from among the plurality of sections when characters indicating the content of the predetermined information cannot be extracted from the section with the highest ranking; extracting characters indicating the content of the predetermined information from the second highest ranked section. Character extraction method.

2. The input variables of the trained model further include the type of document appearing in the image; When ranking the plurality of sections, the computer is caused to rank the plurality of sections based on output variables of the trained model when the predetermined information, the image data, and the document type are input as input variables to the trained model. The character extraction method according to claim 1 .

Images