Information processing system, information processing method, and program

Abstract

To provide an information processing system and the like capable of reducing characters that cannot be recognized.SOLUTION: An information processing system provided with at least one or more processors is provided. In the information processing system, the processor specifies a registration element expressing a unit element obtained by dividing a recognized recognition character string from registration elements registered in a dictionary, as a recognition result of a character from a learning image, in the case that an artificial intelligence module recognizes characters from the learning image including the character string to perform machine learning in a specification step. Both the unit element and the registration element are an element smaller than a sub-word. In a learning step, the artificial intelligence module is made to execute machine learning on the basis of the specified recognition result and correct answer information on the character string included in the learning image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] Patent Document 1 discloses a technology for formal language understanding processing, which analyzes a formal language sentence input into a computer and outputs the results, in which the input formal language sentence is compared with the contents of a knowledge dictionary expressed in formal language, the formal language sentence input into a data input unit is understood, and the result of whether the input formal language sentence was understood is notified. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-121367 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology of Patent Document 1 often fails to correctly recognize words that are not registered in the knowledge dictionary, such as proper nouns such as company names and addresses.

[0005] In view of the above circumstances, the present invention provides an information processing system and the like that can reduce the number of unrecognizable characters. [Means for solving the problem]

[0006] According to one aspect of the present invention, an information processing system including at least one processor is provided. In this information processing system, in the identification step, when an artificial intelligence module recognizes characters from a training image containing a character string and performs machine learning, the processor identifies, from registered elements registered in a dictionary, registered elements representing unit elements obtained by dividing the recognized recognition character string as a recognition result of the character from the training image. Both the unit elements and the registered elements are elements smaller than subwords. In the learning step, the processor causes the artificial intelligence module to perform machine learning based on the identified recognition result and correct answer information for the character string included in the training image.

[0007] According to this embodiment, it is possible to reduce the number of characters that cannot be recognized. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of the overall configuration of an AIOCR system 1. FIG. [Figure 2] 2 is a diagram illustrating an example of a hardware configuration of a server device 10. FIG. [Figure 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of a user terminal 20. [Figure 4] FIG. 10 is an activity diagram illustrating an example of machine learning processing. [Figure 5] FIG. 10 is a diagram illustrating an example of an AI management screen. [Figure 6] FIG. 10 is a diagram illustrating an example of a character-based dictionary. [Figure 7] FIG. 10 is a diagram showing another example of learning images and correct answer information. [Figure 8] FIG. 10 is an activity diagram illustrating an example of character recognition processing. [Figure 9] FIG. 10 is a diagram illustrating an example of a service provision screen. [Figure 10] FIG. 10 is a diagram illustrating an example of a recognition result screen. [Figure 11] FIG. 10 is a diagram illustrating an example of a learning image. [Figure 12] FIG. 10 is a diagram illustrating an example of a character number table. [Figure 13] FIG. 10 is a diagram illustrating an example of a character number table. [Figure 14] FIG. 10 is a diagram illustrating another example of a learning image. [Figure 15] FIG. 10 is a diagram showing an example of a displayed recognition result screen. [Figure 16] FIG. 10 is a diagram showing an example of a displayed recognition result screen. [Figure 17] FIG. 10 is a diagram showing an example of a replacement target indication image. [Figure 18] FIG. 10 is a diagram showing an example of displayed text. [Figure 19] FIG. 4 is a diagram showing an example of an image to be read; [Figure 20] FIG. 10 is a diagram illustrating an example of correct answer information. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below with reference to the accompanying drawings. Various features shown in the following embodiments can be combined with each other.

[0010] Incidentally, the program for realizing the software appearing in one embodiment may be provided as a non-transitory computer-readable medium, or may be provided so that it can be downloaded from an external server, or may be provided so that the program is started on an external computer and its functions are realized on a client terminal (so-called cloud computing).

[0011] Furthermore, various information processing according to an embodiment may realize input and output corresponding to the input. Here, the form of information referenced in such information processing (hereinafter referred to as reference information) is not limited as long as an output is obtained as a result of the input. The reference information may be, for example, rule-based information such as a database, a lookup table, or a predetermined function (including a decision formula such as a regression formula constructed using a statistical method), a trained model that has previously trained the correlation between input and output, or a large-scale language model that can output a desired result by inputting a prompt.

[0012] In one embodiment, a "unit" may include, for example, a combination of hardware resources implemented by a circuit in the broad sense and software information processing that can be specifically realized by these hardware resources. In one embodiment, various information is handled, and this information is represented, for example, by physical values ​​of signal values ​​representing voltage and current, high and low signal values ​​as a binary bit set consisting of 0 or 1, or quantum superposition (so-called quantum bits), and communication and calculations can be performed on a circuit in the broad sense.

[0013] Furthermore, a circuit in a broad sense is a circuit realized by at least an appropriate combination of a circuit, circuitry, a processor, a memory, etc. That is, it includes an application specific integrated circuit (ASIC), a programmable logic device (e.g., a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA)), etc.

[0014] <Embodiment> 1. System Configuration The system configuration according to the embodiment will be described below. Fig. 1 is a diagram showing an example of the overall configuration of the AIOCR system 1. Fig. 1 shows an overview of each device provided in the AIOCR system 1 and users who use those devices. Each overview will be explained as needed with reference to other figures.

[0015] The AIOCR system 1 is an information processing system that performs information processing such as character recognition processing to recognize characters contained in a string of characters from an image containing the string of characters. Hereinafter, recognizing characters is also referred to as character reading. The AIOCR system 1 also provides, for example, a predetermined service, and users of the service recognize characters from images of predetermined documents, papers, or written documents, and use the recognized characters to perform processing related to the service.

[0016] The AIOCR system 1 includes a communication line 2, a server device 10, a user terminal 20, and an administrator terminal 30. The communication line 2 is not particularly limited, but may be configured, for example, by the Internet network. The communication line 2 may also include a local area network, a mobile communication network, a VPN (Virtual Private Network), etc. The communication line 2 mediates the exchange of data between devices connected to the communication line. In the example of FIG. 1, the server device 10 is connected to the communication line 2 by a wired connection, and the user terminal 20 and the administrator terminal 30 are connected wirelessly. Note that the connection of each device to the communication line 2 may be wired or wireless.

[0017] The server device 10 is an information processing device that executes information processing such as character recognition processing and service processing for providing the above-mentioned services. The server device 10 stores a dictionary database DB1 and a service database DB2. The dictionary database DB1 stores dictionary data used for character recognition. The service database DB2 stores data used in the service processing.

[0018] The server device 10 also executes information processing using AI (Artificial Intelligence) technology. The server device 10 includes an artificial intelligence module 100. The artificial intelligence module 100 is a module adjusted to realize a predetermined function using AI technology, and hereinafter will also be referred to simply as "AI." The artificial intelligence module 100 is adjusted to realize a function (so-called AIOCR function) of recognizing and outputting a character string when an image including the character string is input. The artificial intelligence module 100 can also improve the performance of the AIOCR function by performing machine learning.

[0019] The user terminal 20 is a terminal whose user is a user of the service provided by the AIOCR system 1, and is, for example, a personal computer, smartphone, or tablet terminal. The user terminal 20 displays a service usage screen and accepts operations by service users. The administrator terminal 30 is a terminal whose user is a system administrator who manages the AIOCR system 1, and is, for example, a personal computer. The administrator terminal 30 displays a management screen for managing the system and accepts operations by the system administrator.

[0020] The server device 10 executes a display process for displaying images on the user terminal 20 and the administrator terminal 30, and an authentication process for authenticating the service users and system administrators who use the user terminal 20 and the administrator terminal 30.

[0021] The server device 10 performs processes such as generating and transmitting an HTML (Hyper Text Markup Language) file as display processing, and uses the browser functions to display a web page showing a system screen on the user terminal 20 and the administrator terminal 30. Note that the user terminal 20 and the administrator terminal 30 may install an application program for using the AIOCR system 1, and the server device 10 may perform processes such as generating and transmitting display data in that application as display processing. The server device 10 controls the display of the user terminal 20 and the administrator terminal 30 by performing these display processes.

[0022] The server device 10 stores authentication information (user ID, password, etc.) for authenticating users who use the AIOCR system 1, such as service users and system administrators, and authenticates users who input the authentication information. By authenticating users, the server device 10 can restrict access to data and assign identification information to data entered by users to make the data identifiable.

[0023] 2. Hardware Configuration The hardware configuration according to the first embodiment will be described below. 2 is a diagram showing an example of the hardware configuration of server device 10. Server device 10 includes a control unit 11, a storage unit 12, a communication unit 13, and a bus 14. Bus 14 electrically connects the various units included in server device 10.

[0024] (Control unit 11) The control unit 11 has at least one processor. The at least one processor may be configured by, for example, a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), one or more integrated circuits, one or more discrete circuits, or a combination thereof (not shown).

[0025] The control unit 11 is a computer that realizes various functions related to the AIOCR system 1 by reading out predetermined programs stored in the storage unit 12. In other words, information processing by software stored in the storage unit 12 is specifically realized by the control unit 11, which is an example of hardware, and can be executed as each functional unit included in the control unit 11. Note that the control unit 11 is not limited to being a single unit, and may be implemented with multiple control units 11 for each function. A combination of these may also be used.

[0026] (Storage unit 12) The memory unit 12 stores various information defined above. This can be implemented, for example, as a storage device such as a solid state drive (SSD) or a hard disk drive (HDD) that stores various programs related to the AIOCR system 1 executed by the control unit 11, or as a memory such as a random access memory (RAM) that stores temporarily required information (arguments, arrays, etc.) related to program calculations. The memory unit 12 stores various programs, variables, etc. related to the AIOCR system 1 executed by the control unit 11.

[0027] (Communications Department 13) The communication unit 13 is configured by a communication module. The communication module may be a wireless communication module conforming to standards such as IEEE802.11a / b / g / n / ac / ax, LTE, 5G, or 6G, or may be a wired communication module conforming to standards such as IEEE802.3. The communication unit 13 is configured to be able to transmit various electrical signals from the server device 10 to external components. The communication unit 13 is also configured to be able to receive various electrical signals from the external components to the server device 10. More preferably, the communication unit 13 has a network communication function, which allows various information to be communicated between the server device 10 and external devices via the communication line 2.

[0028] Fig. 3 is a diagram showing an example of the hardware configuration of user terminal 20. User terminal 20 includes control unit 21, memory unit 22, communication unit 23, input unit 24, output unit 25, and bus 26. Bus 26 electrically connects the various units included in user terminal 20. Control unit 21, memory unit 22, and communication unit 23 are similar hardware to control unit 11, memory unit 12, and communication unit 13 shown in Fig. 2, although their specifications, models, etc. may differ.

[0029] (Input unit 24) The input unit 24 has keys, buttons, a touch screen, a mouse, etc., and receives input from the user. The input unit 24 may also have a microphone and have the function of receiving voice input from the user.

[0030] (Output section 25) The output unit 25 has a display, a speaker, etc., and displays visual information generated in a manner that is visible to the user, such as a screen, an image, an icon, or text, on the display surface of the display, and outputs sound including voice.

[0031] 3 has the same hardware configuration as the user terminal 20. In the explanation of the administrator terminal 30, only the control unit 31 is assigned a different reference numeral from the control unit 21 of the user terminal 20.

[0032] 3. Information Processing The following describes information processing according to the embodiment. In the following description, the server device 10, the user terminal 20, and the administrator terminal 30 are described as the subjects of each information processing, but these information processings are executed by at least one processor provided in the AIOCR system 1, i.e., the processor provided in the control unit of each device. In addition to the character recognition processing described above, the AIOCR system 1 also executes machine learning processing that causes the artificial intelligence module 100 to perform machine learning to realize the AIOCR function.

[0033] Figure 4 is an activity diagram showing an example of machine learning processing. Machine learning is generally performed based on a large amount of training data, but in the example shown in Figure 4, for ease of explanation, the process of machine learning performed by the artificial intelligence module 100 based on one piece of training data will be described as an example. Note that even when a large amount of training data is used at once, the processes performed using each piece of training data are basically the same, and are performed sequentially or in parallel.

[0034] First, the administrator terminal 30 displays an AI management screen for managing the artificial intelligence module 100 (activity A11). Fig. 5 is a diagram showing an example of an AI management screen. The AI ​​display screen C1 shown in Fig. 5 displays the text "Please enter training data," a training data input field D11, a training image display field D12, a correct answer information display field D13, a dictionary data specification field D14, a machine learning execution button B11, and an end machine learning process button B12.

[0035] The input field D11 is a field for inputting the data name of the training data. As described above, the training data is, for example, data indicating an image containing a character string and the content of the character string. Hereinafter, an image containing a character string that is used for machine learning will be referred to as a "training image," and information indicating the exact content of the character string contained in the training image will be referred to as "correct answer information." It is assumed that the training images and correct answer information are created and prepared in advance.

[0036] Display field D12 is a field for displaying a training image indicated by the input training data, and display field D13 is a field for displaying correct answer information indicated by the input training data. In the example of Figure 5, training image E12 including the character strings "Reiwa 6 xxx...", "yyy...", and "Shinjuku-ku Samon-cho zzz...", and correct answer information E13 indicating these character strings are displayed in display fields D12 and D13, respectively.

[0037] The specification field D14 is a field for specifying a dictionary to be used for machine learning. A dictionary is information in which a large number of linguistic elements are registered. Generally, there are dictionaries in which a large number of words are registered, and dictionaries in which elements called subwords, which are smaller divisions of words, are registered. However, the AIOCR system 1 uses a dictionary in which linguistic elements smaller than subwords are registered. Specifically, a character-based dictionary E14 is used in which a large number of characters are registered, each of which is a single character as a linguistic element. The dictionary E14 is stored in the dictionary database DB1 shown in FIG. 1.

[0038] Fig. 6 is a diagram showing an example of a dictionary by character. In the dictionary database DB1 shown in Fig. 6, the registered elements of the dictionary E14 described in Fig. 5 are registered for each category. Specifically, each of the 50 Japanese syllables from "a" to "n", the palindromic consonants, double consonants, long consonants, voiced and semi-voiced consonants, the numerals from "0" to "9", the alphabets from "A" to "Z" and "a" to "z", and frequently used symbols such as ","", "."", "=", "<", ">" and the like are registered. Note that the characters and categories to be registered are not limited to these, and other elements (e.g., Chinese characters, Korean characters, Arabic characters, etc.) may be registered as long as they are one character and one category.

[0039] The administrator terminal 30 accepts, as operations for inputting teacher data, the system administrator's operation of inputting the teacher data name in the input field D11, the operation of specifying the dictionary in the specification field D14, and the operation performed on the execute button B11 (activity A12). The administrator terminal 30 transmits instruction data indicating the input teacher data (learning image and correct answer information) and the specified dictionary to the server device 10. The server device 10 acquires the teacher data indicated by the transmitted instruction data (activity A21).

[0040] Next, the server device 10 inputs a learning image (for example, learning image E12 shown in FIG. 5) included in the acquired training data to the artificial intelligence module 100 and instructs it to output unit elements (activity A22). A unit element is an element that represents the smallest unit when dividing a character string, and generally, a word, a subword, or a morpheme is used, but in the AIOCR system 1, a single character is used as a unit element. Dividing a character string into unit elements is also called tokenizing or word segmentation.

[0041] The artificial intelligence module 100 is assumed to be equipped with a tokenize function in addition to the above-mentioned AIOCR function, i.e., a character recognition function that recognizes and generates a character string when an image including the character string is input. The artificial intelligence module 100 generates a character string recognized from the input image (hereinafter referred to as a "recognized character string") using the AIOCR function (activity A23). In the example of FIG. 4, the artificial intelligence module 100 generates a recognized character string from a learning image.

[0042] The artificial intelligence module 100 also has a position identification function that identifies the position of each recognized character in the learning image and outputs position information indicating the identified position, but the machine learning process shown in Figure 4 does not use position information and therefore does not output it. Next, the artificial intelligence module 100 uses the tokenize function to output unit elements obtained by dividing the generated recognized character string (activity A24).

[0043] The artificial intelligence module 100 divides a character string into unit elements, each of which has a unit of "one character." For example, the artificial intelligence module 100 divides the character string "Reiwa 6" shown in FIG. 5 into the unit elements "Reiwa," "Wa," "6," and "Year." The artificial intelligence module 100 also divides the character string "Shinjuku-ku Samon-cho" shown in FIG. 5 into the unit elements "Shin," "Yado," "Ku," "Sa," "Mon," and "Cho." The artificial intelligence module 100 outputs the divided unit elements. The server device 10 acquires the output unit elements.

[0044] The server device 10 refers to the dictionary indicated by the instruction data, and identifies a registered element representing the unit element acquired at A24 from the registered elements registered in the referred dictionary (activity A25). For example, the server device 10 identifies the Chinese characters "令", "和", "年" and the number "6" registered in the dictionary E14 as the specific elements representing the unit elements of "令", "和", "6", "年" mentioned above. Also, the server device 10 identifies the Chinese characters "新", "宿", "区", "左", "門", "町" registered in the dictionary E14 as the specific elements representing the unit elements of "新", "宿", "区", "左", "門", "町". The server device 10 also identifies the corresponding registered elements for other unit elements.

[0045] Next, the server device 10 divides the correct answer information included in the acquired teacher data into unit elements (activity A31). For example, the server device 10 uses a word segmentation parser that divides into single-character units to divide the correct answer information into unit elements with one character as a unit. When the learning image E12 shown in FIG. 5 is input at A22, the server device 10 also divides the correct answer information E13 shown in FIG. 5 into unit elements. The server device 10 inputs the unit elements obtained by the division into the artificial intelligence module 100 and instructs the artificial intelligence module 100 to perform machine learning (activity A32).

[0046] More specifically, the server device 10 instructs the artificial intelligence module 100 to perform machine learning based on the learning image input at A22, the unit elements acquired at A24, the specific elements identified at A25, and the unit elements of the correct answer information input at A31. When the instruction of A32 is given, the artificial intelligence module 100 generates a learning model according to the instruction (activity A33). The learning model is information representing a rule (also called logic, etc.) for outputting correct answer data from input data.

[0047] In the machine learning process shown in Fig. 4, the artificial intelligence module 100 assigns, to each unit element (each character) recognized in A23 and divided in A24, either a correct label indicating that the registered element identified as a result of A25 (identification of registered element) matches the correct unit element, or an incorrect label indicating that the registered element does not match the correct element. The artificial intelligence module 100 then learns rules for outputting correct data based on the rules used to recognize characters labeled with correct answers and the rules used to recognize characters labeled with incorrect answers. The artificial intelligence module 100 generates a learning model representing the rules learned in this way.

[0048] When the generation of the learning model is complete, a message to that effect is displayed on the AI ​​display screen C1. If the system administrator wishes to input more training data, he or she operates A12 again, and if the system administrator wishes to end the machine learning process, he or she operates the end button B12 shown in Figure 5. The administrator terminal 30 determines whether or not the end button B12 has been operated (activity A34), and if not, returns to A12 and continues operation, but if this operation has been performed, the machine learning process is ended.

[0049] Next, the FT process, which performs fine-tuning on the learning model generated by the machine learning process, will be described. Fine-tuning is the process of adding new training data to a learned learning model, allowing it to learn, and adding new functions. In the AIOCR system 1, for example, the FT process is a process for adding a function to read a predetermined target of interest (such as insurance type or claim amount) in a document of a specific format (such as an insurance premium deduction certificate or a bill). The FT process is performed in the same procedure as the machine learning process shown in FIG. 4. The learning images and correct answer information input in the FT process will be described with reference to FIG. 7.

[0050] FIG. 7 is a diagram showing another example of a training image and correct answer information. In FIG. 7, a training image E15 different from that shown in FIG. 5 and correct answer information E16 are input on the AI ​​display screen C1 shown in FIG. 5. An image of an insurance premium deduction certificate including information on items such as "policyholder name," "insurance policyholder name," "insurance company name," "insurance type," and "insurance premium" is input as the training image E15. The training image E15 displays a character string indicating that the "insurance policyholder name" is "Yamada Taro," the "insurance company name" is "BCD Insurance Co., Ltd.", the "insurance type" is "medical insurance," and the "insurance premium" is "150,000."

[0051] On the other hand, the correct answer information E16 contains character strings indicating the correct answers for each item, such as "Policyholder Name = . . . ", "Recipient = Yamada Taro," "Insurance Company Name = BCD Insurance Co., Ltd.," "Insurance Type = Medical Insurance," and "Nursing Care Medical Insurance Premium = 1,500,000." Here, the item names "Recipient" and "Nursing Care Medical Insurance Premium" are different from the item names "Insurance Policyholder Name" and "Insurance Premium" in the learning image E15. The reason for this is as follows.

[0052] As the learning images, deduction certificates from various companies are used, and for example, even in a section that means the same "recipient," some companies use the item name "recipient" and other companies use the item name "name of policyholder." In contrast, the item names in the correct answer information are the item names used in the service used by the service user (in the examples from Figure 7 onwards, it is assumed that a service that handles procedures related to year-end tax adjustments is used), and the same item names are used no matter which company's deduction certificate is read.

[0053] Therefore, by performing fine tuning using the learning image E15 and the correct answer information E16, the artificial intelligence module 100 learns that the character string "Yamada Taro" read from the section "Name of policyholder Yamada Taro" is a character string indicating a "recipient." Therefore, when another deduction certificate from the same company is read, the artificial intelligence module 100 will read the character string following the "Name of policyholder" as a character string indicating a "recipient."

[0054] Furthermore, in the above service, there are multiple types of "insurance premiums," such as "individual pension insurance premiums," "general life insurance premiums," and "long-term care and medical insurance premiums." Meanwhile, the learning image E15 only shows the item name "insurance premiums." In this case, by performing fine tuning using the learning image E15 and the correct answer information E16, the artificial intelligence module 100 learns that when "insurance type = medical insurance," the character string "150000" read from the section "insurance premium 150000" is a character string indicating "long-term care and medical insurance premiums."

[0055] By executing the FT process, the artificial intelligence module 100 generates a learning model that indicates the additional learning content as described above.

[0056] Next, we will explain the character recognition process that uses a learning model generated by machine learning to recognize characters from an image containing a string of characters. Below, we will explain the character recognition process performed when a service that handles procedures related to the above-mentioned year-end tax adjustment is provided as an example. 8 is an activity diagram showing an example of character recognition processing. First, the user terminal 20 displays a screen for providing a service to be used by the service user (activity A41).

[0057] Fig. 9 is a diagram showing an example of a service provision screen. The service provision screen C2 shown in Fig. 9 is a screen for inputting the contents of a deduction certificate as input items required for procedures related to year-end tax adjustment. The service provision screen C2 displays a character string saying "Please enter the contents of your deduction certificate," a display field D21 for an image to be read, a display field D22 for input items, an image selection button B21, a read execution button B22, and an input completion button B23.

[0058] The reading image is an image from which the artificial intelligence module 100 reads a character string. For example, the service user converts a paper deduction certificate sent by mail or the like into an image using a scanner or digital camera in advance, and stores the image in the user terminal 20. Then, the service user operates the selection button B21 to select the stored image of the deduction certificate as the reading image. The reading image E21 selected in this way is displayed in the display field D21.

[0059] The reading image E21 includes item names such as "policyholder name," "policyholder name," "insurance company name," "insurance type," and "premium," as well as character strings indicating that the "insurance holder name" is "Kanada Susumu," the "insurance company name" is "Kanada Susumu Insurance Co., Ltd.", the "insurance type" is "medical insurance," and the "premium" is "240000." When the execute button B22 is operated in the state shown in FIG. 9, the user terminal 20 determines the reading image E21 displayed in the display field D21 as the reading image to be input to the artificial intelligence module 100 (activity A42), and transmits instruction data indicating the reading image and an instruction to read the character string from the reading image to the server device 10.

[0060] The server device 10 acquires the scanning image E21 indicated by the transmitted instruction data (activity A51). Next, the server device 10 inputs the acquired scanning image E21 to the artificial intelligence module 100 and instructs the module 100 to output a unit element in order to read a character string from the scanning image in accordance with the instruction indicated by the instruction data (activity A52). In the example of Fig. 8, the artificial intelligence module 100 first generates a recognized character string from the character string recognized from the input scanning image E21 (activity A53).

[0061] Next, the artificial intelligence module 100 divides the generated recognition character string into unit elements using the tokenize function (activity A54). Next, the server device 10 executes the same process as A25 (identifying registered elements) shown in FIG. 4 in activity A55. At this time, the server device 10 identifies the registered elements using the same dictionary as the dictionary used in the machine learning process (dictionary E14 in the example of FIG. 4). Next, the server device 10 generates screen data for a recognition result screen including the identified registered elements as a recognition result (activity A61). The server device 10 transmits the generated screen data to the user terminal 20. The user terminal 20 displays the recognition result screen indicated by the transmitted screen data (activity A62).

[0062] Fig. 10 is a diagram showing an example of a recognition result screen. The service provision screen C2 shown in Fig. 10 is an example of the recognition result screen, and character strings read from the reading image E21 are input as character strings indicating each item in a display field D22. The item names shown in the display field D22 (such as "policyholder name," "policy number," "beneficiary," "insurance company name," "insurance type," "insurance period," "individual pension insurance premium," "general life insurance premium," and "long-term care medical insurance premium") are the item names used in the above-mentioned services.

[0063] For example, the character string "Kaneda Susumu" read from the reading image E21 is displayed in the input item field "Recipient" in the display field D22, and the character string "Kaneda Susumu Insurance Co., Ltd." also read is displayed in the input item field "Insurance Company Name" in the display field D22. Furthermore, the character strings "Medical Insurance" and "240000" read from the reading image E21 are displayed in the input item fields "Insurance Type" and "Long-Term Care Medical Insurance Premium" in the display field D22, respectively.

[0064] For example, the artificial intelligence module 100 determines that the character string written after the "Name of the insured" read from the scanning image E21 is a character string indicating the input item "Recipient." The artificial intelligence module 100 also determines that the character strings written after the "Name of the insurance company" and "Type of insurance" are character strings indicating the input items "Name of the insurance company" and "Type of insurance," respectively.

[0065] Based on this judgment result, the artificial intelligence module 100 outputs the character strings "Recipient = Susumu Kaneda," "Insurance company name = Kinshin Insurance Co., Ltd.", and "Insurance type = Medical insurance" divided into unit elements. Based on these judgment results by the artificial intelligence module 100 and the output unit elements, the server device 10 displays "Susumu Kaneda" in the "Recipient" column of display column D22, displays "Kinshin Insurance Co., Ltd." in the "Insurance company name" column, and displays "Medical insurance" in the "Insurance type" column.

[0066] Furthermore, the artificial intelligence module 100 determines that the "insurance premium" read from the reading image E21, combined with the fact that the "insurance type" also read from the reading image E21 is "medical insurance," is a character string indicating the input item "long-term care medical insurance premium." Based on this determination result, the artificial intelligence module 100 outputs the character string "long-term care medical insurance premium = 240000" divided into unit elements. Based on this determination result by the artificial intelligence module 100 and the output unit elements, the server device 10 displays "240000" in the "long-term care medical insurance premium" column of the display column D22. In this way, the server device 10 displays the read character string based on the reading result by the artificial intelligence module 100.

[0067] If the service user finds any errors in the input displayed in the display field D22, they can correct those errors. In this case, the user terminal 20 accepts the correction operation by the service user (activity A63). When the service user determines that all input items have been entered correctly, they operate the input completion button B23. Upon accepting the operation of the input completion button B23, the user terminal 20 determines that the input of the recognition character string read from the scanning image has been completed (activity A64), stores the input character string in, for example, the service database DB2, and ends the character recognition process. After the character recognition process, a process using the stored character string (procedure processing related to year-end tax adjustment in the example of FIG. 8 etc.) is executed.

[0068] As described above, when the artificial intelligence module 100 recognizes characters from a training image containing a string of characters and performs machine learning, the server device 10 functions as an example of an identification unit that identifies, from the registered elements registered in the dictionary, registered elements that represent unit elements into which the recognized recognition string is divided, as the recognition result of the characters from the training image.

[0069] The division of the recognized character string into unit elements is performed by the artificial intelligence module 100, as explained in the example of FIG. 5. The artificial intelligence module 100 divides the recognized character string into unit elements, for example, each of which has a unit of "one character." Furthermore, the server device 10 identifies the registered elements using, for example, the character-based dictionary E14 shown in FIG. 6. In this way, both the unit elements and the registered elements are elements smaller than subwords.

[0070] In activity A24 shown in Fig. 4, the artificial intelligence module 100 divides recognized character strings such as "Reiwa 6" and "Shinjuku-ku Samon-cho" shown in Fig. 5 into unit elements. In addition, in activity A25 shown in Fig. 4, the server device 10 identifies registered elements such as "Reiwa," "Wa," "6," "Year," "New," "Inn," "Ward," "Left," "Mon," and "Machi" as registered elements corresponding to those unit elements.

[0071] The server device 10 functions as an example of a learning unit that causes the artificial intelligence module 100 to perform machine learning based on the identified recognition result and the correct answer information for the character string included in the training image. The server device 10 performs machine learning based on, for example, the instruction A32 shown in Fig. 4 (an instruction to perform machine learning based on the training image, the recognized character string, the specific element, and the correct answer information).

[0072] When using a dictionary with words and subwords as unit elements during machine learning, character strings that are not registered in the dictionary may be split as unit elements. For example, "Reiwa" and "Shinjuku" shown in Figure 5 are proper nouns, but they are well-known and may be registered in the dictionary. However, common proper nouns such as "Samoncho" and "Kinshin Insurance Co., Ltd." shown in Figure 9 are likely not registered in the dictionary. In addition, buzzwords, new words, and coined words are also likely not registered in the dictionary.

[0073] If a character string representing a word not registered in such a dictionary is segmented into a unit element, the registered element in the dictionary representing that unit element cannot be identified. In this case, since comparison with correct answer information is not even possible, for example, an unknown label is assigned instead of the above-mentioned correct answer label and incorrect answer label, resulting in no character recognition result.

[0074] In contrast, the AIOCR system 1 uses a character-by-character dictionary, so even for character strings such as general proper nouns, buzzwords, new words, or coined words, registered elements are always identified for the characters contained in those character strings. This allows the AIOCR system 1 to reduce the number of characters that cannot be recognized compared to when words or subwords are used as registered elements in the dictionary, and in particular, to increase the success rate of recognizing character strings that indicate proper nouns, etc. that cannot be registered as words or subwords.

[0075] In addition, in the AIOCR system 1, the unit elements divided by tokenizing A24 and the registered elements registered in the dictionary are both elements with one character as a unit. According to this aspect, registered elements are always identified for all characters regardless of the recognition character string, so as long as the character string is expressed in the language of the characters registered in the dictionary, it is possible to eliminate unrecognizable characters.

[0076] Furthermore, in the AIOCR system 1, the server device 10 (an example of a learning unit) performs machine learning using an image of a document having predetermined written information (hereinafter referred to as a "predetermined document") as a learning image. The predetermined document is, for example, a document of a specific format such as a deduction certificate shown in FIG. 7, an invoice, a receipt, an application form, or an application form. The machine learning referred to here is machine learning performed by the FT processing described in FIG. 7. Next, the server device 10 functions as an example of a second recognition unit that causes the artificial intelligence module 100 that has performed the above-mentioned machine learning to recognize characters from a reading image representing the predetermined document.

[0077] The server device 10 functions as an example of an input unit that inputs a character string represented by the recognized characters into an input field of a system for inputting information to be written on a predetermined document. For example, in the example of Fig. 10, the server device 10 inputs characters recognized from a scanning image E21 showing a deduction certificate into a display field D22. According to this aspect, machine learning is performed using the predetermined document itself as a learning image, and therefore, compared to the case where, for example, a random document is used as a learning image, the accuracy of transcribing characters from the document into the system can be improved.

[0078] <Variation: Dictionary> The dictionary used in the AIOCR system 1 is not limited to the above dictionary. For example, a dictionary in which not only registration elements in units of one character but also registration elements in units of two or more characters are mixed may be used. However, even in that case, it is desirable that elements smaller than subwords are registered in the dictionary as registration elements, and by making the unit elements also elements smaller than subwords, it is possible to reduce unrecognizable characters as in the above case.

[0079] <Modified Example: Difficult-to-Recognize Image> As the learning image, an image in which it is deliberately difficult to read a character string may be used. In that case, for example, the server device 10 (an example of the learning unit) uses an image in which a part of the character string is made difficult to recognize as the learning image to cause the artificial intelligence module 100 to perform machine learning.

[0080] FIG. 11 is a diagram showing an example of a learning image. In the learning image E31 shown in FIG. 11, there are shown a character string F31 such as "Reiwa 6 years xxx...", "yyy...", and "Zamonmachi, Shinjuku-ku zzz..." shown in FIG. 5, and mask images G31 and G32 that hide a part of those character strings. The mask image G31 hides the "wa" in "Reiwa", and the mask image G32 hides the "juku" in "Shinjuku". Note that the method of making a part of the character string difficult to recognize is not limited to the mask image. For example, the target character may be made the same color as the background color so that it cannot be seen.

[0081] When the learning image E31 is input, since a recognition character string that does not include "wa" and "juku" is generated by the artificial intelligence module 100 in A23 shown in FIG. 4, the registration elements for "wa" and "juku" are not specified in A25 either. However, when the server device 10 instructs the artificial intelligence module 100 to perform machine learning in A32, it makes an instruction based on the correct answer information including "wa" and "juku". As a result, even if "wa" and "juku" are not registration elements, the artificial intelligence module 100 generates a learning model that represents a rule for estimating "wa" and "juku" from the surrounding character strings based on the other surrounding registration elements.

[0082] By using a learning model generated by such machine learning, the artificial intelligence module 100 can, for example, recognize the character string "Reiwa 6" by inferring the character "wa" from the surrounding characters, even if the character "wa" in the character string "Reiwa 6" is faded and difficult to recognize. As a result, even if the character string shown in the image to be read contains unclear characters, the accuracy of character recognition can be improved compared to when difficult-to-recognize images are not used in machine learning.

[0083] Furthermore, when the artificial intelligence module 100 makes a recognition error, the server device 10 (an example of a learning unit) may use an image in which the erroneously recognized character and characters surrounding the character are made difficult to recognize as a learning image, and perform the next machine learning process. For example, when "Shinjuku-ku Samon-cho" is misrecognized as "Shin-ku Samon-cho" in the learning image E31 shown in Fig. 11 (for example, when the server device 10 is unable to recognize "Yado" and recognizes that there is a space without characters between "Shin" and "Ku"), the server device 10 overlays a mask image not only on "Yado" but also on, for example, "Kado" to make it difficult to recognize, and then performs the machine learning process.

[0084] In this way, the more characters that are made difficult to recognize, the more likely it is that the correct character can be guessed from fewer characters, improving recognition accuracy. As a result, the recognition accuracy can be further improved for misrecognized characters and their surroundings, so that even if misrecognition occurs, the recurrence of misrecognition can be suppressed compared to when only certain characters are made difficult to recognize.

[0085] Furthermore, when the server device 10 (an example of a learning unit) causes the artificial intelligence module 100 to perform machine learning using, as a learning image, an image in which part of a character string is made difficult to recognize, the server device 10 may perform machine learning by increasing the number of characters that are made difficult to recognize according to the number of times machine learning has been performed in the past. The server device 10 uses, for example, a character number table that associates the number of times machine learning has been performed in the past with the number of characters that are difficult to recognize.

[0086] FIG. 12 is a diagram showing an example of a character count table. In the character count table TB41 shown in FIG. 12, the number of difficult-to-recognize characters "N1", "N2", and "N3" (N is a natural number: N1 < N2 < N3) are respectively associated with the number of learning times "less than M1", "M1 or more and less than M2", and "M2 or more" (M is a natural number). The server device 10 first stores the number of times the machine learning process shown in FIG. 4 has been executed. The number of times here is the number of times the machine learning process has been executed for the same learning image, but it is not limited to this. For example, when there is a common part of a certain number of characters or more in the correct character string indicated by the correct answer information, it may be added to the number of times.

[0087] Next, when the server device 10 executes the machine learning process, it refers to the number of times the machine learning process has been executed so far and the number of difficult-to-recognize characters associated in the character count table TB41. The server device 10 randomly selects the number of characters among the characters included in the learning image, and makes them difficult to recognize by a method such as superimposing a mask image. Then, the server device 10 acquires the processed learning image as teacher data and executes the machine learning process.

[0088] According to such an aspect, as the machine learning process is executed, the number of difficult-to-recognize characters increases, and it becomes possible to infer the correct characters from fewer characters, improving the recognition accuracy. As a result, the recognition accuracy can be continuously improved.

[0089] Note that the recognition accuracy may be improved by methods other than increasing the number of difficult-to-recognize characters. For example, the server device 10 may increase the difficulty level of the location where the characters to be made difficult to recognize are represented according to the number of times machine learning has been performed in the past, and cause machine learning to be performed. In this case, the server device 10 uses, for example, a character count table in which the number of times of past machine learning is associated with the difficulty level of the location where the difficult-to-recognize characters are represented.

[0090] Fig. 13 is a diagram showing an example of a character count table. In the character count table TB42 shown in Fig. 13, the number of learning times "less than M1," "M1 or more and less than M2," and "M2 or more" are associated with the recognition difficulty levels "low," "medium," and "high N3," respectively. When performing machine learning processing, server device 10 refers to the number of times the machine learning processing has been performed so far and the recognition difficulty levels associated in the character count table TB42.

[0091] The server device 10 processes characters included in the learning image that are to be made difficult to recognize, making them more difficult to recognize based on the recognition difficulty level. For example, the server device 10 increases the difficulty of recognition by placing a graphic image near or superimposing the characters that are to be made difficult to recognize. This is because the closer the lines included in the graphic image and the lines representing the characters are to each other, the more difficult it becomes to distinguish between the lines and recognize the characters alone.

[0092] For example, if the recognition difficulty is "low," the server device 10 does not place a graphic image; if the recognition difficulty is "medium," the server device 10 places the graphic image near the character to be made difficult to recognize; and if the recognition difficulty is "high," the server device 10 places the graphic image superimposed on the character to be made difficult to recognize. Then, the server device 10 acquires the processed learning image as training data and executes machine learning processing. According to this aspect, the more the machine learning processing is executed, the more difficult it becomes to recognize difficult characters, and the higher the recognition accuracy, so that the recognition accuracy can be continuously improved.

[0093] Furthermore, the method of making characters difficult to recognize is not limited to the above-described method. When the server device 10 (an example of a learning unit) causes the artificial intelligence module 100 to perform machine learning using an image in which a part of a character string is made difficult to recognize as a learning image, the server device 10 (an example of a learning unit) may make the character difficult to recognize by hiding part of the character.

[0094] Fig. 14 is a diagram showing another example of a training image. Training image E51 shown in Fig. 14 displays character strings F51, "Reiwa 6 xxx...", "yyy...", and "Shinjuku-ku Samon-cho zzz...", as shown in Fig. 5, and mask images G51 and G52 that hide part of the characters. Mask image G51 hides the upper half of the "wa" character in "Reiwa", and mask image G52 hides the lower half of the "yado" character in "Shinjuku".

[0095] Note that a method for partially hiding characters may be to make the hidden part of the character the same color as the background color so that it is not visible. Also, the hidden part of the character may be the right half or the left half, etc., and the percentage of the character that is hidden may be a percentage other than half (e.g., 10% to 90%). The server device 10 causes the artificial intelligence module 100 to perform machine learning using training images in which characters are partially hidden, such as training image E51, as training data. In scanned images of paper documents, characters may be partially hidden, so this embodiment can improve recognition accuracy even for characters that are partially hidden.

[0096] Note that when making a part of a character string difficult to recognize, the server device 10 may make two or more consecutive characters difficult to recognize. For example, the server device 10 makes the two characters "kusa" in the character string "Shinjuku-ku Samon-cho" difficult to recognize, and causes the artificial intelligence module 100 to perform machine learning. As a result, the artificial intelligence module 100 will infer the character "kusa" between "Shinjuku" and "Kamon-cho" from the recognized characters "Shinjuku" and "Kamon-cho" and the correct answer information "Shinjuku-ku Samon-cho."

[0097] At that time, the artificial intelligence module 100 also learns the width and number of characters of the characters that have been made difficult to recognize. Therefore, for example, if there are two characters between "Shinjuku" and "Moncho", the artificial intelligence module 100 will infer the character "kusa"; however, if there is only one character or three or more characters between them, if the artificial intelligence module 100 has learned a character string that fills that number of characters (for example, "Shinjuku Samoncho" or "Shinjuku-ku no Samoncho"), it will infer the character between "Shinjuku" and "Moncho" in that character string, but if it has not learned such a character string, it will determine that it is a space with no characters.

[0098] In this way, when the recognized character string recognized from the reading image includes areas of characters that are insufficiently recognized (areas of characters partially hidden by mask images G51 and G52 shown in Figure 14) or areas that may contain characters to be recognized (areas hidden by mask images G31 and G32 shown in Figure 11), the artificial intelligence module 100 functions as an example of an estimation unit that estimates characters that will enter those areas from characters recognized around those areas.

[0099] <Variation: Letter Score> In the AIOCR system 1, a score indicating the likelihood of characters read by the AIOCR function may be used. First, the server device 10 functions as an example of a recognition unit that causes the artificial intelligence module 100, which has performed machine learning by a learning unit, to recognize characters from an image to be read that includes a character string. For example, the server device 10 causes the artificial intelligence module 100 to recognize characters in the character recognition process shown in FIG. 8.

[0100] Next, the server device 10 functions as an example of a score acquisition unit that acquires a score indicating the likelihood of characters recognized from a scanning image by the artificial intelligence module 100. When recognizing characters, the artificial intelligence module 100 calculates, for example, some value that comparatively represents the characteristics of the characters, and can output a score using that value. In this case, the server device 10 causes the artificial intelligence module 100 to output a score and acquires the output score.

[0101] For each location to be read, the artificial intelligence module 100 can output the probability that the shape written at that location is the character, and the probability can be used as a score. For example, if the shape of "tei" is written at a certain location in the image, the artificial intelligence module 100 can output a score such that the probability that the character written at that location is "tei" is 80% and the probability that it is "koto" is 15%.

[0102] Next, the server device 10 functions as an example of a first display control unit that controls the display of the reading image and the characters recognized from the reading image according to the score acquired for the characters. The server device 10 (an example of a first display control unit) displays the recognized characters in a format according to the acquired score, for example. Specifically, the server device 10 transmits screen data to the user terminal 20 that has transmitted the reading image, showing a recognition result screen in which the characters recognized from the reading image are displayed in a format according to the score acquired for the characters.

[0103] Fig. 15 is a diagram showing an example of a displayed recognition result screen. The service provision screen C6 shown in Fig. 15 is an example of a recognition result screen, and differs from the service provision screen C2 shown in Fig. 9 only in the display field D62 for input items. The display field D62 has a field for displaying the score of each recognition result in addition to the input items. In the example of Fig. 15, the recognition results "AAAA", "BBBB", "CCCC", "DDDD", "EEEE", "FFFF", and "JJJJ" are input in the "policyholder name", "policy number", "beneficiary", "insurance type", "contract date", "insurance period", and "long-term care medical insurance premium".

[0104] In the example of FIG. 15, when the score is greater than or equal to the first threshold, the server device 10 increases the font size; when the score is greater than or equal to the second threshold and less than the first threshold, it uses a font of normal size; and when the score is less than the second threshold, it displays each recognition result with an underlined style. For example, it can be seen that for the first two characters of "AAAA" in "Contractor Name", the score is greater than or equal to the first threshold, and for the last two characters, the score is greater than or equal to the second threshold and less than the first threshold. Also, since the third character of "CCCC" is underlined, it can be seen that the score is less than the second threshold. By displaying the characters in a manner according to the score in this way, the accuracy of character recognition can be intuitively grasped.

[0105] Note that the manner according to the score is not limited to the above example. For example, the color of the font or the font itself may be made different (such as making the accurate recognition result in Gothic font), or in particular, a specific symbol or the score value itself may be added near the characters with a low score. In any case, it is desirable to display in a manner such that the magnitude of the score of the recognized characters can be intuitively understood.

[0106] <Variant example: Character replacement> When the score of the recognition result is obtained as described above, the server device 10 (an example of the first display control unit) may use the recognition result with a score less than the threshold as a low-score character, and replace and display the low-score character with a character estimated to be correct.

[0107] The artificial intelligence module 100, for example, replaces the character with the next highest score after the low-score character as the character estimated to be correct. For example, when the character "宿" in "Shinjuku Ward" has other lines drawn and is misrecognized as "新縮区", it may happen that "縮" has the highest score but is a low-score character, and the score of the original correct character "宿" is lower than the score of "縮". In such a case, by replacing it with the character with the second highest score, the correct character can be displayed.

[0108] Furthermore, the artificial intelligence module 100 may estimate the true character of a low-score character from, for example, the recognition results of the characters before and after the low-score character. For example, the artificial intelligence module 100 extracts character strings similar to the character string containing the low-score character from previously recognized character strings, and estimates the character used in the same position as the low-score character from the extracted character string as the true character. In the example of the aforementioned misrecognition of "Shinjuku-ku" as "Shinshuku-ku," the artificial intelligence module 100 can identify a similar previous recognition result for "Shinjuku-ku" from the characters "Shin" and "Ku" before and after the low-score character "Shuku," and therefore estimates that "Yado," which is in the same position as "Shuku," is the correct character.

[0109] Alternatively, the artificial intelligence module 100 may store a dictionary of words for character replacement, extract words similar to a string containing a low-score character, and estimate a character in the extracted word that is used in the same position as the low-score character as the correct character. According to each of the above-described aspects, a sentence can be recognized more accurately than when character replacement is not performed.

[0110] Furthermore, when there is a low-score character whose acquired score is less than a threshold, server device 10 (an example of a first display control unit) may display candidate characters that are estimated to be correct, and may replace the low-score character with a character selected from the candidates and display the character. Server device 10 may use a character estimated to be correct using the method described above as a candidate character to replace the low-score character, or may use a character that has the next highest score after the low-score character when recognizing a character string (for example, a character with a score between 2nd and 4th).

[0111] Fig. 16 is a diagram showing an example of a displayed recognition result screen. In Fig. 16, the same service provision screen C6 as in Fig. 15 is displayed, and it is assumed that the third character of "CCCC" is a low-score character. In this case, for example, when a service user points the pointer at the low-score character, the server device 10 identifies characters that are candidates for replacing the low-score character and displays a presentation image F61 presenting the identified replacement candidate characters. The presentation image F61 presents the replacement candidate characters "e," "a," and "o."

[0112] The service user selects a character from the presented replacement candidate characters that they believe is appropriate as the third character of "CCCC." When one of the presented replacement candidate characters is selected, the server device 10 displays the selected character in place of the low-score character in the input item display field D62. This embodiment allows human judgment to be incorporated into character recognition, and also reduces the effort required to correct low-score characters compared to when replacement candidate characters are not presented.

[0113] Furthermore, the server device 10 (an example of a first display control unit) may superimpose an image showing a portion where characters whose calculated scores are equal to or less than a threshold (low-score characters) are displayed on the image to be read. The image showing a portion where low-score characters are displayed is an image showing characters to be replaced, and will be referred to as a "replacement target indication image" below.

[0114] Fig. 17 is a diagram showing an example of a replacement target indication image. In Fig. 17, the same service provision screen C6 as in Fig. 16 is displayed, and the third character of "CCCC" in the learning image E61 is assumed to be a low-score character. For example, when a service user points the pointer at a low-score character as in the example of Fig. 16, the server device 10 displays the presentation image F61 and also displays a replacement target indication image F62 superimposed on the learning image E61, indicating the character that has been recognized as the low-score character.

[0115] The replacement target indication image F62 is an image in the shape of a speech bubble containing the character string "replacement target," with the pointed portion pointing to the position of the character recognized as the low-scoring character. Note that the replacement target indication image is not limited to this, and may also be an image such as a frame, underline, or highlight that surrounds the replacement target. In either case, it is easier to grasp the parts of the scanning image that are difficult to recognize compared to when the replacement target indication image is not displayed.

[0116] <Variation: Text display> 4 is made to recognize characters from the image to be read, the server device 10 (second display control unit) may display the image to be read by superimposing a text image showing the recognized characters on the position of the characters. The text image here is an image of text expressed using a character font.

[0117] In this case, when the server device 10 causes the artificial intelligence module 100 to recognize a character string in the image to be read, the server device 10 instructs the module 100 to also output position information indicating the position of the recognized character string. The server device 10 acquires the position information output by the artificial intelligence module 100 based on this instruction. The server device 10 then generates an image in which text indicating the recognized character string is superimposed on the position indicated by the acquired position information, and displays the image on the user terminal 20.

[0118] Fig. 18 is a diagram showing an example of displayed text. The example in Fig. 18 shows a portion of a scanning image E71 including an input field D71 for "contract date." A handwritten character string F71, "Reiwa 6 (Reiwa 6)," is written in the input field D71. The server device 10 acquires the recognized character string "Reiwa 6 (Reiwa 6)" from the character string F71, and then displays a text image F72 indicating the acquired recognized character string in the input field D71, superimposed on the character string F71.

[0119] Note that the server device 10 may display the text image F72 above or below the character string F71 (or to the left or right if there is space) without superimposing it on the character string F71. The image to be scanned is an image obtained by scanning or photographing a paper document, so the characters may be handwritten or printed, but may be faded or difficult to read due to wrinkles or folds. In such cases, displaying the above text allows the characters to be displayed more clearly in the image to be scanned than when no text is displayed.

[0120] <Variation: Two ways of character recognition> Character recognition may be performed using other means other than the artificial intelligence module 100. The other means may be, for example, a character recognition module having an algorithm for recognizing characters from an image containing a character string, specifically a module having a well-known OCR function that operates using a preset algorithm, in other words, a module having a rule-based character recognition function different from the AIOCR function.

[0121] In this case, the server device 10 first functions as an example of a first recognition unit that causes the artificial intelligence module 100, which has performed the machine learning shown in Fig. 4, to recognize characters from a reading image including a character string. The server device 10 also functions as an example of a second recognition unit that causes the character recognition module described above to recognize characters from a reading image. The server device 10 then functions as an example of an adoption unit that compares the recognition result from the first recognition unit (hereinafter referred to as the "first recognition result") with the recognition result from the second recognition unit (hereinafter referred to as the "second recognition result") and adopts the more correct character.

[0122] As a method for comparing the accuracy of characters, for example, if the character recognition module is capable of calculating scores like the artificial intelligence module 100, the server device 10 may obtain a score for each character in the first recognition result and the second recognition result and compare the obtained scores. However, without being limited to this, the server device 10 may, for example, check whether the first recognition result and the second recognition result are registered in a word dictionary, and if only one of them is registered, determine that it is correct.

[0123] Furthermore, if both words are registered but different, the server device 10 may further compare the preceding and following words and determine that the word with a more natural connection is correct. The naturalness of the word connection may be determined using, for example, LLM (Large Language Model) technology. This embodiment can improve recognition accuracy compared to using only one type of recognition result.

[0124] The method of using the first recognition result and the second recognition result is not limited to the above. For example, the server device 10 first acquires the first recognition result and its score, and if the statistical value of the acquired score (such as the average, median, or minimum) is equal to or greater than a threshold, the server device 10 performs the remaining processing using the first recognition result. Furthermore, if the statistical value of the acquired score is less than the threshold, the server device 10 acquires the second recognition result and its score, and if the statistical value of the acquired score is greater than the statistical value of the first recognition result, the server device 10 performs the remaining processing using the second recognition result. Furthermore, if the statistical value of the second recognition result is smaller than the statistical value of the first recognition result, the server device 10 performs the remaining processing using the first recognition result. Furthermore, the second recognition result may be acquired before the first recognition result. In either case, the recognition accuracy can be improved compared to when only one type of recognition result is used.

[0125] <Variation: File name creation> When the artificial intelligence module 100 that has performed the machine learning shown in Figure 4 is made to recognize characters from a reading image, the server device 10 may function as an example of a creation unit that creates a file name that includes characters from the recognized characters that represent specified content.

[0126] The predetermined content is, for example, the content to be included in the file name to be given to data containing the recognition character string. The predetermined content is, for example, content indicating whether the document shown by the image to be read is electronic data or scanned paper data, specifically words such as "electronic" or "paper." The predetermined content is also specific information described in the document shown by the image to be read, specifically information such as "product," "service name," "date," or "price."

[0127] Fig. 19 is a diagram showing an example of a scanning image. The scanning image E81 shown in Fig. 19 is an image showing an invoice, and displays biller information F81 of "SSS," order date F82 of "yyyy.mm.dd," product name F83 of "MMMM," ​​and total amount F84 of "￥ZZZ" as predetermined content. For example, the server device 10 creates a file name including these predetermined content, such as "SSS_yyyy.mm.dd_MMMM_ZZZ.xls," which concatenates the biller information F81, order date F82, product name F83, and total amount F84 in that order.

[0128] Server device 10 may function as an example of a generator that generates a file with the created file name. In the example of Figure 19, server device 10 generates a list file that lists, for example, biller information F81, order date F82, product name F83, and total amount F84, and saves the created list file with the created file name. The file saved in this way is used, for example, in the settlement process for advance payments.

[0129] The list file may be generated by the user, in which case the file name is renamed by the server device 10. The file to be renamed is not limited to a newly generated file, but may also be, for example, a file of an image to be scanned. This embodiment can reduce the effort required to save data such as invoices.

[0130] <Variation: Learning Character Position> In the AIOCR function, not only characters but also character positions may be learned. In this case, the server device 10 functions as an example of a position acquisition unit that acquires recognition position information. Here, the recognition position information is information indicating the result of recognition by the artificial intelligence module 100 of the position in the learning image of the character indicated by the identified recognition result.

[0131] The AI ​​module 100 calculates a value representing a feature from the pixel value of the location where the character to be recognized is displayed, and therefore realizes a position output function that outputs the position information of the character in the learning image. In this case, the server device 10 acquires the recognition character string at A23 shown in Fig. 4, for example, and acquires the recognition position information of each character included in the recognition character string.

[0132] The artificial intelligence module 100 may output the location information as follows: For example, the artificial intelligence module 100 is adjusted to realize an object output function that outputs a monochrome image (an image having pixel values ​​of "0" to "1" where "1" represents white and "0" represents black) in which the area where a reading target (e.g., "insurance type") preset as an object of interest is written is as bright white as possible, and the area where the reading target is not written is as dark black as possible.

[0133] Specifically, the artificial intelligence module 100 uses the object output function to output the probability that the reading object is written (for example, a probability expressed as 0 to 1) for each pixel. As a result, the probability value for each pixel is output as recognition position information, such that, for example, pixels around the location where the reading object is written have a value close to "1" and pixels in other locations have a value close to "0". Next, as correct answer information, a monochrome image is prepared in which the areas where the reading object (for example, "insurance type") is written are set to white and areas where it is not written are set to black for each pixel of each image.

[0134] Fig. 20 is a diagram showing an example of correct answer information. The correct answer information G91 shown in Fig. 20 is targeted to read the claimant information F81 of the reading image E81 shown in Fig. 19. The correct answer information G91 is a monochrome image including a rectangular reading area G92 that surrounds the area including the claimant information F81 in white, and a non-reading area G93 that displays the rest in black. Note that in Fig. 20, the correct answer information G91 is shown superimposed on the reading image E81 to make it easier to understand the relationship between the reading area and the correct answer information.

[0135] More specifically, the correct answer information G91 includes the coordinates of the diagonal corners of the rectangular read portion G92 as information indicating the read portion G92. The correct answer information G91 may further be associated with label information ("claimant" in the example of FIG. 20) corresponding to the character string displayed in the portion indicated by the read portion G92. The correct answer information G91 may also include coordinates indicating a plurality of read portions and label information corresponding to each character string displayed in those read portions.

[0136] Then, the server device 10 causes the artificial intelligence module 100 to perform machine learning so that the monochrome image output by the target output function is as close as possible to the monochrome image of the correct answer information. Specifically, in A31 shown in Fig. 4, the server device 10 inputs correct answer information indicating the correct answer of the character string and the correct answer of the position of the character string, such as correct answer information G91, to the artificial intelligence module 100 that realizes the AIOCR function and the target output function.

[0137] As described above, the server device 10 (an example of a learning unit) may cause the artificial intelligence module 100 to perform machine learning based on the identified recognition result, the recognition position information acquired for the character indicated by the recognition result, and the character string included in the learning image and the correct answer information for the position of each character included in the character string. After performing machine learning in this way, the artificial intelligence module 100 outputs the recognized character string recognized by the AIOCR function and the recognition position information of each character output by the target output function.

[0138] By performing the machine learning process as described above, the artificial intelligence module 100 can learn the exact positions of characters recognized from the learning image, improving the accuracy in outputting the positions of recognized characters compared to when position learning is not performed, and making it possible to grasp the positions of recognized characters more accurately. Note that the terms "position of a character" and "location where a character is displayed" are interchangeable.

[0139] The above machine learning not only makes it possible to accurately determine the position of the read string, but also improves the accuracy of the read string. For example, when learning by providing correct answer information indicating that the correct answer for "bill amount" is "100 yen," if the same image contains the string "unit price 100 yen," the machine learning may end up reading the string "100 yen" from the incorrect location, i.e., the location where "unit price" is displayed, rather than "bill amount."

[0140] On the other hand, when machine learning is performed on the artificial intelligence module 100 by providing correct answer information indicating the correct position as described in Fig. 20, the artificial intelligence module 100 learns not only the correct character string "billing amount" being "100 yen", but also the correct location where "100 yen" is written. As a result, even if the character string "unit price 100 yen" is written in the same image, it is possible to train the module to read the character string from the correct location (the location where "billing amount" is written), and it is possible to prevent the module from learning to read the character string from the incorrect location (the location where "unit price" is written).

[0141] <Example of variation: Variation of composition> 1 and the like (overall configuration, hardware configuration, functional configuration, etc.) are merely examples, and other configurations may be used as long as they are not inconvenient for implementation. For example, the server device 10 may be distributed across two or more devices, or may be provided in the form of SaaS (Software as a Service) or a cloud computing system.

[0142] Furthermore, the information processing performed by the server device 10, the user terminal 20, and the administrator terminal 30 may be collectively executed by a device that integrates these. Furthermore, among the processing performed by the server device 10, those that can also be executed by the artificial intelligence module 100 may be executed by the artificial intelligence module 100, and among the processing performed by the artificial intelligence module 100, those that can also be executed by the server device 10 may be executed by the server device 10. In short, as long as the information processing required for the entire AIOCR system 1 is executed, the devices that execute that information processing may have any configuration.

[0143] The output destination of information or data (hereinafter referred to as "information, etc.") may be another device, a display, a memory unit (including an internal memory unit and an external memory unit), an email address, an account of another system, etc. Acquisition of information, etc. includes acquiring information, etc. generated by the device itself, as well as acquiring information, etc. transmitted from another device. The table, etc. (table, database, etc.) in which parameters are associated is not limited to the illustrated table, etc., and the number of parameters may be reduced or increased. Furthermore, information, etc. corresponding to parameters may be obtained using a mathematical formula, a conditional formula, etc., without using a table, etc.

[0144] The above-described embodiments are information processing devices such as the server device 10 and the user terminal 20, and information processing systems such as the AIOCR system 1 including the server device 10 and the user terminal 20, but may also be information processing methods. The information processing methods include the same steps as those executed by the information processing system. The above-described embodiments may also be programs. The programs cause a computer to execute the same steps as those executed by the information processing system.

[0145] <Additional Notes> Furthermore, it may be provided in the following aspects.

[0146] (1) An information processing system having at least one processor, wherein, in an identification step, when an artificial intelligence module recognizes characters from a training image containing a character string and performs machine learning, the processor identifies, from registered elements registered in a dictionary, registered elements representing unit elements obtained by dividing the recognized recognition character string as a recognition result of the character from the training image, and both the unit elements and the registered elements are elements smaller than a subword; and in a learning step, the information processing system causes the artificial intelligence module to perform machine learning based on the identified recognition result and correct answer information for the character string contained in the training image.

[0147] According to this embodiment, it is possible to reduce the number of characters that cannot be recognized.

[0148] (2) In the information processing system described in (1) above, the unit element and the registered element are both elements each having a unit of one character.

[0149] According to this aspect, it is possible to eliminate unrecognizable characters.

[0150] (3) In the information processing system described in (1) or (2) above, in the learning step, the processor causes the artificial intelligence module to perform machine learning using as the learning image an image in which part of a character string is made difficult to recognize, and if the artificial intelligence module makes a misrecognition, causes the next machine learning to be performed using as the learning image an image in which the misrecognized character and characters surrounding the misrecognized character are made difficult to recognize.

[0151] According to this aspect, it is possible to prevent recurrence of erroneous recognition.

[0152] (4) In the information processing system described in any one of (1) to (3) above, in the learning step, the processor causes the artificial intelligence module to perform machine learning using an image in which part of a character string has been made difficult to recognize as the learning image, and performs machine learning by increasing the number of characters that are made difficult to recognize or increasing the difficulty of the parts where the characters that are made difficult to recognize are represented depending on the number of times the machine learning has been performed in the past.

[0153] According to this aspect, it is possible to continuously improve the recognition accuracy.

[0154] (5) In the information processing system described in any one of (1) to (4) above, when the processor uses an image in which part of a character string is made difficult to recognize as the learning image to have the artificial intelligence module perform machine learning in the learning step, the processor makes the character difficult to recognize by hiding part of the character.

[0155] According to this aspect, it is possible to improve the recognition accuracy even for characters that are partially hidden.

[0156] (6) In the information processing system described in any one of (1) to (5) above, in the recognition step, the processor causes an artificial intelligence module that has performed machine learning in the learning step to recognize characters from a reading image including a character string, in the score acquisition step, the artificial intelligence module acquires a score indicating the likelihood of the characters recognized from the reading image, and in the first display control step, controls the display of the reading image and the characters recognized from the reading image in accordance with the score acquired for the characters.

[0157] According to this embodiment, the accuracy of character recognition can be intuitively grasped.

[0158] (7) In the information processing system described in (6) above, in the first display control step, the processor displays the recognized characters in a manner corresponding to the obtained score.

[0159] According to this embodiment, the accuracy of character recognition can be intuitively grasped.

[0160] (8) In the information processing system described in (6) or (7) above, in the first display control step, the processor replaces low-score characters whose obtained scores are less than a threshold with characters that are estimated to be correct and displays them.

[0161] According to this embodiment, it is possible to recognize sentences more accurately.

[0162] (9) In the information processing system described in any one of (6) to (8) above, in the first display control step, if there is a low-score character whose obtained score is less than a threshold, the processor displays candidate characters that are estimated to be correct, and replaces the low-score character with a character selected from the candidates and displays the character.

[0163] According to this embodiment, it is possible to incorporate human judgment into character recognition.

[0164] (10) In the information processing system described in any one of (6) to (9) above, in the first display control step, the processor displays an image indicating the location where the acquired score is below a threshold value, superimposed on the reading image.

[0165] According to this aspect, it is possible to easily grasp parts that are difficult to recognize.

[0166] (11) In the information processing system described in any one of (1) to (10) above, in the recognition step, the processor causes an artificial intelligence module that has performed machine learning in the learning step to recognize characters from a reading image including a character string, and in the second display control step, displays the reading image by superimposing a text image showing the recognized characters on the position of the characters.

[0167] According to this aspect, the characters can be displayed more clearly in the image to be read.

[0168] (12) In the information processing system described in any one of (1) to (11) above, in the first recognition step, the processor causes an artificial intelligence module that has performed machine learning in the learning step to recognize characters from a reading image that includes a character string, and in the second recognition step, causes a character recognition module having an algorithm that recognizes characters from an image that includes a character string to recognize characters from the reading image, and in the adoption step, compares the recognition result in the first recognition step with the recognition result in the second recognition step and adopts the character that is more correct.

[0169] According to this aspect, it is possible to improve the recognition accuracy.

[0170] (13) In the information processing system described in any one of (1) to (12) above, in the recognition step, the processor causes an artificial intelligence module that has performed machine learning in the learning step to recognize characters from a reading image that includes a character string, in the creation step, creates a file name that includes characters from the recognized characters that represent predetermined content, and in the generation step, generates a file with the created file name.

[0171] According to this aspect, it is possible to reduce the effort required to save data including the recognition results.

[0172] (14) In the information processing system described in any one of (1) to (13) above, in the position acquisition step, the processor acquires recognition position information, which indicates the result of the artificial intelligence module recognizing the position in the training image of the character indicated by the identified recognition result, and in the learning step, the processor causes the artificial intelligence module to perform machine learning based on the identified recognition result, the recognition position information acquired for the character indicated by the recognition result, and correct answer information for the character string included in the training image and the position of each character included in the character string.

[0173] According to this aspect, the position of the recognized character can be grasped more accurately.

[0174] (15) In the information processing system described in any one of (1) to (14) above, in the learning step, the processor performs machine learning using an image of a document having predetermined written information as the learning image, in the second recognition step, causes the artificial intelligence module that performed machine learning in the learning step to recognize characters from a reading image representing the document, and in the input step, inputs a string of characters represented by the recognized characters into an input field of a system for inputting the written information.

[0175] According to this embodiment, it is possible to improve the accuracy when transcribing documents into the system.

[0176] (16) An information processing method, comprising the steps of the information processing system according to any one of (1) to (15) above.

[0177] According to this embodiment, it is possible to reduce the number of characters that cannot be recognized.

[0178] (17) A program that causes a computer to execute each step of the information processing system according to any one of (1) to (15) above.

[0179] According to this embodiment, it is possible to reduce the number of characters that cannot be recognized. Of course, this is not the case. Furthermore, the above-described embodiments and modifications may be combined in any desired manner.

[0180] Finally, while various embodiments of the present invention have been described, these are presented by way of example only and are not intended to limit the scope of the invention. The novel embodiments may be embodied in various other forms, and various omissions, substitutions, and modifications may be made without departing from the spirit of the invention. The embodiments and their modifications are intended to be included within the scope and spirit of the invention, as well as within the scope of the inventions and their equivalents as defined in the appended claims. [Explanation of symbols]

[0181] 1: AIOCR system 2: Communication line 10: Server device 11: Control section 20: User terminal 21: Control unit 30: Administrator terminal 31: Control unit 100: Artificial Intelligence Module DB1: Dictionary database DB2: Service Database E21: Image to be read E31: Training image E51: Learning image E61: Training image E71: Image to be read E81: Image to be read

Claims

1. An information processing system including at least one processor, the processor: In the identification step, when the artificial intelligence module recognizes characters from a learning image including a character string and performs machine learning, a registered element representing a unit element obtained by dividing the recognized recognition character string is identified from registered elements registered in a dictionary as a recognition result of the character from the learning image; The unit element and the registered element are both elements smaller than a subword, In the learning step, the artificial intelligence module is caused to perform machine learning based on the identified recognition result and correct answer information of the character string included in the learning image. Information processing system.

2. 2. The information processing system according to claim 1, The unit element and the registered element are both elements with one character as a unit. Information processing system.

3. 2. The information processing system according to claim 1, the processor: In the learning step, an image in which a part of a character string is made difficult to recognize is used as the learning image to cause the artificial intelligence module to perform machine learning, and if the artificial intelligence module makes an erroneous recognition, an image in which the erroneously recognized character and characters surrounding the erroneous character are made difficult to recognize is used as the learning image to cause the next machine learning. Information processing system.

4. 2. The information processing system according to claim 1, the processor: In the learning step, an image in which a part of a character string is made difficult to recognize is used as the learning image to cause the artificial intelligence module to perform machine learning, and the number of characters made difficult to recognize is increased or the difficulty level of the part in which the characters made difficult to recognize are displayed is increased depending on the number of times the machine learning has been performed in the past. Information processing system.

5. 2. The information processing system according to claim 1, the processor: In the learning step, when an image in which a part of a character string is made difficult to recognize is used as the learning image to cause the artificial intelligence module to perform machine learning, the character is made difficult to recognize by hiding a part of the character. Information processing system.

6. 2. The information processing system according to claim 1, the processor: In the recognition step, the artificial intelligence module that has performed the machine learning in the learning step is made to recognize characters from an image to be read that includes a character string; In the score acquisition step, the artificial intelligence module acquires a score indicating the likelihood of the character recognized from the image to be read, In the first display control step, display of the image to be read and characters recognized from the image to be read are controlled in accordance with the score obtained for the characters. Information processing system.

7. 7. The information processing system according to claim 6, the processor: In the first display control step, the recognized character is displayed in a manner corresponding to the acquired score. Information processing system.

8. 7. The information processing system according to claim 6, the processor: In the first display control step, the acquired low-score characters having scores less than a threshold are replaced with characters that are estimated to be correct and displayed. Information processing system.

9. 7. The information processing system according to claim 6, the processor: In the first display control step, when there is a low-score character whose acquired score is less than a threshold, candidates for a character that is estimated to be correct are displayed, and the low-score character is replaced with a character selected from the candidates and displayed. Information processing system.

10. 7. The information processing system according to claim 6, the processor: In the first display control step, an image showing a portion where the acquired character having a score equal to or less than a threshold is displayed is superimposed on the image to be read. Information processing system.

11. 2. The information processing system according to claim 1, the processor: In the recognition step, the artificial intelligence module that has performed the machine learning in the learning step is made to recognize characters from an image to be read that includes a character string; In the second display control step, a text image showing the recognized characters is superimposed on the position of the recognized characters and the image for reading is displayed. Information processing system.

12. 2. The information processing system according to claim 1, the processor: In the first recognition step, the artificial intelligence module that has performed the machine learning in the learning step is made to recognize characters from an image to be read that includes a character string; In the second recognition step, a character recognition module having an algorithm for recognizing characters from an image including a character string is caused to recognize characters from the image to be read; In the adoption step, the recognition result in the first recognition step is compared with the recognition result in the second recognition step, and the more correct character is adopted. Information processing system.

13. 2. The information processing system according to claim 1, the processor: In the recognition step, the artificial intelligence module that has performed the machine learning in the learning step is made to recognize characters from an image to be read that includes a character string; In the creation step, a file name is created that includes characters representing predetermined content from among the recognized characters. In the generating step, a file having the created file name is generated. Information processing system.

14. 2. The information processing system according to claim 1, the processor: In the position acquisition step, recognized position information is acquired, the recognition position information indicates a result of recognition by the artificial intelligence module of a position in the learning image of the character indicated by the specified recognition result, In the learning step, the artificial intelligence module is caused to perform machine learning based on the identified recognition result, the recognition position information acquired for the character indicated by the recognition result, and correct answer information for a character string included in the learning image and the position of each character included in the character string. Information processing system.

15. 2. The information processing system according to claim 1, the processor: In the learning step, machine learning is performed using an image of a document having predetermined written information as the learning image; In a second recognition step, the artificial intelligence module that has performed the machine learning in the learning step recognizes characters from the image to be read that represents the document; In the input step, a character string represented by the recognized characters is input into an input field of the system for inputting the description. Information processing system.

16. An information processing method, comprising: The information processing system according to any one of claims 1 to 15, Information processing methods.

17. A program, A computer is caused to execute each step of the information processing system according to any one of claims 1 to 15. program.

Images