PROGRAM, INFORMATION PROCESSING APPARATUS AND METHOD

Abstract

To provide a technology for reading a received voucher image and presenting additional information about the voucher to a user.SOLUTION: A server of a voucher reading system stores information (2021) of a user, and model information (2022) for determining a kind of a voucher image and an item corresponding to the kind of the voucher image in accordance with an input of the voucher image, and executes, as its function, the steps of: receiving an input of a voucher image from a user (S111); determining a kind of the voucher image on the basis of model information in accordance with the input of the voucher image (S112); determining an item of the voucher image on the basis of the model information according to the kind of the voucher image (S113); determining additional information to guide the user about a voucher related to the voucher image or the item of the voucher image on the basis of information of the user, the kind of the voucher image and the item of the voucher image (S114); and presenting the determined additional information to the user (S115).SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure relates to a program, an information processing device, and a method. [Background technology]

[0002] Optical character recognition (OCR) is a well-known technology that uses optical reading devices such as cameras and scanners to recognize documents printed on paper forms. There is also a well-known technology that uses image analysis to classify form images.

[0003] As a document classification system technique, Patent Document 1 discloses a document classification system technique that analyzes image information of evidence, classifies the evidence, determines a storage destination, and stores the image information of the evidence. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2020 / 136778 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when documentary evidence is scanned using OCR processing, it is sometimes difficult to immediately determine how to process the image information. While this type of processing is possible on mobile devices such as smartphones, it is often difficult to immediately determine how to process the image information, especially when scanning documentary evidence obtained while on the go. For this reason, there has been a demand for a function that provides guidance on subsequent processing and required information when documentary evidence images are scanned.

[0006] Therefore, in this disclosure, a technique for reading an image of a received evidence and presenting additional information about the evidence to a user will be described. [Means for solving the problem]

[0007] According to one embodiment of the present disclosure, there is provided a program to be executed by a computer having a processor and a memory, wherein the memory stores information about a user who is a user and model information that determines the type of a voucher image and items corresponding to the type of voucher image in accordance with the input of the voucher image, and the program causes the processor to execute the following steps: accepting input of the voucher image from the user; determining the type of the voucher image based on the model information in accordance with the input of the voucher image; determining the items of the voucher image based on the model information in accordance with the type of the voucher image; determining additional information that provides the user with information about the voucher or items of the voucher image related to the voucher image based on the user information, the type of the voucher image, and the items of the voucher image; and presenting the determined additional information to the user. [Effects of the Invention]

[0008] According to the present disclosure, model information for determining the type of evidence image and the items corresponding to the type of evidence image is stored in response to input of the evidence image, the type and items of the evidence image are determined from the evidence image, additional information to be provided to the user is determined, and the information is presented to the user. This makes it possible to present additional information providing guidance about the evidence to the user. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing the overall configuration of a voucher reading system 1 according to a first embodiment. [Figure 2] 2 is a block diagram showing the functional configuration of a terminal device 10 constituting the voucher reading system 1. FIG. [Figure 3] 2 is a block diagram showing the functional configuration of a server 20 that constitutes the voucher reading system 1. FIG. [Figure 4] 2 is a diagram showing an example of the data structure of a user database 2021 stored in the server 20. FIG. [Figure 5]10 is a diagram showing an example of the data structure of a voucher model database 2022 stored in the server 20. FIG. [Figure 6] 10 is a flowchart showing an example of the flow of additional information presentation processing by the voucher reading system 1. [Figure 7] 10 is a flowchart showing an example of the flow of a voucher image transmission process performed by the voucher reading system 1. [Figure 8] 10 is a diagram showing an example of a screen in which guide information is added to a voucher image displayed on the terminal device 10. FIG. [Figure 9] 10 is a diagram showing an example of a screen in which input information is added to a voucher image displayed on the terminal device 10. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The following description and drawings are examples for explaining the present invention, and appropriate omissions and simplifications have been made for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural. In the following description, identical components are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed descriptions thereof will not be repeated.

[0011] <Summary> The following describes a voucher reading system according to the present disclosure. The voucher reading system according to the present disclosure is a system that determines the type and items of a voucher, such as an estimate or invoice, from a captured image of the voucher, determines additional information that provides various guidance to the user, and presents it to the user. The voucher reading system according to the present disclosure is configured to provide various guidance when the user holds a mobile device, such as a smartphone, over the voucher. The voucher reading system according to the present disclosure is also a system provided by so-called SaaS (Software as a Service), for example, as an application service installed on a mobile device, such as a smartphone, or as a web service provided by a cloud server, etc.

[0012] Here, the vouchers that form the basis of the voucher image include various documents required for transactions, such as estimates, invoices, receipts, and acceptance letters. The vouchers also include documents required for various applications at companies, such as travel expense applications, vacation applications, and expense applications. Furthermore, the vouchers also include various application documents or documents submitted to public institutions, such as tax returns, financial statements, business start-up notifications, and corporate registration applications.

[0013] As mentioned above, when receiving evidence while out or away from one's desk, one often takes a photo of the document using the camera function of one's smartphone or other device and scans it using OCR processing. Others simply hold the device up to the device and scan it using OCR processing without taking a photo using the camera function. However, there are cases where one cannot immediately decide what to do with the scanned image of the evidence. Even if the image of the evidence is saved on a smartphone or other device, one may forget about it later, or it may become unclear how the document was obtained. For this reason, there was a need for a function that would provide on-the-spot guidance on the next steps and necessary information for the scanned image of the evidence.

[0014] Therefore, the voucher reading system according to the present disclosure stores model information for determining the type of the voucher image and the items corresponding to the type of the voucher image in response to the input of the voucher image, and determines the type and items of the voucher image based on the model information in response to the input of the voucher image.Additional information to be provided to the user is determined based on the user information and the type and items of the voucher image, and is presented to the user.

[0015] Furthermore, the voucher reading system according to the present disclosure provides information about the voucher related to the voucher image or items of the voucher image as additional information. Specifically, the additional information provided to the user is information about what the voucher is, what subsequent processing (delivery, invoice, payment, etc.) is required, where it should be stored, etc.

[0016] Furthermore, the voucher reading system according to the present disclosure provides, as additional information, input information for inputting various user information into blank fields of the voucher related to the voucher image. Specifically, the input information for inputting various user information is various user information to be input for fields that the user needs to input when the voucher is various application forms (user's own attribute information, current location information, schedule information, etc.).

[0017] With the above configuration, when a user receives evidence while away from home or away from their desk, they can receive guidance on how to process the scanned image of the evidence. This makes it possible to understand how to process the evidence after receiving it, where to store it, etc. Furthermore, it is possible to reduce the time and effort required for users to input items that need to be input by themselves on various application forms when away from home, etc.

[0018] <First Embodiment> The following describes the voucher reading system 1. In the following description, for example, when the terminal device 10 accesses the server 20, the server 20 responds with information for generating a screen on the terminal device 10. The terminal device 10 generates and displays a screen based on the information received from the server 20.

[0019] <1 Overall configuration of the document reading system 1> FIG. 1 is a block diagram showing the overall configuration of a voucher reading system 1 according to the first embodiment. As shown in FIG. 1, the voucher reading system 1 includes a plurality of terminal devices (terminal device 10A and terminal device 10B are shown in FIG. 1; hereinafter, they may be collectively referred to as "terminal device 10") and a server 20. The terminal devices 10 and the server 20 are connected to each other via a network 80 so as to be able to communicate with each other. The network 80 is configured as a wired or wireless network. In this embodiment, the server 20 is a web server (including a cloud server), and exchanges information with the terminal device 10 via web pages. In addition, a web page browser for viewing web pages is installed on the terminal device 10, but a dedicated application for providing the services of the server 20 may also be installed so that the web pages can be viewed using the dedicated application.

[0020] The terminal device 10 is a device operated by each user. Here, a user is a person who uses the terminal device 10 to read an image of a voucher (which may be printed on paper or electronic data), which is a function of the voucher reading system 1, and is, for example, an employee of a business company or a sole proprietor. The terminal device 10 is realized by a desktop personal computer (PC), a laptop PC, or the like. Alternatively, the terminal device 10 may be, for example, a tablet compatible with a mobile communication system, a mobile terminal such as a smartphone, or the like.

[0021] The terminal device 10 is communicatively connected to the server 20 via a network 80. The terminal device 10 is connected to the network 80 by communicating with communication devices such as a wireless base station 81 conforming to communication standards such as 4G, 5G, and LTE (Long Term Evolution), and a wireless LAN router 82 conforming to a wireless LAN (Local Area Network) standard such as IEEE (Institute of Electrical and Electronics Engineers) 802.11. As shown as a terminal device 10B in FIG. 1 , the terminal device 10 includes a communication IF (Interface) 12, an input device 13, an output device 14, a memory 15, a storage unit 16, and a processor 19.

[0022] The communication IF 12 is an interface for inputting and outputting signals so that the terminal device 10 can communicate with external devices. The input device 13 is an input device (e.g., a keyboard, a touch panel, a touch pad, a pointing device such as a mouse, etc.) for receiving input operations from a user. The output device 14 is an output device (e.g., a display, a speaker, etc.) for presenting information to a user. The memory 15 is for temporarily storing programs and data processed by the programs, etc., and is a volatile memory such as a DRAM (Dynamic Random Access Memory). The storage unit 16 is a storage device for saving data, such as a flash memory or an HDD (Hard Disc Drive). The processor 19 is hardware for executing an instruction set written in a program, and is composed of an arithmetic unit, registers, peripheral circuits, etc.

[0023] The server 20 is a device that provides a function of the voucher reading system 1, which is to determine the type and items of a voucher from the image of the voucher, determine additional information that provides various guidance to the user, and present it to the user. The server 20 stores model information that determines the type of the image of the voucher and the items corresponding to the type of image of the voucher, in response to the input of the image of the voucher. The server 20 determines the type and items of the image of the voucher based on the model information in response to the input of the image of the voucher. The server 20 also determines the additional information to provide guidance to the user based on the user information and the type and items of the image of the voucher, and presents it to the user.

[0024] The server 20 is a computer connected to a network 80. The server 20 includes a communication IF 22, an input / output IF 23, a memory 25, a storage 26, and a processor 29.

[0025] The communication IF 22 is an interface for inputting and outputting signals so that the server 20 can communicate with external devices. The input / output IF 23 functions as an interface with an input device for receiving input operations from a user and an output device for presenting information to the user. The memory 25 is for temporarily storing programs and data processed by the programs, etc., and is a volatile memory such as a DRAM (Dynamic Random Access Memory). The storage 26 is a storage device for saving data, such as a flash memory or an HDD (Hard Disc Drive). The processor 29 is hardware for executing an instruction set written in a program, and is composed of an arithmetic unit, registers, peripheral circuits, etc.

[0026] <1.1 Configuration of the terminal device 10> FIG. 2 is a block diagram showing the functional configuration of a terminal device 10 constituting the document reading system 1. As shown in FIG. 2, the terminal device 10 includes multiple antennas (antenna 111, antenna 112), wireless communication units (first wireless communication unit 121, second wireless communication unit 122) corresponding to the respective antennas, an operation reception unit 130 (including a keyboard 131 and a mouse 132), an audio processing unit 140, a microphone 141, a speaker 142, a display 150, a storage unit 170, and a control unit 180. The terminal device 10 also has functions and configurations not specifically shown in FIG. 2 (e.g., a battery for storing power, a power supply circuit for controlling the supply of power from the battery to each circuit, etc.). As shown in FIG. 2, the blocks included in the terminal device 10 are electrically connected by a bus or the like. The terminal device 10 shown in FIG. 2 will be described as a mobile terminal such as a tablet or smartphone compatible with a mobile communication system, but may also be a stationary personal computer (PC) or a laptop PC.

[0027] The antenna 111 emits a signal emitted by the terminal device 10 as a radio wave. The antenna 111 also receives a radio wave from space and provides the received signal to the first radio communication unit 121.

[0028] The antenna 112 emits a signal emitted by the terminal device 10 as a radio wave. The antenna 112 also receives a radio wave from space and provides the received signal to the second radio communication unit 122.

[0029] The first wireless communication unit 121 performs modulation / demodulation processing and the like for transmitting and receiving signals via the antenna 111 so that the terminal device 10 can communicate with other wireless devices. The second wireless communication unit 122 performs modulation / demodulation processing and the like for transmitting and receiving signals via the antenna 112 so that the terminal device 10 can communicate with other wireless devices. The first wireless communication unit 121 and the second wireless communication unit 122 are communication modules including a tuner, an RSSI (Received Signal Strength Indicator) calculation circuit, a CRC (Cyclic Redundancy Check) calculation circuit, a high-frequency circuit, etc. The first wireless communication unit 121 and the second wireless communication unit 122 perform modulation / demodulation and frequency conversion of wireless signals transmitted and received by the terminal device 10, and provide the received signals to the control unit 180.

[0030] The operation reception unit 130 has a mechanism for receiving input operations from the user in accordance with the control of the control unit 180. Specifically, the operation reception unit 130 includes a keyboard 131 and a mouse 132. Note that the operation reception unit 130 may be configured as a touch screen that detects the position of the user's contact on the touch panel, for example, by using a capacitive touch panel.

[0031] The keyboard 131 accepts input operations by the user of the terminal device 10. The keyboard 131 is a device for inputting characters, and outputs input character information to the control unit 180 as an input signal.

[0032] The mouse 132 accepts input operations by the user of the terminal device 10. The mouse 132 is a pointing device for selecting an object displayed on the display 150, and outputs position information of the selected object on the screen and information indicating that a button has been pressed as input signals to the control unit 180.

[0033] The audio processing unit 140 modulates and demodulates audio signals under the control of the control unit 180. The audio processing unit 140 modulates a signal provided from the microphone 141 and provides the modulated signal to the control unit 180. The audio processing unit 140 also provides the audio signal to the speaker 142. The audio processing unit 140 is realized, for example, by a processor for audio processing. The microphone 141 accepts audio input and provides an audio signal corresponding to the audio input to the audio processing unit 140. The speaker 142 converts the audio signal provided from the audio processing unit 140 into audio and outputs the audio to the outside of the terminal device 10.

[0034] Display 150 displays data such as images, videos, and text under the control of control unit 180. Display 150 is realized by, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0035] Camera 160 is a device that receives light with a light receiving element and outputs the received light as a captured image in accordance with the control of control unit 180. Camera 160 may be configured, for example, by an imaging device such as a digital camera or a video camera.

[0036] The storage unit 170 is configured with a memory 15 such as a flash memory and a storage unit 16, and stores data and programs used by the terminal device 10. In one aspect, the storage unit 170 stores user information 171 and evidence image information 172.

[0037] The user information 171 is information about a user who uses the terminal device 10 to read a voucher image, which is a function of the voucher reading system 1. The user information includes information for identifying the user (user ID), the name or title of the business operator or person in charge, and the like.

[0038] The voucher image information 172 is information on the voucher image that is the target of the voucher image reading function of the voucher reading system 1, which is performed by the user using the terminal device 10. The voucher image information may be in any data format, such as jpeg, png, bmp, tiff, gif, eps, or svg.

[0039] The control unit 180 is configured by, for example, the processor 19, and controls the operation of the terminal device 10 by reading a program stored in the storage unit 170 and executing instructions included in the program. The control unit 180 performs the functions of an input operation reception unit 181, a transmission / reception unit 182, a notification control unit 183, and a data processing unit 184 by operating in accordance with a program (not shown) that is an application that is stored in advance in the storage unit 170 of the terminal device 10 and installed, for example.

[0040] The input operation receiving unit 181 performs processing to receive input operations by the user via an input device such as the keyboard 131 .

[0041] The transmitting / receiving unit 182 performs processing for the terminal device 10 to transmit and receive data to and from an external device such as the server 20 in accordance with a communication protocol.

[0042] The notification control unit 183 performs processing to present information to the user. The notification control unit 183 performs processing to display a display image on the display 150, processing to output sound from the speaker 142, and the like.

[0043] The data processing unit 184 performs calculations on data input by the terminal device 10 according to a program, and outputs the calculation results to a memory or the like. The data processing unit 184 may be configured to perform some or all of the functions of the server 20, which will be described later, including a voucher image reception module 2033, a voucher image type determination module 2034, a voucher image item determination module 2035, an additional information determination module 2036, a user presentation module 2037, a user input reception module 2038, and a transmission module 2039, and may be executed by an application installed in the terminal device 10, for example. That is, these functions may be executed by the terminal device 10 or the server 20.

[0044] <1.2 Functional configuration of server 20> 3 is a block diagram showing the functional configuration of the server 20 constituting the voucher reading system 1. As shown in FIG. 3, the server 20 functions as a communication unit 201, a storage unit 202, and a control unit 203.

[0045] The communication unit 201 performs processing for the server 20 to communicate with external devices.

[0046] The storage unit 202 is composed of, for example, a memory 25 and a storage 26, and stores data and programs used by the server 20. The storage unit 202 stores a user database 2021, a voucher model database 2022, and the like.

[0047] The user database 2021 is a database for storing information on users who use the evidence reading system 1, specifically, attribute information on users who use the evidence reading system 1 to read evidence images, which is a function of the evidence reading system 1, and various information such as the types of evidence images that have been read in the past using the evidence reading system 1. Details will be described later.

[0048] The evidence model database 2022 is a database for storing model information for determining the type of evidence image read by the evidence reading system 1 and items corresponding to the type of evidence image. As a specific example, the evidence model database 2022 may be a machine learning model that determines and outputs the type of evidence image and items corresponding to the type of evidence image from image data of a photograph of a document or the like. This machine learning model may include, for example, an OCR model, an object detection model, R-CNN, Fast R-CNN, Faster R-CNN, etc. as implementation examples. Details will be described later.

[0049] The control unit 203 performs the functions shown in various modules, such as a receiving control module 2031, a sending control module 2032, a voucher image reception module 2033, a voucher image type determination module 2034, a voucher image item determination module 2035, an additional information determination module 2036, a user presentation module 2037, a user input reception module 2038, and a sending module 2039, by the processor 29 of the server 20 performing processing according to the program.

[0050] The reception control module 2031 controls the process by which the server 20 receives signals from external devices in accordance with a communication protocol.

[0051] The transmission control module 2032 controls the process in which the server 20 transmits signals to external devices in accordance with a communication protocol.

[0052] The voucher image receiving module 2033 controls the process of receiving the input of a voucher image from the user of the voucher reading system 1. The voucher image receiving module 2033 receives the input of the captured voucher image when the camera 160 photographs the voucher in response to the user's operation of the terminal device 10. At this time, the user may photograph the voucher with the camera 160 of the terminal device 10 to receive the input of the voucher image, or may hold the voucher over the camera 160 while it is controlled by a dedicated application of the terminal device 10 to receive the input of the voucher image.

[0053] The voucher images accepted by the voucher image acceptance module 2033 include photographed images of various documents necessary for transactions, such as estimates, invoices, receipts, and receipts, as described above. The voucher images also include documents necessary for various applications at companies, etc., such as travel expense applications, vacation applications, and expense applications. Furthermore, the voucher images also include various application documents or submission documents for public institutions, etc., such as tax returns, financial statements, business start-up notifications, and corporate registration applications. Information on the voucher images may be accepted in any data format, such as PDF (Portable Document Format), Word, XML (eXtensible Markup Language), jpeg, png, bmp, tiff, gif, eps, and svg.

[0054] The voucher image type determination module 2034 controls the process of determining the type of the voucher image based on the model information stored in the voucher model database 2022, in response to the input of the voucher image accepted by the voucher image acceptance module 2033. The voucher image type determination module 2034, for example, reads the accepted voucher image using OCR processing and determines the type of the voucher image. Specific types of voucher images include the above-mentioned estimates, invoices, receipts, acceptance notes, travel expense claims, vacation claims, expense claims, tax returns, financial statements, business start-up notifications, and corporate registration applications.

[0055] The voucher image type determination module 2034 reads the content of the character string described in the voucher image and interprets the meaning of the character string, for example, using an OCR model or the like, which is model information stored in the voucher model database 2022. Furthermore, the voucher image type determination module 2034 acquires position information of the character string in the voucher image, for example, using an object detection model or the like, which is model information stored in the voucher model database 2022, and determines the position indicated by the character string in the voucher image, for example, whether it is a title, an item name, the main text, etc. The voucher image type determination module 2034 determines the type of the voucher image through these processes.

[0056] Furthermore, the voucher image type determination module 2034 may determine the type of the voucher image based on the type of the voucher image that the user has previously read using the voucher reading system 1, which is stored in the user database 2021.

[0057] The voucher image type determination module 2034 stores the determination result in the user database 2021.

[0058] The voucher image item determination module 2035 controls the process of determining the items of the voucher image based on the model information stored in the voucher model database 2022, depending on the type of voucher image determined by the voucher image type determination module 2034. The voucher image item determination module 2035 determines the content of each item described in the voucher, for example, for the voucher image read by the voucher image type determination module 2034 through OCR processing. The items of the voucher image specifically include the title, issuer, name, etc. of the voucher image, as well as the item number, amount, etc. in estimates, invoices, receipts, acceptances, travel expense claims, etc.

[0059] The voucher image item determination module 2035 interprets the meaning of a character string from the content of the character string described in the voucher image, for example, read by the voucher image type determination module 2034 through OCR processing. In addition, the voucher image item determination module 2035 acquires position information of the character string in the voucher image, for example, using an object detection model, which is model information stored in the voucher model database 2022, and determines the position indicated by the character string in the voucher image, for example, whether it is a title, an item name, the main text, etc. The voucher image type determination module 2034 determines the items of the voucher image through these processes.

[0060] In addition, the document image item determination module 2035 may determine the items of the document image based on the types and items of document images that the user has previously read using the document reading system 1, which are stored in the user database 2021.

[0061] The voucher image item determination module 2035 stores the determination result in the user database 2021.

[0062] The additional information determination module 2036 controls the process of determining additional information to be provided to the user based on user information stored in the user database 2021, the type of voucher image determined by the voucher image type determination module 2034, and the items of the voucher image determined by the voucher image item determination module 2035. The additional information determination module 2036 may determine the additional information to be provided to the user based on, for example, model information stored in the voucher model database 2022. The additional information determination module 2036 may also determine the additional information based on information on the type and items of voucher images that the user has previously read using the voucher reading system 1, which information is stored in the user database 2021.

[0063] The additional information determined by the additional information determination module 2036 includes, for example, information to inform the user about the items of the voucher or its image. Specifically, the information to inform the user includes information such as the position of the voucher in transactions and various procedures, when it will be issued, the contents of each item, the confirmations that the user must make regarding the voucher, the follow-up processing that the user must perform regarding the voucher, and the deadlines for confirmation and follow-up processing.

[0064] In this case, the additional information determined by the additional information determination module 2036 may be, for example, link information to a web page that provides information about the voucher. Also, the additional information determined by the additional information determination module 2036 may be, for example, storage destination information that indicates a storage destination for the information on the voucher image.

[0065] The additional information determined by the additional information determination module 2036 includes, for example, input information for blank items in the received voucher image when some or all of the items are blank. The input information for blank items in the voucher image is the name of the user when the item is "name," and is the transportation used by the user (route name, departure point, arrival point, etc.) when the item is "transportation."

[0066] At this time, the additional information determined by the additional information determination module 2036 may be, for example, user information (such as name) stored in the user database 2021. Furthermore, the additional information determined by the additional information determination module 2036 may be, for example, information acquired from the terminal device 10, or location information (corresponding to user location information) acquired from the terminal device 10.

[0067] The additional information determination module 2036 stores the determined additional information in the user database 2021.

[0068] The user presentation module 2037 controls the process of transmitting the additional information determined by the additional information determination module 2036 to the terminal device 10 and presenting it to the user. The user presentation module 2037 transmits the additional information to the terminal device 10 via the communication unit 201.

[0069] As described above, when the function is executed on the terminal device 10, the user presentation module 2037 may not send or receive additional information, but may instead display the additional information on the display 150 and present it to the user through processing by the notification control unit 183 of the terminal device 10.

[0070] The user input receiving module 2038 controls the process of receiving input information from the user regarding the additional information presented to the user by the user presentation module 2037, through the user's operation of the terminal device 10. The input information regarding the additional information includes, for example, information regarding the confirmation result that the user should perform on the voucher, information regarding subsequent processing that the user should perform on the voucher, instruction information for selecting link information to a web page, and instruction information for transitioning to a storage destination for the information in the voucher image. Furthermore, the input information regarding the additional information includes, for example, information for correcting the input information for blank items in the voucher image, and instruction information for processing the voucher (for example, sending to a specified destination).

[0071] The transmission module 2039 controls the process of transmitting information on the evidence image based on the input information received by the user input reception module 2038. The transmission module 2039 transmits the information on the evidence image to, for example, a storage destination of the information on the evidence image or a predetermined transmission destination via the communication unit 201 (or the transmission / reception unit 182).

[0072] <2 Data Structure> Fig. 4 is a diagram showing an example of the data structure of a user database 2021 stored in the server 20. Fig. 5 is a diagram showing an example of the data structure of a voucher model database 2022 stored in the server 20.

[0073] As shown in FIG. 4, each record of the user database 2021 includes an item "user ID," an item "user name," an item "user past evidence information," and the like.

[0074] The item "user ID" is information for identifying each user who uses the document reading system 1 to read the document image.

[0075] The item "user name" is the name or title of the user who uses the documented evidence reading system 1 to read the documented evidence image.

[0076] The item "User past evidence information" is information on the reading of evidence images performed in the past by a user of the evidence reading system 1, and specifically includes an item "Evidence ID", an item "Evidence name", an item "Item information", "Additional information", etc. The item "User past evidence information" stores information on the reading of evidence images performed in the past by the user, for example, in chronological order, linked to information on the user identified by the item "User ID".

[0077] The item “voucher ID” is information for identifying each type of voucher image read by the voucher reading system 1.

[0078] The item "voucher name" is a name indicating the type of voucher image read and determined by the voucher reading system 1. The item "voucher name" stores information such as "quotation," "invoice," and "travel expense application" according to the type of voucher image determined by the voucher image type determination module 2034.

[0079] The item "item information" is a name indicating the item of the voucher image that is read and determined by the voucher reading system 1. The item "item information" stores information such as "item," "issuer," and "route information" according to the item of the voucher image that is determined by the voucher image item determination module 2035.

[0080] The item "additional information" is additional information determined by the voucher reading system 1, and the names of the additional information are shown as examples in Fig. 4. The item "additional information" stores additional information determined by the additional information determination module 2036 according to the type or item of the voucher image, such as "expense settlement," "past amount," and "Tokyo - Shinagawa." As examples of additional information shown in Fig. 4, the additional information "expense settlement" indicates that the user is informed that the item on the receipt is subject to expense settlement, and the additional information "Tokyo - Shinagawa" indicates that the user is informed of the content that should be entered in the route information on the travel expense application form.

[0081] The voucher image type determination module 2034 of the server 20 adds a record for the item "User past voucher information" in conjunction with determining the type of the voucher image (if this is the first time the user is using the voucher reading system 1, a record is added to the user database 2021). The voucher image item determination module 2035 updates the item "item information" corresponding to the relevant item "voucher ID" or adds a record for the item "user past voucher information" in conjunction with determining the item of the voucher image. The additional information determination module 2036 updates the item "additional information" corresponding to the relevant item "voucher ID" in conjunction with determining the additional information.

[0082] As shown in FIG. 5, each record of the evidence model database 2022 includes an item "evidence ID," an item "evidence name," an item "item management information," and the like.

[0083] The item “evidence ID” is information that identifies each type of evidence image read by the evidence reading system 1, and corresponds to the item “evidence ID” in the user database 2021.

[0084] The item “evidence name” is a name indicating the type of evidence image that is read and determined by the evidence reading system 1, and corresponds to the item “evidence name” in the user database 2021.

[0085] The item "item management information" is additional information presented to the user for the items of the evidence image in the evidence reading system 1, and specifically includes the item "item information," the item "confirmation content," the item "input information," and "additional information." The item "item management information" stores additional information to be presented to the user for each item, linked to the information of the evidence image identified by the item "evidence ID." Note that the item "item management information" may store additional information to be presented to the user for the entire evidence, linked to the information of the evidence image identified by the "evidence ID," without being linked to a specific item of the evidence.

[0086] The item “item information” is a name indicating the item of the document image that is read and determined by the document reading system 1, and corresponds to the item “item information” in the user database 2021.

[0087] The item "confirmation content" is information that guides the user about the items of the evidence image among the additional information presented for the item "item information." The item "confirmation content" stores information that prompts the user to confirm, for example, information that guides the user to check past transactions and amounts for "item" and "amount," as additional information determined by the additional information determination module 2036 according to the type or item of the evidence image.

[0088] The item "input information" is the input information for the blank item of the voucher image among the additional information presented for the item "item information" when the item is blank. The item "input information" stores input information for the blank item, for example, information to input a user name for "name", as additional information determined by the additional information determination module 2036 according to the type or item of the voucher image.

[0089] The item “additional information” is additional information determined in the document reading system 1 and corresponds to the item “additional information” in the user database 2021.

[0090] Although not shown in the drawings, model information for determining the type of evidence image to be read and items according to the type of evidence image may be stored in the evidence model database 2022. For example, the evidence model database 2022 may store, as model information, a machine learning model for inferring and determining the feature amounts of the evidence and each item of the evidence image and the type and items of the corresponding evidence image.

[0091] <3 operations> 6 and 7, the additional information presentation process and the voucher image transmission process performed by the voucher reading system 1 in the first embodiment will be described. Note that in the description of the operations, an example will be described in which the terminal device 10 executes each function of the server 20.

[0092] FIG. 6 is a flowchart showing an example of the flow of the additional information presentation process performed by the voucher reading system 1.

[0093] In step S111, the user operates the camera 160 of the terminal device 10 to photograph the document or to obtain (photograph) an image of the document by holding it over the device, and the data processing unit 184 accepts the input of the document image using the function of the document image reception module 2033.

[0094] In step S112, the data processing unit 184 of the terminal device 10, using the function of the evidence image type determination module 2034, determines the type of the evidence image based on the model information (which may be read from the evidence model database 2022 via communication with the server 20, or may be model information stored in the terminal device 10) in response to the input of the evidence image accepted in step S111.

[0095] In step S113, the data processing unit 184 of the terminal device 10, using the function of the evidence image item determination module 2035, determines the items of the evidence image based on the model information (which may be read from the evidence model database 2022 through communication with the server 20, or may be model information stored in the terminal device 10) depending on the type of evidence image determined in step S112.

[0096] In step S114, the data processing unit 184 of the terminal device 10 determines additional information to be provided to the user based on the type of the evidence image, which is the determination result determined in step S112, and the items of the evidence image, which are the determination result determined in step S113, by the function of the additional information determination module 2036. The additional information to be provided to the user may include information to be provided to the user about the evidence or items of the evidence image, or may include input information for blank items in the evidence image.

[0097] In step S115, the data processing unit 184 of the terminal device 10 uses the function of the user presentation module 2037 to display the additional information determined in step S114 on the display 150 and present it to the user.

[0098] As described above, the voucher reading system 1 stores model information for determining the type of voucher image and the items corresponding to the type of voucher image in response to the input of the voucher image, and determines the type and items of the voucher image based on the model information in response to the input of the voucher image. Additional information to be provided to the user is determined based on the user information and the type and items of the voucher image, and is presented to the user. This allows the user to obtain additional information about the voucher, enabling them to understand how to process the voucher after receiving it, where to store it, and so on. Furthermore, it also reduces the effort required for the user to manually enter items that need to be entered for various application forms when away from the office, etc.

[0099] FIG. 7 is a flowchart showing an example of the flow of the evidence image transmission process performed by the evidence reading system 1.

[0100] In step S211, the input operation accepting unit 181 of the terminal device 10 accepts input of input information for the additional information presented in step S115 through an operation by the user. The data processing unit 184 of the terminal device 10 accepts input information from the user for the additional information presented to the user through the function of the user input accepting module 2038.

[0101] In step S212, the data processing unit 184 of the terminal device 10 transmits information about the evidence image based on the input information accepted in step S211 using the function of the transmission module 2039. Here, an example will be described in which information about the evidence image is transmitted to the server 20 and stored in a predetermined storage location. In step S212, information about the evidence image is transmitted to the server 20 via the transmission / reception unit 182.

[0102] In step S222, the server 20 receives the information on the evidence image transmitted in step S212 via the communication unit 201, and stores the information in a predetermined storage location within the server 20.

[0103] As described above, the voucher reading system 1 accepts input information from the user regarding additional information and transmits information on the voucher image. This allows various information to be input for the voucher image of the read voucher, and then transmitted to a server or the like for subsequent processing, thereby reducing the effort required for input by the user.

[0104] <4 Screen example> Hereinafter, examples of screens for the additional information presentation process by the voucher reading system 1 will be described with reference to FIGS.

[0105] Fig. 8 is a diagram showing an example of a screen in which guidance information is added to the evidence image displayed on the terminal device 10. The example of the screen in Fig. 8 shows an example of a screen in which additional information determined by the function of the additional information determination module 2036 is presented together with the evidence image received by the function of the evidence image reception module 2033 by the function of the user presentation module 2037. This corresponds to step S115 in Fig. 6.

[0106] 8, a voucher image 1501 is displayed on the display 150 of the terminal device 10. In addition to the voucher image 1501, additional information 1502 and 1503 are displayed on the display 150 so as to be superimposed on the voucher image 1501.

[0107] 8, the voucher image 1501 is an invoice, and the function of the voucher image type determination module 2034 reads the item "voucher name" stored in the voucher model database 2022 and determines that it is an invoice. Furthermore, the function of the voucher image item determination module 2035 reads the item "item management information" stored in the voucher model database 2022 and determines that the item is the item of the voucher image stored in that item. Then, the function of the additional information determination module 2036 presents the item "additional information" stored in the voucher model database 2022 to the user.

[0108] In the example shown in Fig. 8, link information to the storage destination is shown in additional information 1502 as storage destination information indicating where the invoice is stored. Also, information informing the user of what they should check regarding the total amount in the invoice is shown in additional information 1503. This allows the user to obtain additional information about the voucher, making it possible for them to understand what they should check regarding the voucher, how to process the voucher after receiving it, where to store it, etc.

[0109] Fig. 9 is a diagram showing an example of a screen in which input information is added to a voucher image displayed on the terminal device 10. The example of the screen in Fig. 9 shows an example of a screen in which additional information determined by the function of the additional information determination module 2036 is presented together with the voucher image received by the function of the voucher image reception module 2033 by the function of the user presentation module 2037. This corresponds to step S115 in Fig. 6.

[0110] 9, a voucher image 1601 is displayed on the display 150 of the terminal device 10. In addition to this voucher image 1601, input information 1602, 1603, 1604, and 1605 for blank items in the voucher image 1601 is also displayed on the display 150 so as to fill in the blanks in the voucher image 1601.

[0111] 9, the voucher image 1601 is a travel expense application form, and the function of the voucher image type determination module 2034 reads the item "voucher name" stored in the voucher model database 2022 and determines that it is a travel expense application form. Furthermore, the function of the voucher image item determination module 2035 reads the item "item management information" stored in the voucher model database 2022 and determines that the item is the item of the voucher image stored in that item and is input information. Then, the function of the additional information determination module 2036 presents the item "additional information" stored in the voucher model database 2022 to the user as input information.

[0112] In the example shown in FIG. 9, input information 1602 is shown as information for inputting the user's department name into the department name input field. Input information 1603 is shown as information for inputting the user's name into the name input field. Input information 1604 is shown as information for inputting the transportation means used by the user (departure point, arrival point) into the transportation means input field. Furthermore, input information 1605 is shown as information for inputting the transportation costs of the transportation means used by the user into the transportation expense amount input field. This allows the user to obtain input information as additional information about the voucher, thereby reducing the effort required for the user to enter items that need to be entered by the user on various application forms.

[0113] <Summary> As described above, according to this embodiment, model information for determining the type of evidence image and the items corresponding to the type of evidence image is stored in response to the input of the evidence image, and the type and items of the evidence image are determined based on the model information in response to the input of the evidence image. Additional information to be provided to the user is determined based on the user information and the type and items of the evidence image and is presented to the user. This allows the user to obtain additional information about the evidence, making it possible to understand how to process the evidence after receiving it, where to store it, etc.

[0114] Furthermore, according to this embodiment, additional information is provided about the voucher related to the image of the voucher or the items of the image of the voucher. Specifically, the additional information provided to the user is information about what the voucher is, what subsequent processing (delivery, invoice, payment, etc.) is required, where it should be stored, etc. This makes it possible to understand what processing to do after receiving the voucher, where it should be stored, etc.

[0115] Furthermore, according to this embodiment, as additional information, input information for inputting various user information into blank fields of the evidence related to the evidence image is provided. Specifically, the input information for inputting various user information is various user information (user's own attribute information, current location information, schedule information, etc.) that should be input into fields that the user needs to input when the evidence is various application forms. This makes it possible to reduce the effort required for the user to input fields that the user needs to input into various application forms.

[0116] Although the embodiments of the disclosure have been described above, they can be implemented in various other forms and can be implemented with various omissions, substitutions, and modifications. These embodiments and modifications, as well as those with omissions, substitutions, and modifications, are included in the technical scope of the claims and their equivalents.

[0117] <Additional Notes> The matters described in the above embodiments will be supplemented below.

[0118] (Appendix 1) A program to be executed by a computer having a processor 29 and a memory 25, wherein the memory 25 stores user information (2021) and model information (2022) for determining the type of a voucher image and items corresponding to the type of the voucher image in accordance with the input of the voucher image, and the program causes the processor 29 to execute the following steps: accepting input of the voucher image from the user (S111); determining the type of the voucher image based on the model information in accordance with the input of the voucher image (S112); determining the items of the voucher image based on the model information in accordance with the type of the voucher image (S113); determining additional information that provides the user with information about the voucher or items of the voucher image related to the voucher image based on the user information, the type of the voucher image, and the items of the voucher image (S114); and presenting the determined additional information to the user (S115).

[0119] (Appendix 2) The memory 25 stores information on evidence images previously received from the user as user information, and in the step of determining additional information, determines the additional information based on the information on evidence images previously received from the user, (Appendix 1) program.

[0120] (Appendix 3) A program described in (Appendix 1) or (Appendix 2), in which, in the step of determining additional information, link information to a web page providing information about the voucher related to the voucher image is determined as information providing information about the voucher, and in the step of presenting it to the user, the web page to which the link information links is presented to the user.

[0121] (Appendix 4) A program described in (Appendix 1) or (Appendix 2), in which, in the step of determining additional information, storage location information indicating the storage location of the information of the evidence image is determined as the additional information, and in the step of presenting it to the user, the storage location of the information of the evidence image related to the storage location information is presented to the user.

[0122] (Supplementary Note 5) The program according to (Supplementary Note 1), further causing the program to execute a step (S211) of accepting input information from the user in response to the additional information presented to the user.

[0123] (Supplementary Note 6) The program according to (Supplementary Note 5), further causing the program to execute a step (S212) of transmitting information on the evidence image based on input information from a user.

[0124] (Appendix 7) An information processing device comprising a control unit 203 and a memory unit 202, wherein the memory unit 202 stores user information (2021) and model information (2022) for determining the type of a voucher image and items corresponding to the type of the voucher image according to the input of the voucher image, and the control unit 203 executes the steps of: accepting input of a voucher image from the user (S111); determining the type of the voucher image based on the model information according to the input of the voucher image (S112); determining the items of the voucher image based on the model information according to the type of the voucher image (S113); determining additional information for informing the user about the voucher or items of the voucher image related to the voucher image based on the user information, the type of the voucher image, and the items of the voucher image (S114); and presenting the determined additional information to the user (S115).

[0125] (Appendix 8) A method to be executed by a computer having a processor 29 and a memory 25, wherein the memory 25 stores user information (2021) and model information (2022) for determining the type of a voucher image and items corresponding to the type of the voucher image in accordance with the input of the voucher image, and the method comprises the steps of the processor 29 executing a step (S111) of accepting input of the voucher image from the user, a step (S112) of determining the type of the voucher image based on the model information in accordance with the input of the voucher image, a step (S113) of determining the items of the voucher image based on the model information in accordance with the type of the voucher image, a step (S114) of determining additional information that provides the user with information about the voucher or items of the voucher image related to the voucher image based on the user information, the type of the voucher image, and the items of the voucher image, and a step (S115) of presenting the determined additional information to the user. [Explanation of symbols]

[0126] 1: Voucher reading system 10: Terminal device 10A: Terminal equipment 10B: Terminal device 13: Input device 14: Output device 15: Memory 16: Storage section 19: Processor 20: Server 25: Memory 26: Storage 29: Processor 80: Network 81: Wireless base station 82: Wireless LAN router 111: Antenna 112: Antenna 121: First wireless communication unit 122: Second wireless communication unit 130: Operation reception unit 131: Keyboard 132: Mouse 140: Audio processing unit 141:Mike 142: Speaker 150: Display 160: Camera 170: Storage section 171: User information 172: Evidence image information 180: Control unit 181: Input operation reception unit 182: Transmitter / receiver 183: Notification control section 184: Data processing section 201: Communications Department 202: Storage section 203: Control unit 2021: User Database 2022: Evidence model database 2031: Reception control module 2032: Transmission control module 2033: Voucher image reception module 2034: Voucher image type determination module 2035: Voucher image item determination module 2036: Additional information determination module 2037: User presentation module 2038: User input acceptance module 2039: Transmission module

Claims

1. A program to be executed by a computer having a processor and a memory, The memory includes information of a user, storing model information for determining a type of the evidence image and an item corresponding to the type of the evidence image in response to the input of the evidence image; The program causes the processor to: receiving an input of the evidence image from the user; determining a type of the evidence image based on the model information in response to the input of the evidence image; determining an item of the evidence image based on the model information according to the type of the evidence image; determining additional information for informing the user about the evidence related to the evidence image or the items of the evidence image based on the user information, the type of the evidence image, and the items of the evidence image; presenting the determined additional information to the user; The memory stores information of the evidence image previously received from the user as the information of the user; In the step of determining the type of the evidence image, the type of the evidence image is determined based on information of the evidence image previously received from the user and read and the model information; In the step of determining the items of the evidence image, the items of the evidence image are determined based on information of the evidence image previously received from the user and read and the model information; In the step of determining the additional information, a program determines the additional information, including information regarding subsequent processing that the user should perform on the evidence related to the evidence image or information regarding confirmation that the user should perform, based on information about the evidence image that was previously received and read from the user.

2. In the step of determining the additional information, link information to a Web page providing information about the evidence related to the evidence image is determined; 2. The program according to claim 1, wherein in the step of presenting to the user, the Web page that is the link destination of the link information is presented to the user.

3. In the step of determining the additional information, storage destination information indicating a storage destination of information of the evidence image is determined as the additional information; 2. The program according to claim 1, wherein in the step of presenting to the user, a storage location of information of the evidence image related to the storage location information is presented to the user.

4. The program further comprises:

2. The program according to claim 1, further comprising: a step of receiving input information from the user in response to the additional information presented to the user.

5. The program further comprises: The program according to claim 4 , further comprising a step of transmitting information about the evidence image based on input information from the user.

6. An information processing device including a control unit and a storage unit, The storage unit includes information of a user who is a user, storing model information for determining a type of the evidence image and an item corresponding to the type of the evidence image in response to the input of the evidence image; The control unit is receiving an input of the evidence image from the user; determining a type of the evidence image based on the model information in response to the input of the evidence image; determining an item of the evidence image based on the model information according to the type of the evidence image; determining additional information for informing the user about the evidence related to the evidence image or the items of the evidence image based on the user information, the type of the evidence image, and the items of the evidence image; presenting the determined additional information to the user; The storage unit stores information on the evidence image previously received from the user as the information on the user, In the step of determining the type of the evidence image, the type of the evidence image is determined based on information of the evidence image previously received from the user and read and the model information; In the step of determining the items of the evidence image, the items of the evidence image are determined based on information of the evidence image previously received from the user and read and the model information; In the step of determining the additional information, an information processing device determines the additional information, including information regarding subsequent processing that the user should perform on the evidence related to the evidence image or information regarding confirmation that the user should perform, based on information about the evidence image that was previously received and read from the user.

7. 1. A method for performing a process on a computer having a processor and a memory, comprising: The memory includes information of a user, storing model information for determining a type of the evidence image and an item corresponding to the type of the evidence image in response to the input of the evidence image; The method further comprises the processor: receiving an input of the evidence image from the user; determining a type of the evidence image based on the model information in response to the input of the evidence image; determining an item of the evidence image based on the model information according to the type of the evidence image; determining additional information for informing the user about the evidence related to the evidence image or the items of the evidence image based on the user information, the type of the evidence image, and the items of the evidence image; presenting the determined additional information to the user; The memory stores information of the evidence image previously received from the user as the information of the user; In the step of determining the type of the evidence image, the type of the evidence image is determined based on information of the evidence image previously received from the user and read and the model information; In the step of determining the items of the evidence image, the items of the evidence image are determined based on information of the evidence image previously received from the user and read and the model information; A method in which, in the step of determining the additional information, the additional information is determined based on information of the evidence image previously received and read from the user, the information including information regarding subsequent processing that the user should perform on the evidence related to the evidence image or information regarding confirmation that the user should perform.

Images