Program, information processing device and method

Abstract

To provide a technology for reading a received voucher image and presenting additional information about the voucher to a user.SOLUTION: A server of a voucher reading system stores information (2021) of a user, and model information (2022) for determining a kind of a voucher image and an item corresponding to the kind of the voucher image in accordance with an input of the voucher image, and executes, as its function, the steps of: receiving an input of a voucher image from a user (S111); determining a kind of the voucher image on the basis of model information in accordance with the input of the voucher image (S112); determining an item of the voucher image on the basis of the model information according to the kind of the voucher image (S113); determining additional information to guide the user about a voucher related to the voucher image or the item of the voucher image on the basis of information of the user, the kind of the voucher image and the item of the voucher image (S114); and presenting the determined additional information to the user (S115).SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present disclosure relates to a program, an information processing device, and a method. [Background technology]

[0002] Optical character recognition (OCR) is a well-known technology that uses an optical reader such as a camera or scanner to recognize documents printed on paper forms. There is also a known technology that classifies form images by image analysis.

[0003] As a technology for a document classification system, Patent Document 1 discloses a technology for a document classification system that analyzes image information of evidence, classifies the evidence, determines a storage destination, and stores the image information of the evidence. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2020 / 136778 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when documentary evidence is scanned using OCR processing, there are cases where it is not possible to immediately decide how to process the image information. Although such processing is possible on mobile devices such as smartphones, it is often not possible to immediately decide how to process the image information, especially when scanning documentary evidence obtained while on the go. For this reason, there has been a demand for a function that provides guidance on subsequent processing and required information when documentary evidence images are scanned.

[0006] Therefore, in this disclosure, a technique will be described in which a received evidence image is read and additional information about the evidence is presented to a user. [Means for solving the problem]

[0007] According to one embodiment of the present disclosure, a program to be executed on a computer having a processor and a memory, wherein the memory stores information of a user who is a user, and model information that determines the type of the evidence image and items corresponding to the type of the evidence image in response to the input of the evidence image, and the program causes the processor to execute the following steps: accepting input of the evidence image from the user; determining the type of the evidence image based on the model information in response to the input of the evidence image; determining the items of the evidence image based on the model information in response to the type of the evidence image; determining additional information that provides the user with information about the evidence or items of the evidence image related to the evidence image based on the user information, the type of the evidence image, and the items of the evidence image; and presenting the determined additional information to the user. Effect of the Invention

[0008] According to the present disclosure, model information for determining the type of the evidence image and the items corresponding to the type of the evidence image is stored in response to the input of the evidence image, the type and items of the evidence image are determined from the evidence image, additional information to be provided to the user is determined, and the additional information is presented to the user. This makes it possible to present the additional information to provide information about the evidence to the user. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an overall configuration of a voucher reading system 1 according to a first embodiment. [Diagram 2] 2 is a block diagram showing a functional configuration of a terminal device 10 constituting the voucher reading system 1. FIG. [Diagram 3] 2 is a block diagram showing a functional configuration of a server 20 constituting the voucher reading system 1. FIG. [Figure 4] 2 is a diagram showing an example of the data structure of a user database 2021 stored in the server 20. FIG. [Diagram 5]13 is a diagram showing an example of the data structure of a voucher model database 2022 stored in the server 20. FIG. [Figure 6] 13 is a flowchart showing an example of a flow of additional information presentation processing performed by the voucher reading system 1. [Figure 7] 13 is a flowchart showing an example of a flow of a voucher image transmission process performed by the voucher reading system 1. [Figure 8] 13 is a diagram showing an example of a screen in which guide information is added to a voucher image displayed on the terminal device 10. FIG. [Figure 9] 13 is a diagram showing an example of a screen in which input information is added to a voucher image displayed on the terminal device 10. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. The following description and drawings are examples for explaining the present invention, and are omitted and simplified as appropriate for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural. In the following description, the same parts are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed descriptions thereof will not be repeated.

[0011] <Summary> The document reading system according to the present disclosure will be described below. The document reading system according to the present disclosure is a system for determining the type and items of a document such as an estimate or invoice from an image of the document, determining additional information for providing various kinds of guidance to the user, and presenting the information to the user. The document reading system according to the present disclosure is configured to provide various kinds of guidance when the user holds a mobile terminal such as a smartphone over the document. The document reading system according to the present disclosure is a system provided by so-called SaaS (Software as a Service), for example, as an application service installed on a mobile terminal such as a smartphone, or as a Web service by a cloud server or the like.

[0012] Here, the vouchers that are the basis of the voucher image include various documents necessary for transactions, such as estimates, invoices, receipts, receipts, etc. The vouchers also include documents necessary for various applications in companies, etc., such as travel expense applications, vacation applications, expense applications, etc. Furthermore, the vouchers also include various application documents or various documents submitted to public institutions, etc., such as tax returns, financial statements, business start-up notifications, corporate registration applications, etc.

[0013] As mentioned above, when receiving evidence when out or away from one's desk, one takes a picture of the evidence with the camera function of one's smartphone or the like and reads it through OCR processing. In addition, it is also possible to read the evidence through OCR processing by simply holding the smartphone over the device without taking a picture with the camera function. However, there are cases where it is not possible to immediately decide how to process the evidence image of the evidence that has been read. Even if the evidence image is saved on a smartphone or the like, it may be forgotten later, or it may become unclear how the evidence was obtained. For this reason, there has been a demand for a function that provides guidance on the subsequent processing and necessary information, etc., for the evidence image of the evidence that has been read.

[0014] Therefore, the evidence reading system according to the present disclosure stores model information for determining the type of evidence image and items corresponding to the type of evidence image in response to the input of the evidence image, and determines the type and items of the evidence image based on the model information in response to the input of the evidence image. Based on the user information and the type and items of the evidence image, additional information to be provided to the user is determined and presented to the user.

[0015] In addition, the evidence reading system according to the present disclosure provides information on the evidence related to the evidence image or the items of the evidence image as additional information. The additional information provided to the user is, specifically, information on what the evidence is, what subsequent processing (delivery, billing, payment, etc.) is required, where it should be stored, etc.

[0016] In addition, the evidence reading system according to the present disclosure provides input information for inputting various user information into blank items of the evidence related to the evidence image as additional information. Specifically, the input information for inputting various user information is various user information (user's own attribute information, current location information, schedule information, etc.) to be input into items that the evidence is various application forms and the user needs to input.

[0017] With the above-mentioned configuration, it is possible to receive guidance on how to process the scanned image of evidence when the evidence is received while away from home or away from one's desk. This makes it possible to know how to process the evidence after receiving it, where to store it, etc. In addition, it is possible to reduce the effort of the user having to input items that need to be input by the user on various application forms when away from home, etc.

[0018] <Embodiment 1> The following describes the voucher reading system 1. In the following description, for example, when the terminal device 10 accesses the server 20, the server 20 responds with information for generating a screen on the terminal device 10. The terminal device 10 generates and displays a screen based on the information received from the server 20.

[0019] <1 Overall configuration of document reading system 1> FIG. 1 is a block diagram showing the overall configuration of the evidence reading system 1 of the first embodiment. As shown in FIG. 1, the evidence reading system 1 includes a plurality of terminal devices (terminal device 10A and terminal device 10B are shown in FIG. 1. Hereinafter, they may be collectively referred to as "terminal device 10") and a server 20. The terminal device 10 and the server 20 are connected to each other so as to be able to communicate with each other via a network 80. The network 80 is configured as a wired or wireless network. In this embodiment, the server 20 is a Web server (including a cloud server), and exchanges information with the terminal device 10 through Web pages. In addition, a Web page browser for viewing Web pages is installed in the terminal device 10, but a dedicated application for providing the service of the server 20 may be installed and configured to be viewable through the dedicated application.

[0020] The terminal device 10 is a device operated by each user. Here, the user is a person who uses the terminal device 10 to read an image of a voucher (which may be printed on paper or may be electronic data), which is a function of the voucher reading system 1, and is, for example, an employee of a business enterprise or a sole proprietor. The terminal device 10 is realized by a stationary PC (Personal Computer), a laptop PC, or the like. In addition, the terminal device 10 may be, for example, a tablet compatible with a mobile communication system, a mobile terminal such as a smartphone, or the like.

[0021] The terminal device 10 is communicatively connected to the server 20 via a network 80. The terminal device 10 is connected to the network 80 by communicating with communication devices such as a wireless base station 81 compatible with communication standards such as 4G, 5G, and LTE (Long Term Evolution), and a wireless LAN router 82 compatible with wireless LAN (Local Area Network) standards such as IEEE (Institute of Electrical and Electronics Engineers) 802.11. As shown as the terminal device 10B in FIG. 1, the terminal device 10 includes a communication IF (Interface) 12, an input device 13, an output device 14, a memory 15, a storage unit 16, and a processor 19.

[0022] The communication IF 12 is an interface for inputting and outputting signals so that the terminal device 10 can communicate with an external device. The input device 13 is an input device (e.g., a keyboard, a touch panel, a touch pad, a pointing device such as a mouse, etc.) for receiving an input operation from a user. The output device 14 is an output device (a display, a speaker, etc.) for presenting information to a user. The memory 15 is for temporarily storing a program and data processed by the program, etc., and is a volatile memory such as a DRAM (Dynamic Random Access Memory). The storage unit 16 is a storage device for saving data, and is, for example, a flash memory or a HDD (Hard Disc Drive). The processor 19 is hardware for executing an instruction set described in a program, and is composed of an arithmetic unit, a register, a peripheral circuit, etc.

[0023] The server 20 is a device that provides a function of determining the type and items of a voucher from the voucher image, which is a function of the voucher reading system 1, determining additional information that provides various guidance to the user, and presenting it to the user. The server 20 stores model information that determines the type of the voucher image and the items according to the type of the voucher image according to the input of the voucher image. The server 20 determines the type and items of the voucher image based on the model information according to the input of the voucher image. The server 20 also determines the additional information to provide guidance to the user based on the user information and the type and items of the voucher image, and presents it to the user.

[0024] The server 20 is a computer connected to a network 80. The server 20 includes a communication IF 22, an input / output IF 23, a memory 25, a storage 26, and a processor 29.

[0025] The communication IF 22 is an interface for inputting and outputting signals so that the server 20 can communicate with an external device. The input / output IF 23 functions as an interface with an input device for receiving an input operation from a user and an output device for presenting information to a user. The memory 25 is for temporarily storing programs and data processed by the programs, etc., and is a volatile memory such as a DRAM (Dynamic Random Access Memory). The storage 26 is a storage device for saving data, such as a flash memory or a HDD (Hard Disc Drive). The processor 29 is hardware for executing an instruction set described in a program, and is composed of an arithmetic unit, a register, a peripheral circuit, etc.

[0026] <1.1 Configuration of the terminal device 10> FIG. 2 is a block diagram showing a functional configuration of the terminal device 10 constituting the document reading system 1. As shown in FIG. 2, the terminal device 10 includes a plurality of antennas (antenna 111, antenna 112), wireless communication units (first wireless communication unit 121, second wireless communication unit 122) corresponding to each antenna, an operation reception unit 130 (including a keyboard 131 and a mouse 132), a voice processing unit 140, a microphone 141, a speaker 142, a display 150, a storage unit 170, and a control unit 180. The terminal device 10 also has functions and configurations (e.g., a battery for storing power, a power supply circuit for controlling the supply of power from the battery to each circuit, etc.) that are not particularly shown in FIG. 2. As shown in FIG. 2, each block included in the terminal device 10 is electrically connected by a bus or the like. The terminal device 10 shown in FIG. 2 will be described as a mobile terminal such as a tablet or a smartphone compatible with a mobile communication system, but may be a stationary PC (Personal Computer) or a laptop PC.

[0027] The antenna 111 emits a signal generated by the terminal device 10 as a radio wave. The antenna 111 also receives a radio wave from space and provides the received signal to the first wireless communication unit 121.

[0028] The antenna 112 radiates a signal generated by the terminal device 10 as a radio wave. The antenna 112 also receives the radio wave from space and provides the received signal to the second radio communication unit 122.

[0029] The first wireless communication unit 121 performs modulation / demodulation processing and the like for transmitting and receiving signals via the antenna 111 so that the terminal device 10 can communicate with other wireless devices. The second wireless communication unit 122 performs modulation / demodulation processing and the like for transmitting and receiving signals via the antenna 112 so that the terminal device 10 can communicate with other wireless devices. The first wireless communication unit 121 and the second wireless communication unit 122 are communication modules including a tuner, a Received Signal Strength Indicator (RSSI) calculation circuit, a Cyclic Redundancy Check (CRC) calculation circuit, a high-frequency circuit, and the like. The first wireless communication unit 121 and the second wireless communication unit 122 perform modulation / demodulation and frequency conversion of wireless signals transmitted and received by the terminal device 10, and provide the received signals to the control unit 180.

[0030] The operation reception unit 130 has a mechanism for receiving an input operation from a user in response to the control of the control unit 180. Specifically, the operation reception unit 130 includes a keyboard 131 and a mouse 132. Note that the operation reception unit 130 may be configured as a touch screen that detects the position of the user's contact with the touch panel, for example, by using a capacitive touch panel.

[0031] The keyboard 131 accepts input operations by the user of the terminal device 10. The keyboard 131 is a device for inputting characters, and outputs input character information to the control unit 180 as an input signal.

[0032] The mouse 132 accepts input operations by the user of the terminal device 10. The mouse 132 is a pointing device for selecting an object displayed on the display 150, and outputs position information of an object selected on the screen and information indicating that a button has been pressed to the control unit 180 as input signals.

[0033] The audio processing unit 140 modulates and demodulates an audio signal under the control of the control unit 180. The audio processing unit 140 modulates a signal provided from the microphone 141 and provides the modulated signal to the control unit 180. The audio processing unit 140 also provides the audio signal to the speaker 142. The audio processing unit 140 is realized by, for example, a processor for audio processing. The microphone 141 accepts audio input and provides an audio signal corresponding to the audio input to the audio processing unit 140. The speaker 142 converts the audio signal provided from the audio processing unit 140 into audio and outputs the audio to the outside of the terminal device 10.

[0034] Display 150 displays data such as images, videos, and text under the control of control unit 180. Display 150 is realized by, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0035] Camera 160 is a device that receives light with a light receiving element and outputs the light as a captured image in accordance with the control of control unit 180. Camera 160 may be configured with an imaging device such as a digital camera or a video camera.

[0036] The storage unit 170 is configured with the memory 15 such as a flash memory and the storage unit 16, and stores data and programs used by the terminal device 10. In one aspect, the storage unit 170 stores user information 171 and evidence image information 172.

[0037] The user information 171 is information about a user who uses the terminal device 10 to read an evidence image, which is a function of the evidence reading system 1. The user information includes information for identifying the user (user ID), the name or title of the business operator or person in charge, and the like.

[0038] The evidence image information 172 is information on an evidence image that is the target of reading of the evidence image, which is a function of the evidence reading system 1, by the user using the terminal device 10. The evidence image information may be in any data format such as jpeg, png, bmp, tiff, gif, eps, svg, etc.

[0039] The control unit 180 is constituted by, for example, the processor 19, and controls the operation of the terminal device 10 by reading a program stored in the storage unit 170 and executing instructions included in the program. The control unit 180 fulfills the functions of an input operation receiving unit 181, a transmission / reception unit 182, a notification control unit 183, and a data processing unit 184 by operating in accordance with a program (not shown) that is an application that is stored in advance in the storage unit 170 of the terminal device 10 and installed, for example.

[0040] The input operation receiving unit 181 performs processing for receiving input operations by a user via an input device such as the keyboard 131 .

[0041] The transmission / reception unit 182 performs processing for the terminal device 10 to transmit and receive data to and from an external device such as the server 20 in accordance with a communication protocol.

[0042] The notification control unit 183 performs a process of presenting information to the user. The notification control unit 183 performs a process of causing the display 150 to display a display image, a process of causing the speaker 142 to output a sound, and the like.

[0043] The data processing unit 184 performs a process of performing calculations on data inputted by the terminal device 10 according to a program and outputting the calculation results to a memory or the like. The data processing unit 184 may be configured to perform some or all of the functions of the server 20 described later, which are the voucher image reception module 2033, the voucher image type determination module 2034, the voucher image item determination module 2035, the additional information determination module 2036, the user presentation module 2037, the user input reception module 2038, and the transmission module 2039, and may be executed by an application installed in the terminal device 10, for example. That is, these functions may be executed by the terminal device 10 or the server 20.

[0044] <1.2 Functional configuration of server 20> 3 is a block diagram showing a functional configuration of the server 20 constituting the voucher reading system 1. As shown in FIG. 3, the server 20 exerts the functions of a communication unit 201, a storage unit 202, and a control unit 203.

[0045] The communication unit 201 performs processing for the server 20 to communicate with external devices.

[0046] The storage unit 202 is composed of, for example, a memory 25 and a storage 26, and stores data and programs used by the server 20. The storage unit 202 stores a user database 2021, a voucher model database 2022, and the like.

[0047] The user database 2021 is a database for retaining various information such as information on users who use the evidence reading system 1, specifically, attribute information on users who use the evidence reading system 1 to read evidence images, which is a function of the evidence reading system 1, and types of evidence images previously read using the evidence reading system 1. Details will be described later.

[0048] The evidence model database 2022 is a database for holding model information for determining the type of evidence image read in the evidence reading system 1 and items according to the type of evidence image. As a specific example, the evidence model database 2022 may be a machine learning model that determines and outputs the type of evidence image and items according to the type of evidence image from image data obtained by photographing a document or the like. This machine learning model may include, for example, an OCR model, an object detection model, R-CNN, Fast R-CNN, Faster R-CNN, etc. as implementation examples. Details will be described later.

[0049] The control unit 203 performs functions indicated in various modules, such as a receiving control module 2031, a transmitting control module 2032, a voucher image reception module 2033, a voucher image type determination module 2034, a voucher image item determination module 2035, an additional information determination module 2036, a user presentation module 2037, a user input reception module 2038, and a transmitting module 2039, by the processor 29 of the server 20 performing processing according to a program.

[0050] The reception control module 2031 controls the process in which the server 20 receives a signal from an external device in accordance with a communication protocol.

[0051] The transmission control module 2032 controls the process in which the server 20 transmits signals to external devices in accordance with a communication protocol.

[0052] The evidence image receiving module 2033 controls a process of receiving an input of an evidence image from a user of the evidence reading system 1. The evidence image receiving module 2033 receives an input of a captured evidence image, as the camera 160 captures an image of the evidence by the user's operation of the terminal device 10. At this time, the user may capture an image of the evidence with the camera 160 of the terminal device 10 and receive an input of the evidence image, or may hold the evidence over the camera 160 under the control of a dedicated application of the terminal device 10 to receive an input of the evidence image.

[0053] The evidence images accepted by the evidence image acceptance module 2033 include various documents necessary for transactions as described above, such as photographed images of estimates, invoices, receipts, receipts, etc. The evidence images also include documents necessary for various applications in companies, etc., such as travel expense applications, vacation applications, expense applications, etc. Furthermore, the evidence images also include various application documents or various submission documents to public institutions, etc., such as tax returns, financial statements, business start-up notifications, and corporate registration applications. Information on the evidence images may also be accepted in any data format, such as PDF (Portable Document Format), Word, XML (eXtensible Markup Language), jpeg, png, bmp, tiff, gif, eps, svg, etc.

[0054] The voucher image type determination module 2034 controls a process of determining the type of the voucher image based on the model information stored in the voucher model database 2022 in response to the input of the voucher image accepted by the voucher image acceptance module 2033. The voucher image type determination module 2034, for example, reads the accepted voucher image by OCR processing and determines the type of the voucher image. The types of voucher images are specifically the above-mentioned estimate, invoice, receipt, acceptance note, travel expense application form, vacation application form, expense application form, final tax return, financial statement, business start-up notification, corporate registration application form, etc.

[0055] The evidence image type determination module 2034 reads the contents of the character string described in the evidence image and interprets the meaning of the character string, for example, by an OCR model or the like which is model information stored in the evidence model database 2022. In addition, the evidence image type determination module 2034 acquires position information of the character string in the evidence image, for example, by an object detection model or the like which is model information stored in the evidence model database 2022, and determines the position indicated by the character string in the evidence image, for example, whether it is a title, an item name, a main text, etc. The evidence image type determination module 2034 determines the type of the evidence image by these processes.

[0056] In addition, the evidence image type determination module 2034 may determine the type of the evidence image based on the type of the evidence image that is stored in the user database 2021 and that the user has previously read using the evidence reading system 1.

[0057] The evidence image type determination module 2034 stores the determination result in the user database 2021.

[0058] The voucher image item determination module 2035 controls a process of determining the items of the voucher image based on the model information stored in the voucher model database 2022 according to the type of the voucher image determined by the voucher image type determination module 2034. The voucher image item determination module 2035 determines the contents of each item described in the voucher, for example, for the voucher image read by the voucher image type determination module 2034 through OCR processing. The items of the voucher image specifically include the title, issuer, name, etc. in the above voucher image, and also include items, amounts, etc. in estimates, invoices, receipts, receipts, travel expense applications, etc.

[0059] The evidence image item determination module 2035 interprets the meaning of a character string from the content of the character string described in the evidence image read by the evidence image type determination module 2034 through OCR processing, for example. The evidence image item determination module 2035 also acquires position information of the character string in the evidence image by, for example, an object detection model, which is model information stored in the evidence model database 2022, and determines the position indicated by the character string in the evidence image, for example, whether it is a title, an item name, or the main text. The evidence image type determination module 2034 determines the items of the evidence image through these processes.

[0060] In addition, the evidence image item determination module 2035 may determine the items of the evidence image based on the types and items of evidence images stored in the user database 2021 and previously read by the user using the evidence reading system 1.

[0061] The voucher image item determination module 2035 stores and memorizes the determination result in the user database 2021.

[0062] The additional information determination module 2036 controls a process of determining additional information to be provided to a user based on user information stored in the user database 2021, the type of the evidence image determined by the evidence image type determination module 2034, and the items of the evidence image determined by the evidence image item determination module 2035. The additional information determination module 2036 may determine additional information to be provided to a user based on model information stored in the evidence model database 2022, for example. The additional information determination module 2036 may also determine additional information based on information on the type and items of the evidence image that the user has previously read using the evidence reading system 1, which is stored in the user database 2021.

[0063] The additional information determined by the additional information determination module 2036 includes, for example, information to inform the user about the items of the voucher or the voucher image. Specifically, the information to inform the user includes information such as the position of the voucher in transactions and various procedures, the timing of issuance, the contents of each item, the confirmation contents that the user should make regarding the voucher, the follow-up processing that the user should make regarding the voucher, and the deadline for confirmation and follow-up processing.

[0064] At this time, the additional information determined by the additional information determination module 2036 may be, for example, link information to a Web page that provides information about the evidence. Also, the additional information determined by the additional information determination module 2036 may be, for example, storage destination information indicating a storage destination of the information on the evidence image.

[0065] The additional information determined by the additional information determination module 2036 includes, for example, input information for blank items in the accepted evidence image when some or all of the items are blank. The input information for blank items in the evidence image is the name of the user when the item is "name," and is the transportation used by the user (route name, departure point, arrival point, etc.) when the item is "transportation."

[0066] At this time, the additional information determined by the additional information determination module 2036 may be, for example, user information (such as name) stored in the user database 2021. In addition, the additional information determined by the additional information determination module 2036 may be, for example, information acquired from the terminal device 10, or location information (corresponding to user location information) acquired from the terminal device 10.

[0067] The additional information determination module 2036 stores the determined additional information in the user database 2021.

[0068] The user presentation module 2037 controls the process of transmitting the additional information determined by the additional information determination module 2036 to the terminal device 10 and presenting it to the user. The user presentation module 2037 transmits the additional information to the terminal device 10 via the communication unit 201.

[0069] As described above, when the function is being executed on the terminal device 10, the user presentation module 2037 may not send or receive additional information, but may instead display the additional information on the display 150 and present it to the user through processing by the notification control unit 183 of the terminal device 10.

[0070] The user input receiving module 2038 controls the process of receiving input information from the user for the additional information presented to the user by the user presentation module 2037 through the user's operation of the terminal device 10. The input information for the additional information includes, for example, information on the confirmation result that the user should make for the voucher, information on the subsequent processing that the user should make for the voucher, instruction information for selecting link information to a web page, and instruction information for transition to a storage destination for the information on the voucher image. The input information for the additional information includes, for example, information for correcting the input information for blank items in the voucher image, and instruction information for processing the voucher (for example, sending to a specified destination).

[0071] The transmission module 2039 controls a process of transmitting information on the evidence image based on the input information accepted by the user input acceptance module 2038. The transmission module 2039 transmits the information on the evidence image to, for example, a storage destination of the information on the evidence image or a predetermined transmission destination via the communication unit 201 (or the transmission / reception unit 182).

[0072] <2 Data Structure> Fig. 4 is a diagram showing an example of the data structure of the user database 2021 stored in the server 20. Fig. 5 is a diagram showing an example of the data structure of the evidence model database 2022 stored in the server 20.

[0073] As shown in FIG. 4, each record of the user database 2021 includes an item "user ID", an item "user name", an item "user past evidence information", and the like.

[0074] The item “user ID” is information for identifying each user who uses the evidence reading system 1 to read an evidence image.

[0075] The item “user name” is the name or title of the user who uses the evidence reading system 1 to read the evidence image.

[0076] The item "user past evidence information" is information on the reading of evidence images performed in the past by a user of the evidence reading system 1, and specifically includes an item "evidence ID", an item "evidence name", an item "item information", and "additional information", etc. The item "user past evidence information" stores information on the reading of evidence images performed in the past by the user, for example, in chronological order, linked to information on the user identified by the item "user ID".

[0077] The item “evidence ID” is information for identifying each type of evidence image read by the evidence reading system 1.

[0078] The item "evident name" is a name indicating the type of the evidence image read and determined by the evidence reading system 1. In the item "evident name", information such as "quotation", "invoice", "travel expenses application", etc. is stored according to the type of the evidence image determined by the evidence image type determination module 2034.

[0079] The item "item information" is a name indicating an item of the evidence image read and determined by the evidence reading system 1. In the item "item information", information such as "item type", "issuer", and "route information" is stored according to the item of the evidence image determined by the evidence image item determination module 2035.

[0080] The item "additional information" is additional information determined by the evidence reading system 1, and the names of the additional information are shown as examples in Fig. 4. The item "additional information" stores additional information determined by the additional information determination module 2036 according to the type or item of the evidence image, such as "expense settlement", "past amount", and "Tokyo - Shinagawa". As examples of additional information shown in Fig. 4, the additional information "expense settlement" indicates that the user is informed that the item on the receipt is subject to expense settlement, and the additional information "Tokyo - Shinagawa" indicates that the user is informed of the contents to be entered in the route information of the travel expense application form.

[0081] The voucher image type determination module 2034 of the server 20 adds a record to the item "user past voucher information" as it determines the type of the voucher image (if the user is using the voucher reading system 1 for the first time, it adds a record to the user database 2021). The voucher image item determination module 2035 updates the item "item information" corresponding to the relevant item "voucher ID" or adds a record to the item "user past voucher information" as it determines the item of the voucher image. The additional information determination module 2036 updates the item "additional information" corresponding to the relevant item "voucher ID" as it determines the additional information.

[0082] As shown in FIG. 5, each record of the evidence model database 2022 includes an item "Evidence ID", an item "Evidence name", an item "Item management information", and the like.

[0083] The item “Evidence ID” is information that identifies each type of evidence image read by the evidence reading system 1, and corresponds to the item “Evidence ID” in the user database 2021.

[0084] The item “evident name” is a name indicating the type of evidence image to be read and determined by the evidence reading system 1, and corresponds to the item “evident name” of the user database 2021.

[0085] The item "item management information" is additional information presented to the user for items of the evidence image in the evidence reading system 1, and specifically includes the item "item information", the item "confirmation content", the item "input information", and "additional information". The item "item management information" stores additional information to be presented to the user for each item, linked to the information of the evidence image identified by the item "evidence ID". Note that the item "item management information" may store additional information to be presented to the user for the entire evidence, linked to the information of the evidence image identified by the "evidence ID", without being linked to a specific item of the evidence.

[0086] The item “item information” is a name indicating the item of the evidence image that is read and determined by the evidence reading system 1, and corresponds to the item “item information” of the user database 2021.

[0087] The item "confirmation contents" is information that guides the user about the items of the evidence image among the additional information presented for the item "item information". The item "confirmation contents" stores information that prompts the user to confirm, for example, information to guide the user to confirm past transactions and amounts for "item" and "amount", as additional information determined by the additional information determination module 2036 according to the type or item of the evidence image.

[0088] The item "input information" is the input information for the blank item of the document image among the additional information presented for the item "item information" when the item of the document image is blank. The item "input information" stores input information for the blank item, for example, information to input a user name for "name", as additional information determined by the additional information determination module 2036 according to the type or item of the document image.

[0089] The item “additional information” is additional information determined in the evidence reading system 1 and corresponds to the item “additional information” in the user database 2021.

[0090] Although not shown in the figure, model information for determining the type of evidence image to be read and items according to the type of evidence image may be stored in the evidence model database 2022. For example, the evidence model database 2022 may store, as model information, a machine learning model for inferring and determining the feature amount of the evidence and each item of the evidence image and the type and item of the corresponding evidence image.

[0091] <3 operations> 6 and 7, the additional information presentation process and the evidence image transmission process performed by the evidence reading system 1 in the first embodiment will be described. Note that in the description of the operations, an example in which the terminal device 10 executes each function of the server 20 will be described.

[0092] FIG. 6 is a flowchart showing an example of a flow of the additional information presentation process performed by the voucher reading system 1.

[0093] In step S111, the user operates the camera 160 of the terminal device 10 to photograph the document of evidence or to obtain (photograph) an image of the document of evidence by holding the document of evidence over the camera 160, and the data processing unit 184 accepts the input of the image of the document of evidence through the function of the document of evidence image receiving module 2033.

[0094] In step S112, the data processing unit 184 of the terminal device 10, by using the function of the evidence image type determination module 2034, determines the type of the evidence image based on the model information (which may be model information stored in the evidence model database 2022 through communication with the server 20, or may be model information stored in the terminal device 10) in response to the input of the evidence image accepted in step S111.

[0095] In step S113, the data processing unit 184 of the terminal device 10, by the function of the evidence image item determination module 2035, determines the items of the evidence image based on the model information (which may be model information stored in the evidence model database 2022 through communication with the server 20, or may be model information stored in the terminal device 10) depending on the type of evidence image determined in step S112.

[0096] In step S114, the data processing unit 184 of the terminal device 10 determines additional information to be provided to the user based on the type of the evidence image, which is the determination result determined in step S112, and the items of the evidence image, which are the determination result determined in step S113, by the function of the additional information determination module 2036. The additional information to be provided to the user may include information to be provided to the user about the evidence or the items of the evidence image, and may include input information for blank items in the evidence image.

[0097] In step S115, the data processing unit 184 of the terminal device 10 uses the function of the user presentation module 2037 to display the additional information determined in step S114 on the display 150, and presents it to the user.

[0098] As described above, the evidence reading system 1 stores model information for determining the type of evidence image and the items corresponding to the type of evidence image according to the input of the evidence image, and determines the type and items of the evidence image based on the model information according to the input of the evidence image. Based on the user information and the type and items of the evidence image, additional information to be provided to the user is determined and presented to the user. This allows the user to obtain additional information about the evidence, making it possible to know how to process the evidence after receiving it, where to store it, etc. Also, it becomes possible to reduce the effort required for the user to input items that the user needs to input about various application forms when away from home, etc.

[0099] FIG. 7 is a flowchart showing an example of a flow of the evidence image transmission process performed by the evidence reading system 1.

[0100] In step S211, the input operation accepting unit 181 of the terminal device 10 accepts input of input information for the additional information presented in step S115 by a user operation. The data processing unit 184 of the terminal device 10 accepts input information from the user for the additional information presented to the user by the function of the user input accepting module 2038.

[0101] In step S212, the data processing unit 184 of the terminal device 10 transmits information on the evidence image based on the input information accepted in step S211 by the function of the transmission module 2039. Here, an example in which information on the evidence image is transmitted to the server 20 and stored in a predetermined storage destination will be described. In step S212, information on the evidence image is transmitted to the server 20 via the transmission / reception unit 182.

[0102] In step S222, the server 20 receives the information on the evidence image transmitted in step S212 via the communication unit 201, and stores the information in a predetermined storage location in the server 20.

[0103] As described above, the evidence reading system 1 accepts input information from the user for additional information and transmits information on the evidence image. This allows various information to be input for the evidence image of the read evidence, and then transmitted to a server or the like for subsequent processing, thereby reducing the effort required for input by the user.

[0104] <4 Screen example> Hereinafter, with reference to FIG. 8 and FIG. 9, a screen example of the additional information presentation process by the voucher reading system 1 will be described.

[0105] Fig. 8 is a diagram showing an example of a screen in which guide information is added to the evidence image displayed on the terminal device 10. The example of the screen in Fig. 8 shows an example of a screen in which the evidence image accepted by the function of the evidence image acceptance module 2033 and the additional information determined by the function of the additional information decision module 2036 are presented together with the evidence image by the function of the user presentation module 2037. This corresponds to step S115 in Fig. 6.

[0106] 8, a voucher image 1501 is displayed on the display 150 of the terminal device 10. In addition to the voucher image 1501, additional information 1502 and 1503 are displayed on the display 150 so as to be superimposed on the voucher image 1501.

[0107] 8, the voucher image 1501 is an invoice, and the function of the voucher image type determination module 2034 reads the item "voucher name" stored in the voucher model database 2022 and determines that it is an invoice. Also, the function of the voucher image item determination module 2035 reads the item "item management information" stored in the voucher model database 2022 and determines that the item is the item of the voucher image stored in that item. Then, the function of the additional information determination module 2036 presents the item "additional information" stored in the voucher model database 2022 to the user.

[0108] In the example shown in Fig. 8, link information to the storage destination is shown in additional information 1502 as storage destination information indicating where the invoice is stored. Also, information informing the user of what should be confirmed by the user regarding the total amount in the invoice is shown in additional information 1503. This allows the user to obtain additional information about the voucher, and therefore makes it possible for the user to understand what should be confirmed about the voucher, how to process the voucher after receiving it, where to store it, and so on.

[0109] Fig. 9 is a diagram showing an example of a screen in which input information is added to the evidence image displayed on the terminal device 10. The example of the screen in Fig. 9 shows an example of a screen in which the evidence image accepted by the function of the evidence image acceptance module 2033 and the additional information determined by the function of the additional information decision module 2036 are presented together with the evidence image by the function of the user presentation module 2037. This corresponds to step S115 in Fig. 6.

[0110] 9, a voucher image 1601 is displayed on the display 150 of the terminal device 10. In addition to the voucher image 1601, input information 1602, 1603, 1604, and 1605 for blank items in the voucher image 1601 are displayed on the display 150 so as to fill in the blanks in the voucher image 1601.

[0111] 9, the voucher image 1601 is a travel expense claim form, and the function of the voucher image type determination module 2034 reads the item "voucher name" stored in the voucher model database 2022 and determines that it is a travel expense claim form. Also, the function of the voucher image item determination module 2035 reads the item "item management information" stored in the voucher model database 2022 and determines that the item is the item of the voucher image stored in that item and is input information. Then, the function of the additional information determination module 2036 presents the item "additional information" stored in the voucher model database 2022 to the user as input information.

[0112] In the example shown in FIG. 9, the department name input field is shown in input information 1602 as information for inputting the user's department name. The name input field is shown in input information 1603 as information for inputting the user's name. The transportation input field is shown in input information 1604 as information for inputting the transportation used by the user (starting point, arrival point). Furthermore, the transportation expense amount input field is shown in input information 1605 as information for inputting the transportation expenses of the transportation used by the user. This allows the user to obtain input information as additional information about the voucher, making it possible to reduce the effort required for the user to input items that need to be input by the user on various application forms.

[0113] <Summary> As described above, according to the present embodiment, model information for determining the type of the evidence image and the items corresponding to the type of the evidence image is stored in accordance with the input of the evidence image, and the type and items of the evidence image are determined based on the model information in accordance with the input of the evidence image. Based on the user information and the type and items of the evidence image, additional information to be provided to the user is determined and presented to the user. This allows the user to obtain additional information about the evidence, making it possible to know how to process the evidence after receiving it, where to store it, and so on.

[0114] Furthermore, according to this embodiment, the additional information is provided on the evidence related to the evidence image or on the items of the evidence image. The additional information provided to the user is, specifically, information on what the evidence is, what subsequent processing (delivery, billing, payment, etc.) is required, where it should be stored, etc. This makes it possible to understand how to process the evidence after receiving it, where it should be stored, etc.

[0115] Furthermore, according to this embodiment, as additional information, input information for inputting various user information into blank items of the evidence related to the evidence image is provided. Specifically, the input information for inputting various user information is various user information (user's own attribute information, current location information, schedule information, etc.) to be input into items that the user needs to input when the evidence is various application forms. This makes it possible to reduce the effort required for the user to input items that the user needs to input into various application forms.

[0116] Although the disclosed embodiments have been described above, they can be implemented in various other forms and can be implemented with various omissions, substitutions, and modifications. These embodiments and modifications, as well as omissions, substitutions, and modifications, are included in the technical scope of the claims and their equivalents.

[0117] <Additional Notes> The matters described in the above embodiments will be supplemented below.

[0118] (Appendix 1) A program to be executed by a computer having a processor 29 and a memory 25, wherein the memory 25 stores user information (2021) and model information (2022) for determining the type of a voucher image and items corresponding to the type of the voucher image in response to an input of the voucher image, and the program causes the processor 29 to execute the following steps: accepting input of a voucher image from a user (S111); determining the type of the voucher image based on the model information in response to the input of the voucher image; determining the items of the voucher image based on the model information in response to the type of the voucher image (S113); determining additional information that provides the user with information about the voucher or items of the voucher image related to the voucher image based on the user information, the type of the voucher image, and the items of the voucher image (S114); and presenting the determined additional information to the user (S115).

[0119] (Appendix 2) The memory 25 stores information on evidence images previously received from the user as user information, and in the step of determining additional information, determines the additional information based on the information on evidence images previously received from the user, (Appendix 1) in the program.

[0120] (Appendix 3) A program described in (Appendix 1) or (Appendix 2), in which, in the step of determining additional information, link information to a web page providing information about the evidence related to the evidence image is determined as information providing information about the evidence, and in the step of presenting it to the user, the web page to which the link information links is presented to the user.

[0121] (Appendix 4) A program described in (Appendix 1) or (Appendix 2), in which, in the step of determining additional information, storage destination information indicating a storage destination of information of the evidential image is determined as the additional information, and in the step of presenting it to the user, the storage destination of the information of the evidential image related to the storage destination information is presented to the user.

[0122] (Supplementary Note 5) The program according to (Supplementary Note 1), further causing the program to execute a step (S211) of accepting input information from the user in response to the additional information presented to the user.

[0123] (Supplementary Note 6) The program according to (Supplementary Note 5), further comprising a step (S212) of transmitting information of the evidence image based on input information from a user.

[0124] (Appendix 7) An information processing device comprising a control unit 203 and a memory unit 202, wherein the memory unit 202 stores user information (2021) and model information (2022) for determining the type of a voucher image and items corresponding to the type of the voucher image according to an input of the voucher image, and the control unit 203 executes the steps of: accepting input of a voucher image from a user (S111); determining the type of the voucher image based on the model information according to the input of the voucher image (S112); determining the items of the voucher image based on the model information according to the type of the voucher image (S113); determining additional information for informing the user about the voucher or items of the voucher image related to the voucher image based on the user information, the type of the voucher image, and the items of the voucher image (S114); and presenting the determined additional information to the user (S115).

[0125] (Appendix 8) A method to be executed by a computer including a processor 29 and a memory 25, wherein the memory 25 stores user information (2021) and model information (2022) for determining the type of a voucher image and items corresponding to the type of the voucher image in response to an input of the voucher image, and the method includes the steps of: the processor 29 executes a step (S111) of accepting input of a voucher image from the user; a step (S112) of determining the type of the voucher image based on the model information in response to the input of the voucher image; a step (S113) of determining the items of the voucher image based on the model information in response to the type of the voucher image; a step (S114) of determining additional information that provides the user with information about the voucher or items of the voucher image related to the voucher image based on the user information, the type of the voucher image, and the items of the voucher image; and a step (S115) of presenting the determined additional information to the user. [Explanation of symbols]

[0126] 1: Voucher reading system 10: Terminal device 10A: Terminal device 10B: Terminal device 13: Input device 14: Output device 15: Memory 16: Storage section 19: Processor 20: Server 25: Memory 26: Storage 29: Processor 80: Network 81: Wireless base station 82: Wireless LAN router 111: Antenna 112: Antenna 121: First wireless communication unit 122: Second wireless communication unit 130: Operation reception section 131: Keyboard 132: Mouse 140: Audio processing unit 141: Mike 142: Speaker 150: Display 160: Camera 170: Storage section 171: User information 172: Evidence image information 180: Control section 181: Input operation reception section 182: Transmitter / receiver 183: Notification control section 184: Data processing section 201: Communications Department 202: Storage section 203: Control section 2021: User Database 2022: Evidence model database 2031: Receiving control module 2032: Transmission control module 2033: Voucher image reception module 2034: Evidence image type determination module 2035: Voucher image item judgment module 2036: Additional information determination module 2037: User presented module 2038: User input reception module 2039: Transmission module

Claims

1. A program to be executed by a computer having a processor and a memory, The memory includes information of a user, storing model information for determining a type of the evidence image and an item corresponding to the type of the evidence image in response to the input of the evidence image; The program causes the processor to: receiving an input of the evidence image from the user; determining a type of the evidence image based on the model information in response to the input of the evidence image; determining an item of the evidence image based on the model information according to the type of the evidence image; determining additional information for informing the user about the evidence related to the evidence image or the items of the evidence image based on the user information, the type of the evidence image, and the items of the evidence image; and presenting the determined additional information to the user.

2. The memory stores information of the evidence image previously received from the user as the information of the user; 2. The program according to claim 1, wherein in the step of determining the additional information, the additional information is determined based on information of the evidence image previously received from the user.

3. In the step of determining the additional information, link information to a Web page providing information about the evidence related to the evidence image is determined; 3. The program according to claim 1, wherein in the step of presenting to the user, the Web page that is the link destination of the link information is presented to the user.

4. In the step of determining the additional information, storage destination information indicating a storage destination of information of the evidence image is determined as the additional information; 3. The program according to claim 1, wherein in the step of presenting to the user, a storage location of information of the evidence image related to the storage location information is presented to the user.

5. The program further comprises:

2. The program according to claim 1, further comprising: a step of receiving input information from the user in response to the additional information presented to the user.

6. The program further comprises: The program according to claim 5 , further comprising a step of transmitting information about the evidence image based on input information from the user.

7. An information processing device including a control unit and a storage unit, The storage unit includes information of a user who is a user, storing model information for determining a type of the evidence image and an item corresponding to the type of the evidence image in response to the input of the evidence image; The control unit is receiving an input of the evidence image from the user; determining a type of the evidence image based on the model information in response to the input of the evidence image; determining an item of the evidence image based on the model information according to the type of the evidence image; determining additional information for informing the user about the evidence related to the evidence image or the items of the evidence image based on the user information, the type of the evidence image, and the items of the evidence image; presenting the determined additional information to the user.

8. 1. A method for performing a process on a computer having a processor and a memory, comprising: The memory includes information of a user, storing model information for determining a type of the evidence image and an item corresponding to the type of the evidence image in response to the input of the evidence image; The method further comprises the processor: receiving an input of the evidence image from the user; determining a type of the evidence image based on the model information in response to the input of the evidence image; determining an item of the evidence image based on the model information according to the type of the evidence image; determining additional information for informing the user about the evidence related to the evidence image or the items of the evidence image based on the user information, the type of the evidence image, and the items of the evidence image; presenting the determined additional information to the user.

Images