Image processing device, control method thereof, and program

Abstract

To provide a technology for making it possible to perform transfer processing by appropriately acquiring information necessary for transfer processing from an attached image to an electronic mail in an image processing device.SOLUTION: An image formation device registers a mail address of a transmission source indicating a transmission source of an image of a document (invoice) containing information necessary for the transfer process using a resource server providing a bank service. The image formation device displays a list of one or more electronic mails received from the registered transmission source mail addresses among electronic mails received from external devices, and accepts user operations for selecting electronic mails of a processing object from the list. The image formation device performs the transfer process on the basis of information obtained by performing character recognition process to images attached to the electronic mails of the processing object.SELECTED DRAWING: Figure 15

Description

[Technical field]

[0001] The present invention relates to an image processing apparatus, a control method thereof, and a program. [Background technology]

[0002] There is known an automated transaction system that reads character information written on an account transfer request form or the like by optical character recognition and performs a transfer or transfer process based on the obtained character information. Patent Document 1 discloses an automated transaction system that acquires character data from an image obtained by imaging a transfer destination form by character recognition, identifies the name of a financial institution, branch name, account, and account number from the character data, and performs a transaction based on the identified character data.

[0003] In addition, in services that use the Internet to perform transfer or money transfer processing, users generally access the service using a web browser. In recent years, there have also been mechanisms provided that allow users to use such services without using a web browser, by using Web APIs (applications) published by the service provider. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2013-109726 A Summary of the Invention [Problem to be solved by the invention]

[0005] When using Web API to process transfers as described above, it is expected that documents such as invoices containing the information required for transfer processing will be received as email attachments. In such cases, it is necessary to select the image to be used for the transfer processing from multiple images received by email, which can make it difficult to find the desired image.

[0006] SUMMARY OF THE PRESENT DISCLOSURE An object of the present invention is to provide a technique for making it easier to search for an image to be used in a transfer process from among a plurality of images received by email in an image processing device. [Means for solving the problem]

[0007] An image processing device according to one embodiment of the present invention is characterized in that it comprises: a registration means for registering a sender email address indicating the sender of an image of a document containing information necessary for transfer processing using a server device that provides bank services; a reception means for displaying on a display unit a list of one or more emails received from an external device that have been received from a sender email address registered by the registration means, and receiving a user operation for selecting an email to be processed from the list; and an execution means for executing transfer processing using the server device based on information obtained by performing character recognition processing on an image attached to the email to be processed. Effect of the Invention

[0008] According to the present invention, it becomes possible for an image processing device to easily search for an image to be used in a transfer process from among a plurality of images received by email. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 shows an example of the configuration of a communication system. [Diagram 2] FIG. 1 is a block diagram showing an example of the configuration of an image forming apparatus. [Diagram 3] FIG. 2 is a diagram showing a configuration of an operation unit of the image forming apparatus and an example of an operation screen displayed on the operation unit; [Figure 4] A diagram showing an example of a settings screen for authenticating the use of banking services [Diagram 5] A diagram showing an example of a web browser screen for authenticating to use bank services [Figure 6] A diagram showing an example of the authentication sequence for using banking services [Figure 7] Flowchart showing the steps of the initial transfer process [Figure 8] FIG. 13 is a diagram showing an example of a screen displayed during the initial transfer process. [Figure 9] FIG. 13 is a diagram showing an example of a screen displayed during the initial transfer process. [Figure 10] Diagram showing an example of an invoice image attached to an email [Figure 11] FIG. 13 is a diagram showing an example of a screen displayed during the initial transfer process. [Figure 12] A diagram showing an example of the sequence of the transfer process (S717) [Figure 13] FIG. 13 is a diagram showing an example of a screen displayed during the initial transfer process. [Figure 14] A diagram showing an example of a home screen with shortcut buttons added [Figure 15] Flowchart showing the procedure for the second and subsequent transfers [Figure 16] A diagram showing an example of the screen and invoice image displayed during the second or subsequent transfer process. [Figure 17] Flowchart showing the procedure for checking inconsistencies in transfer destination information (S1512) [Figure 18] FIG. 13 is a diagram showing an example of a confirmation screen for the process of confirming inconsistencies. [Figure 19] FIG. 13 shows an example of a screen displayed during a second or subsequent transfer process (second embodiment); [Figure 20] Figure showing an example of a screen for editing filter conditions DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] [First embodiment] <Communication Systems> FIG. 1 shows an example of the configuration of a communication system according to the first embodiment. The communication system includes an image forming apparatus 101 and a PC 102 connected via a network 100 configured by a wired LAN or a wireless LAN. The network 100 is connected to the Internet 103. The communication system further includes an authorization server 104, a resource server 105, a storage server 106, and a mail server 107 connected via the Internet. The image forming apparatus 101, the PC 102, and these server devices on the Internet 103 are configured to be able to communicate with each other via the network 100 and the Internet 103. Note that image data transmitted and received between devices included in the communication system may be any of electronic data obtained by converting an image into RAW data, electronic data corresponding to an image format such as TIFF or JPEG, and electronic data corresponding to a PDF format.

[0012] Image forming apparatus 101 is an example of an image processing apparatus, and has various functions such as a printing function, a copying function, a scanning function, and a facsimile (FAX) function, as well as an e-mail sending and receiving function. Image forming apparatus 101 can receive e-mails sent from an external device (not shown) from mail server 107. Image forming apparatus 101 can also form (print) an image attached to the received e-mail on a sheet, and transfer the image to an external device such as PC 102, storage server 106, or another device.

[0013] The PC 102 is used to check the status and configure the image forming apparatus 101. The PC 102 can refer to or update the configuration values ​​of the image forming apparatus 101 using the HTTP protocol. The authorization server 104 is a server device that has a function of authorizing the use of a bank service API provided as a Web API. The resource server 105 is a server device that provides bank services such as bank transfers and the issuance of deposit and withdrawal statements. The storage server 106 is a server device that functions as online storage and is used to store image data and configuration information. The mail server 107 is a server device used to send and receive e-mails.

[0014] <Image forming device> 2 is a block diagram showing an example of the configuration of the image forming apparatus 101. The image forming apparatus 101 includes a control unit 110, and an operation unit 116, a reading unit 118, and a printing unit 120, which are connected to the control unit. The control unit 110 includes a CPU 111, and controls the operation of the entire image forming apparatus 101. The control unit 110 includes the CPU 111, a ROM 112, a RAM 113, a storage 114, an operation unit I / F (interface) 115, a reading unit I / F 117, a printing unit I / F 119, an external storage I / F 121, a communication I / F 123, and an image processing unit 124.

[0015] The CPU 111 performs various controls such as reading control, printing control, and communication control by reading and executing a control program stored in the ROM 112 or the storage 114. The ROM 112 stores a control program executable by the CPU 111. The ROM 112 also stores a boot program, font data, and the like. The RAM 113 is the main memory of the CPU 111 and is used as a work area and as a temporary storage area for expanding the control programs stored in the ROM 112 and the storage 114. The storage 114 stores image data, print data, an address book, shortcuts, various programs, and various setting information. The storage 114 in this embodiment is configured with a flash memory, but may be configured with an auxiliary storage device such as an SSD, HDD, or eMMC.

[0016] In the image forming apparatus 101 of this embodiment, the CPU 111 executes one program stored in one or more memories (RAM 113) to execute the processes shown in the flowcharts described below. However, at least a part of the processes shown in the flowcharts may be executed using a hardware circuit such as an ASIC or an FPGA.

[0017] The reading unit I / F 117 is an interface that connects the reading unit 118 and the control unit 110. The reading unit 118 reads an image of a document and generates image data (such as binary data) corresponding to the image. The image data generated by the reading unit 118 can be used for transmission to an external device, storage in an external recording device, and printing on a sheet (recording paper). The printing unit I / F 119 is an interface that connects the printing unit 120 and the control unit 110. The CPU 111 transfers image data for printing (image data of an image to be printed) to the printing unit 120 via the printing unit I / F 119. As a result, the printing unit 120 prints an image on a sheet fed from a paper feed cassette (not shown) using toner supplied from a cartridge (not shown) based on the received image data.

[0018] The operation unit I / F 115 is an interface that connects the operation unit 116 and the control unit 110. The operation unit 116 includes a display unit (display) having a touch panel function, and is used to display information to a user and to accept input from the user. The external storage I / F 121 is an interface that connects the external storage device 122 and the control unit 110. The CPU 111 can store image data in the external storage device 122 via the external storage I / F 121. In this embodiment, it is assumed that the external storage I / F 121 is a USB interface and the external storage device 122 is a USB memory, but an external storage device other than a USB memory, such as an SD card, may be connectable to the external storage I / F 121.

[0019] The control unit 110 is connected to the network 100 by a communication I / F 123. The communication I / F 123 is capable of communicating with the PC 102 and the like via the network 100, and is also capable of communicating with each server device on the Internet 103. For example, the communication I / F 123 can transmit image data to the PC 102 and various setting information within the image forming apparatus 101, send and receive e-mails via the mail server 107, and transmit image data to the storage server 106.

[0020] The communication I / F 123 connects to the authorization server 104 and obtains an authorization code for the bank service API provided by the resource server 105. The communication I / F 123 also connects to the resource server 105 and presents the authorization code for the bank service API to execute the bank service via the bank service API. The communication I / F 123 is configured to enable registration or modification of various setting information for obtaining the authorization code for the bank service API from the PC 102 via an HTTP connection.

[0021] The image processing unit 124 performs image processing on image data (input image data) generated by the reading unit 118 and image data (output image data) used for printing by the printing unit 120. The image processing unit 124 may perform image processing such as resolution conversion, color conversion, compression / decompression, and halftoning. The image processing unit 124 may also perform character recognition processing (OCR processing) for recognizing characters from an image and outputting character data (character information). For example, the image processing unit 124 may output character data by performing character recognition processing on the image data generated by the reading unit 118 and image data attached to an e-mail received from the mail server 107 by the communication I / F 123. In this embodiment, the CPU 111 may use the image processing unit 124 to perform character recognition processing on an image attached to an e-mail to be processed in order to obtain information necessary for a transfer process using the resource server 105.

[0022] <Operation section and home screen> 3 shows an example of the configuration of the operation unit 116 and an example of an operation screen displayed on a display unit (display) of the operation unit 116. The operation unit 116 is provided with a display unit 300 having a touch panel function, and LED lamps 306 and 307. The LED lamps 306 and 307 are used to notify a user of the status of the image forming apparatus 101. The LED lamp 306 is turned on while the image forming apparatus 101 is executing a job, and the LED lamp 307 is turned on when some kind of error occurs in the image forming apparatus 101.

[0023] 3 shows a home screen displayed immediately after the start-up of the image forming apparatus 101 as an example of an operation screen displayed on the display unit 300. The home screen is an example of an operation screen for selecting a function possessed by the image forming apparatus 101, and is used to instruct the execution of each function of the image forming apparatus 101. The home screen displays buttons corresponding to each function possessed by the image forming apparatus 101, such as copy, fax, scan, menu, and address book. By pressing (touching) a button displayed on the home screen, the user can instruct the execution of a function corresponding to the button. For example, by pressing button 302 displayed on the home screen, the execution of a bank transfer (transfer processing using the resource server 105) and the registration of transfer destination information can be instructed.

[0024] <Authentication for using banking services> Figure 4 shows an example of a settings screen for authentication to use bank services. The user information set using the displayed settings screen is used as the user information of the sender when making a bank transfer (transfer process) using the bank service.

[0025] In the image forming apparatus 101 of this embodiment, when the user selects a “Menu” button 301 displayed on the home screen, a setting menu for each application is displayed on the display unit 300. The setting screen 400 shown in Fig. 4(A) is displayed on the operation unit 116 when the user instructs settings related to authentication for using a bank service by using the setting menu of an application for using the bank service.

[0026] The setting screen 400 shown in FIG. 4(A) has input fields 401, 402 for inputting a user ID and password as user information. The password input in input field 402 is displayed in obscured characters. The user ID and password input in input fields 401, 402 are collated in the authentication process with a user ID and password that have been registered in advance for using the bank service. The setting screen 400 further has an input field 403 for inputting a registered name for the bank service. The registered name input in input field 403 is used as a name for identifying the remittance source in the transfer process. This registered name is set in association with the remittance source account.

[0027] When a login button 404 on the setting screen 400 is pressed, the image forming apparatus 101 performs authentication processing using the authorization server 104 according to the sequence shown in Fig. 6. When authentication based on the user information input via the setting screen 400 is successful, the image forming apparatus 101 displays a screen 410 shown in Fig. 4(B) on the operation unit 116. On the screen 410, the user ID and password input in the input fields 401 and 402 of the setting screen 400 are displayed in obscured characters, and only the registered name 411 is displayed. When a logout button 412 on the screen 410 is pressed, the image forming apparatus 101 returns the display screen of the operation unit 116 to the setting screen 400 and clears the input fields 401 to 403 in the setting screen 400.

[0028] On the other hand, if authentication based on the user information input via the setting screen 400 fails, the image forming apparatus 101 displays a screen 420 shown in Fig. 4C on the operation unit 116. When a "Yes" button 421 on the screen 420 is pressed, the display screen of the operation unit 116 is returned to the setting screen 400, and the input fields 401 to 403 in the setting screen 400 are cleared.

[0029] Note that image forming apparatus 101 of the present embodiment may be configured so that a web browser installed in PC 102 is used to make an HTTP connection from PC 102 to image forming apparatus 101, allowing similar settings to be made from PC 102. In this case, web browser screens 500, 510, and 520 shown in Fig. 5 are displayed on a display unit (not shown) of PC 102. Web browser screens 500, 510, and 520 have the same contents as screens 400, 410, and 420 shown in Figs. 4(A) to (C), respectively, and realize the same functions.

[0030] 6 shows an example of a sequence of authentication for using a bank service. The user operates the operation unit 116 to instruct the setting of user information (login setting) for using (accessing) the bank service from the image forming apparatus 101 (S601). In response to this instruction, the CPU 111 displays an input screen for user information (setting screen 400 shown in FIG. 4(A)) on the operation unit 116 (S602). On the setting screen 400, the user inputs a user ID and a password as user information in input fields 401 and 402, respectively, and presses a login button 404 (S603). In response to the pressing of the login button 404, the CPU 111 transmits a token request including the input user ID and password to the authorization server 104 (S604).

[0031] When the authorization server 104 receives a token request from the image forming apparatus 101, it performs authentication for use of the bank service based on the user ID and password included in the token request. Specifically, the authorization server 104 checks whether the user ID and password included in the token request are legitimate by comparing them with pre-registered user IDs and passwords. When the authorization server 104 checks that the user ID and password included in the token request are legitimate, it transmits a token request response including an access token indicating the result of authentication for use of the bank service to the image forming apparatus 101 (S605).

[0032] In the image forming apparatus 101, the CPU 111 displays a process completion screen indicating the result of the use authentication on the operation unit 116 based on the token request response received from the authorization server 104 (S606). Specifically, the CPU 111 associates the access token included in the token request response with the user information and stores it in the storage 114 as the user information. After completing the storage of the access token, the CPU 111 displays a screen 410 shown in FIG. 4B on the operation unit 116 and ends the process according to the sequence of FIG. 6. On the other hand, if the token request response includes an access token indicating that the authentication has failed, the CPU 111 displays a screen 420 shown in FIG. 4C on the operation unit 116 and ends the process according to the sequence of FIG. 6. In that case, the CPU 111 may request the user to re-enter the password.

[0033] <First transfer process> Fig. 7 is a flow chart showing the procedure for the initial transfer process. The processing of each step in Fig. 7 can be realized in image forming apparatus 101 by CPU 111 reading and executing a control program stored in ROM 112 or storage 114. The processing according to the procedure in Fig. 7 is performed when a user instructs execution of transfer processing without using previously registered transfer destination information in the registration processing in S719 to S722 described below. The user can instruct execution of transfer processing (and registration of transfer destination information) without using previously registered transfer destination information, for example, by pressing button 302 on the home screen (Fig. 3).

[0034] First, when the user presses button 302 on the home screen (FIG. 3), CPU 111 determines in S701 whether or not authentication to use the bank service has been completed. If authentication to use the bank service has been completed, CPU 111 advances the process to S702. If authentication to use the bank service has not been completed, CPU 111 advances the process to S723, displays setting screen 400 (FIG. 4(A)) on operation unit 116, and performs authentication to use according to the procedure in FIG. 6. When the process of S723 is completed, the process returns to S701, and it is determined again whether authentication to use the bank service has been completed.

[0035] In S702, the CPU 111 displays a selection screen 800 as exemplified in FIG. 8(A) on the operation unit 116 and accepts a user operation to select an arbitrary transfer source from among the transfer sources registered in advance. The transfer sources selectable by the user in S702 are transfer sources registered in advance by the user in the procedure of FIG. 6. When a transfer source is selected in the selection screen 800 and the "Next" button 802 is pressed, in S703, the CPU 111 displays a screen 810 as exemplified in FIG. 8(B) on the operation unit 116 and accepts an instruction to start receiving mail from the user. When a cancel instruction is accepted by pressing the "Cancel" button 804 on the screen 810 ("YES" in S704), the series of processes according to the procedure of FIG. 7 is terminated. On the other hand, when an instruction to start receiving mail is accepted by pressing the "Start" button 811 on the screen 810 ("NO" in S704), the process proceeds to S705.

[0036] In S705, CPU 111 displays screen 820, as exemplified in Fig. 8(C), on operation unit 116, and starts receiving mail from mail server 107. In this embodiment, an email is received from mail server 107 to which an image (image data of an image of an invoice) used in the transfer process is attached as an image of a document describing information (transfer destination information and transfer amount) necessary for transfer processing using resource server 105. Note that the document describing the transfer destination information and transfer amount necessary for transfer processing may be a document other than an invoice.

[0037] In S706, the CPU 111 determines whether there is an incoming mail from the mail server 107, and if there is an incoming mail, the process proceeds to S707. In S707, the CPU 111 displays a screen 900 exemplified in FIG. 9(A) on the operation unit 116, and accepts a user operation to select an incoming mail to be processed from a list 901 of incoming mails. In this way, the CPU 111 displays a list of e-mails received from the mail server 107 (external device) on the operation unit 116, and accepts a user operation to select an e-mail to be processed from the list. Each incoming mail and image data attached to each incoming mail are temporarily stored in the RAM 113. On the other hand, if there is no incoming mail, the CPU 111 displays a screen 910 exemplified in FIG. 9(B) on the operation unit 116, and ends the series of processes according to the procedure in FIG. 7 in response to the user pressing a "Close" button 911.

[0038] When a user selects an email and presses the "Preview" button 902 on screen 900 displayed on operation unit 116, in S708 CPU 111 displays a preview screen 920, as exemplified in FIG. 9C, on operation unit 116. On preview screen 920, an invoice image corresponding to image data attached to the received email numbered "No. 0001" on screen 900 is displayed. FIG. 10 shows an example of an invoice image based on image data attached to the received email numbered "No. 0001", and the invoice image is displayed on preview screen 920.

[0039] When the user presses the "Confirm" button 921 on the preview screen 920, the CPU 111 determines the image to be processed ("YES" in S709) and proceeds to S710. On the other hand, when the user presses the "Cancel" button 922 on the preview screen 920 ("NO" in S709), the CPU 111 returns the process to S707 and accepts the selection of a received email from the user again on the screen 900.

[0040] In S710, the CPU 111 displays a screen 930, as shown in Fig. 9(D) on the operation unit 116, and executes a process of extracting character information (text objects) from the invoice image. This process is realized by OCR processing on the invoice image. The CPU 111 executes OCR processing on the image data of the invoice image that is stored in the RAM 113 and that was determined to be the processing target in S709. In the OCR processing, the vicinity of each extracted character is searched in the left-right direction, text consisting of characters within a predetermined distance is extracted as a text object, and position information indicating the position of the text object is assigned to the text object.

[0041] When the process of S710 is completed, CPU 111 displays the extracted text object on operation unit 116 in order to obtain information required for the transfer process (transfer destination information and transfer amount). In this embodiment, CPU 111 obtains the transfer destination information and the transfer amount by accepting a user operation on operation unit 116 to associate the extracted text object (character information) with each item of the transfer destination information and the transfer amount.

[0042] Screens 1100, 1110, 1120, 1130, and 1140 illustrated in Fig. 11(A) to (E) are examples of selection screens for selecting a text object, which are displayed in sequence on the operation unit 116. When the "Confirm" button is pressed on each of the screens 1100, 1110, 1120, 1130, and 1140, the display screen on the operation unit 116 transitions to the next screen, and when the "Back" button is pressed, the display screen on the operation unit 116 transitions to the previous screen. When the "Back" button is pressed on the screen 1100, a confirmation screen 1150 illustrated in Fig. 11(F) is displayed on the operation unit 116 to confirm with the user whether or not to interrupt the transfer process. When the "Yes" button 1151 is pressed on the confirmation screen 1150 to instruct the transfer process to be cancelled ("YES" in S711), the CPU 111 ends the series of processes according to the procedure in Fig. 7. When the “No” button 1152 is pressed on the confirmation screen 1150 , the CPU 111 returns the display screen of the operation unit 116 to the screen 1100 .

[0043] In S712, the CPU 111 accepts the selection of text objects corresponding to each item of the transfer destination information and the transfer amount from among the text objects extracted from the invoice image, from the user via the screen displayed on the operation unit 116. In the example of FIG. 11, the selection of text objects indicating the bank name, branch name, account type, account number, and transfer amount is accepted via the screens 1100, 1110, 1120, 1130, and 1140, respectively. In accordance with the selection of the text objects by the user, in S713, the CPU 111 acquires the text objects selected for each item of the transfer destination information and the transfer amount. Thereafter, in S714, the CPU 111 determines whether the selection of the text objects corresponding to each item of the transfer destination information and the transfer amount has been completed, and if it is determined that the selection has been completed, the process proceeds to S715, and if it is determined that the selection has not been completed, the process returns to S711. In the example of FIG. 11, when the "Confirm" button is pressed on the screen 1140, the CPU 111 determines that the selection of the text objects has been completed. In this way, CPU 111 obtains the transfer destination information and the transfer amount by accepting a user operation for selecting a text object (associating text information extracted from the invoice image with each item of the transfer destination information and the transfer amount).

[0044] In S715, CPU 111 displays on operation unit 116 a confirmation screen 1160 for the user to confirm information required for the transfer process (transfer destination information and transfer amount) as exemplified in FIG. 11(G). When the user presses "Confirm" button 1161 on confirmation screen 1160 to instruct execution of the transfer process ("YES" in S716), CPU 111 advances the process to S717. In S717, CPU 111 executes the transfer process using resource server 105, and advances the process to S718. The transfer process in S717 is executed according to the sequence shown in FIG. 12, which will be described later. On the other hand, when the user presses "Cancel" button 1162 on confirmation screen 1160 to instruct cancellation of the transfer process ("NO" in S716), the series of processes according to the procedure in FIG. 7 is terminated.

[0045] At S718, CPU 111 judges whether the transfer process (S717) has been normally completed, and if it has been determined that the transfer process has been normally completed ("YES" at S718), the process proceeds to S719. At S719, CPU 111 displays on operation unit 116 a registration confirmation screen 1300, as exemplified in FIG. 13(A), for prompting the user to confirm the registration of the transfer destination information, and proceeds to S720. On the other hand, if CPU 111 judges that the transfer process has been normally completed ("NO" at S718), the cause of the unsuccessful completion of the transfer process is displayed on operation unit 116, and the series of processes according to the procedure of FIG. 7 is terminated. For example, if the transfer process has not been normally completed due to a failure in the authentication process, CPU 111 displays on operation unit 116 a screen 1320 as exemplified in FIG. 13(C). Furthermore, if the transfer process does not end normally because the transfer destination does not exist, CPU 111 displays a screen 1330, as shown in FIG. 13(D), on operation unit 116.

[0046] When the user presses the "Yes" button 1301 on the registration confirmation screen 1300 to instruct the registration of the transfer destination information ("YES" in S720), the CPU 111 advances the process to S721. On the other hand, when the user presses the "No" button 1302 on the registration confirmation screen 1300 to instruct not to register the transfer destination information ("NO" in S720), the CPU 111 ends the series of processes according to the procedure in FIG.

[0047] In S721, the CPU 111 displays an input screen 1310, as shown in FIG. 13B, on the operation unit 116 and accepts input of a registered name for identifying the transfer destination. The input screen 1310 has an input field 1311 for inputting the registered name. When the "Cancel" button 1312 is pressed on the input screen 1310, the CPU 111 ends the series of processes according to the procedure in FIG. 7. When the registered name is input to the input field 1311 on the input screen 1310 and the "Register" button 1313 is pressed, the CPU 111 registers the transfer destination information (saves it in the storage 114). At that time, the CPU 111 registers, together with the transfer destination information, positional information indicating the positions where each item of the transfer destination information and character information (text object) of the transfer amount were arranged in the invoice image. The transfer destination information includes information indicating at least one of the registered name for identifying the transfer destination, the bank name, the branch name, the account type, and the account number.

[0048] After the registration of the transfer destination information and the location information is completed, next, in S722, CPU 111 also registers the sender email address of the received email selected as the processing target in S707 as a filter condition in association with the transfer destination information and the location information. This filter condition is used as a condition for identifying emails to be displayed on operation unit 116 as candidates for processing among the emails received from the external device. In this way, after the transfer process based on the invoice image attached to the processing target email (S717) is completed, CPU 111 registers the transfer destination information used in the transfer process in association with the sender email address of the email. This makes it possible to reuse the transfer destination information used in the first transfer process for subsequent transfer processes.

[0049] Also, in S722, CPU 111 adds a shortcut button to the home screen, which the user uses to instruct the execution of the next (second and subsequent) transfer process using the registered transfer destination information. This shortcut button is associated with the sender email address registered as a filter condition. FIG. 14 shows an example of such a shortcut, in which shortcut button 1401 is added to the home screen shown in FIG. 3. In this way, CPU 111 adds shortcut button 1401 to the operation screen (home screen) for selecting the functions of image forming apparatus 101, which is displayed on operation unit 116. This shortcut button 1401 is an example of an instruction object for the user to instruct the execution of the transfer process based on the registered transfer destination information in association with the sender email address. In this example, the registration name included in the registered transfer destination information is displayed on shortcut button 1401 so that the user can identify the transfer destination. When creation of shortcut button 1401 is completed, CPU 111 ends the series of processes according to the procedure of FIG. 7.

[0050] In this embodiment, the sender's email address indicating the sender of the invoice image used in the transfer process is registered in association with the transfer destination information, but the present invention is not limited to this. For example, the transfer destination information and the email address (sender's email address) entered by the user using the operation unit 116 may be registered in association with each other.

[0051] <(Transfer Processing (S717)> Fig. 12 shows an example of the sequence of the transfer process (S717). For example, the user instructs execution of the transfer process by pressing the "Confirm" button 1161 on the confirmation screen 1160 shown in Fig. 11(G) (S1201). Upon receiving the instruction, the image forming apparatus 101 transmits a transfer instruction including an access token and transfer destination information to the resource server 105 (S1202).

[0052] When the resource server 105 receives a transfer instruction from the image forming apparatus 101, it extracts the access token from the transfer instruction and queries the authorization server 104 about the access token (S1203). In response to the query, the authorization server 104 returns information about the access token to the resource server 105 (S1204). The resource server 105 determines the validity of the access token based on the information about the access token received from the authorization server 104, and executes the transfer process if the access token is valid (S1205). Furthermore, the resource server 105 notifies the image forming apparatus 101 of the execution result of the transfer process (S1205). On the other hand, if the access token is invalid, the resource server 105 does not execute the transfer process, and notifies the image forming apparatus 101 of the completion of the process (S1206).

[0053] Based on the notification from the resource server 105, the image forming apparatus 101 notifies the user of the result of the transfer process by displaying the result on the operation unit 116 (S1207).

[0054] <Second and subsequent transfer processing> Fig. 15 is a flow chart showing the procedure for second and subsequent transfer processes performed using shortcut button 1401 added to the home screen. The processing of each step in Fig. 15 can be realized in image forming apparatus 101 by CPU 111 reading and executing a control program stored in ROM 112 or storage 114. The processing according to the procedure in Fig. 15 is performed when a user instructs execution of transfer processes based on registered transfer destination information. The user can instruct execution of transfer processes based on registered transfer destination information, for example, by pressing shortcut button 1401 on the home screen (Fig. 3).

[0055] First, when the user presses the shortcut button 1401 displayed on the home screen (FIG. 14), the CPU 111 of the image forming apparatus 101 displays a screen 1600 illustrated in FIG. 16A on the operation unit 116 in S1501, and receives an instruction to start receiving mail from the user. When a cancel instruction is received by pressing the "Cancel" button 1602 on the screen 1600 ("YES" in S1502), the series of processes according to the procedure in FIG. 15 ends. On the other hand, when an instruction to start receiving mail is received by pressing the "Start" button 1601 on the screen 1600 ("NO" in S1502), the CPU 111 displays a screen 820 illustrated in FIG. 8C on the operation unit 116 in S1503, and starts receiving mail from the mail server 107.

[0056] In S1504, CPU 111 determines whether there is an incoming mail from mail server 107, and if there is an incoming mail, proceeds to S1505. In S1504, CPU 111 determines whether there is an incoming mail sent from an address matching the sender mail address registered as a filter condition in association with shortcut button 1401 in the above-mentioned S722. If there is an incoming mail that satisfies the filter condition (i.e., the sender mail address matches the sender mail address registered as a filter condition), proceeds to S1505.

[0057] Note that CPU 111 may determine whether or not there is an email satisfying the filter condition among the emails received in advance and stored in storage 114, without receiving the email in S1503. In other words, CPU 111 may determine whether or not there is an incoming email sent from an address matching the sender email address registered as the filter condition among the emails stored in storage 114. This determination may be performed in response to pressing "Start" 1601 on screen 1600, or in response to pressing shortcut button 1401.

[0058] At S1505, CPU 111 displays screen 1610, as exemplified in Fig. 16(B), on operation unit 116, and accepts from the user the selection of received emails to be processed in list 1611 of received emails. List 1611 displays one or more emails received from registered sender email addresses (i.e., one or more emails that satisfy the filter conditions). In this way, CPU 111 narrows down the emails to be displayed on operation unit 116 as candidates to be processed, among the emails received from external devices, by using the registered sending email address. This makes it possible to easily find an image to be used in the transfer process from among multiple images received by email.

[0059] On the other hand, if there is no received mail, the CPU 111 displays a screen 910 exemplified in FIG. 9B on the operation unit 116, and ends the series of processes according to the procedure in FIG. 15 in response to the user pressing a "close" button 911.

[0060] When the user selects an email and presses the "Preview" button 1612 on screen 1610 displayed on operation unit 116 in S1505, CPU 111 displays preview screen 1620, as exemplified in FIG. 16(C), on operation unit 116 in S1606. On preview screen 1620, an invoice image corresponding to image data attached to the received email with number "No. 0015" on screen 1610 is displayed. FIG. 16(E) shows an example of an invoice image based on image data attached to the received email with number "No. 0015", and the invoice image is displayed on preview screen 1620.

[0061] When the user presses the “Confirm” button 1621 on the preview screen 1620, the CPU 111 determines the image to be processed (YES in S1507) and proceeds to S1508. On the other hand, when the user presses the “Cancel” button 1622 on the preview screen 1620 (NO in S1507), the CPU 111 returns the process to S1505 and accepts the selection of a received email from the user again on the screen 1610.

[0062] In S1508, CPU 111 executes a process of extracting character information (text object) from the invoice image by the same process as S710. When the process of S1508 is completed, in S1509, CPU 111 acquires the transfer destination information and the position information that have been saved (registered) in storage 114 in association with shortcut button 1401. The transfer destination information includes character strings (character information) of each item. The position information indicates the position where each item of the transfer destination information and the transfer amount (character string) were arranged in the invoice image. Furthermore, in S1509, CPU 111 acquires the character string (character information) at the position indicated by the position information acquired in S1509 based on the text object extracted by the process in S1508. In this way, when an e-mail sent from a registered sender e-mail address is to be processed, CPU 111 extracts character information from the position indicated by the position information registered in association with the sender e-mail address in the invoice image attached to the e-mail to be processed.

[0063] Thereafter, in S1510, CPU 111 performs a comparison process to compare character information acquired from the registered transfer destination information with character information acquired based on extraction of text objects from the invoice image. If the comparison process shows that there is no difference between the two (the two are consistent) (NO in S1511), CPU 111 advances the process to S1513. On the other hand, if there is a difference between the two (the two are inconsistent) (YES in S1511), CPU 111 advances the process to S1512 and performs a process to check for inconsistencies between the registered transfer destination information and the transfer destination information acquired from the invoice image. The process of S1512 is executed according to the procedure in FIG. 17, which will be described later. When the process of S1512 is completed, CPU 111 advances the process to S1513.

[0064] At S1513, CPU 111 displays on operation unit 116 a confirmation screen 1630 for the user to confirm the transfer destination information and the transfer amount, as shown in FIG. 16(D). When the user presses "Confirm" button 1631 on confirmation screen 1630 to instruct execution of transfer processing ("YES" at S1514), CPU 111 advances the process to S1515. At S1515, CPU 111 executes the transfer processing. The transfer processing at S1515 is executed according to the sequence shown in FIG. 12. In this way, CPU 111 executes the transfer processing using resource server 105, based on information obtained by performing character recognition processing (OCR processing) on ​​the invoice image attached to the e-mail to be processed. When the transfer processing is completed, CPU 111 ends the series of processes according to the procedure of FIG. 15. On the other hand, if the user presses the "Cancel" button 1632 on the confirmation screen 1630 to instruct cancellation of the transfer process ("NO" in S1514), the series of processes in the procedure of FIG. 15 ends.

[0065] <Checking for inconsistencies> FIG. 17 is a flow chart showing the procedure for the process (S1512) of checking for inconsistencies between registered transfer destination information and transfer destination information acquired from an invoice image.

[0066] In S1701, the CPU 111 extracts the part where the inconsistency occurs between the registered transfer destination information and the transfer destination information extracted from the invoice image. Next, in S1702, the CPU 111 displays the confirmation screen 1800 illustrated in FIG. 18 on the operation unit 116 to display the extracted inconsistency. The confirmation screen 1800 is an example of a screen displayed when an inconsistency occurs in the bank name among the items of the transfer destination information, and shows information on the part where the inconsistency occurs. Note that, when an inconsistency occurs in other items of the transfer destination information (branch name, account type, or account number), information on the part where the inconsistency occurs is similarly displayed. In this way, in S1701, the CPU 111 displays the confirmation screen 1800 on the operation unit 116 for the user to confirm information on the item where the inconsistency occurs in the transfer destination information. This allows the user to confirm whether there is a difference between the format of the invoice corresponding to the registered transfer destination information and the format of the invoice to be processed, making it possible to avoid transfer errors.

[0067] If the "Cancel" button 1802 is pressed on the confirmation screen 1800 ("YES" in S1703), the CPU 111 ends the process according to the procedure in Fig. 17, and also ends the transfer process according to the procedure in Fig. 15. On the other hand, if the user determines that there is no problem as a result of checking for inconsistencies in the transfer destination information, the user can press the "Confirm" button 1801 on the confirmation screen 1800 to execute the transfer process based on the transfer destination information acquired from the invoice image. In this case, when the CPU 111 determines that the "Confirm" button 1801 has been pressed and the user has completed checking for inconsistencies ("YES" in S1704), it ends the process according to the procedure in Fig. 17 and proceeds to S1513. Otherwise, the CPU 111 returns the process to S1702.

[0068] If the processing proceeds to S1513, after the confirmation screen 1800 is displayed, when the user instructs the CPU 111 to execute the transfer process (S1514), the CPU 111 executes the transfer process based on the transfer destination information and transfer amount obtained based on the extracted character information (S1515).

[0069] As described above, in this embodiment, the image forming apparatus 101 registers a sender email address indicating the sender of an image of a document (bill) containing information required for a transfer process using the resource server 105 that provides bank services. The image forming apparatus 101 displays on the operation unit 116 a list of one or more emails received from the registered sender email address among the emails received from the external device, and accepts a user operation to select an email to be processed from the list. Furthermore, the image forming apparatus 101 executes a transfer process using the resource server 105 based on information obtained by performing character recognition processing (OCR processing) on ​​an image attached to the email to be processed. In this way, among the emails received from the external device, the emails to be displayed on the operation unit 116 as candidates to be processed are narrowed down using the registered sender email address. This makes it possible for the image forming apparatus 101 to easily search for an image to be used for a transfer process from multiple images received by email.

[0070] According to one aspect of this embodiment, after completing a transfer process, the image forming device 101 may register the transfer destination information used in the transfer process and the position information indicating the position where each item of the transfer destination information and the text information of the transfer amount were placed in the attached image in association with the sender email address of the email for use in the next or subsequent transfer process. This makes it possible to reuse the registered transfer destination information in the next or subsequent transfer process, making it possible to avoid transfer errors caused by errors in the transfer destination information.

[0071] According to one aspect of this embodiment, during the above-mentioned registration, image forming apparatus 101 may add an instruction object (shortcut button 1401) for the user to instruct execution of a transfer process based on the transfer destination information registered in association with the sender's email address to the home screen displayed on operation unit 116. This allows the user to easily instruct reuse of the transfer destination information registered in association with the sender's email address by operating shortcut button 1401, making it possible to improve user convenience.

[0072] [Second embodiment] In the first embodiment, an example was described in which the sender email address of the received email used in the first transfer process is applied as a filter condition for receiving email for the second or subsequent transfer process using the shortcut button 1401 added to the home screen. In the second embodiment, an example is described in which the sender email address is registered as one of the filter conditions, and the filter condition is edited according to a user's operation. In this embodiment, it is possible to add a further sender email address and subject as such a filter condition, and further to edit and delete the filter condition. In the following, the second embodiment will be described focusing on the differences from the first embodiment, and the description of the parts common to the first embodiment will be omitted.

[0073] In this embodiment, when a second or subsequent transfer process is performed using shortcut button 1401 (FIG. 14) added to the home screen, image forming apparatus 101 (CPU 111) displays screen 1900, as shown in FIG. 19, on operation unit 116. When "START" button 1901 or "CANCEL" button 1902 on screen 1900 is pressed by the user, CPU 111 executes the same process as when corresponding button 1601 or 1602 included in screen 1600 (FIG. 16) is pressed. That is, when button 1901 is pressed to receive an instruction to start receiving mail, CPU 111 starts receiving mail from mail server 107 (S1503).

[0074] In this embodiment, when the "Edit Filter" button 1903 on the screen 1900 is pressed by the user, the CPU 111 displays a screen 2000, as exemplified in FIG. 20(A), on the operation unit 116. The user can use the screen 2000 to edit the filter conditions applied when receiving an e-mail. FIG. 20(A) shows, as an example, a state in which one sender e-mail address is set in the sender e-mail address filter and no subject keyword is set in the subject filter. When the "Edit" button 2001 or 2002 is pressed, the CPU 111 displays a soft keyboard (not shown) on the screen 1900 and accepts an operation from the user to add, edit, or delete the sender e-mail address and the subject keyword. FIG. 20(B) shows an example in which the filter conditions applied when receiving an e-mail are edited by the user on the screen 2000. In the screen 2020 illustrated in FIG. 20(B), one more sender email address is added to the sender email address filter as a filter condition, and the subject keyword “[Invoice Sending]” is added to the subject filter.

[0075] When the "Close" button 2003 (2013) is pressed on the screen 2000 (2010), the CPU 111 displays the screen 1900 again on the operation unit 116 and accepts the user's instruction (instruction to start receiving mail). When the "Start" button 1901 is pressed by the user with the filter conditions changed in this way, the CPU 111 judges whether or not there is an incoming mail that satisfies the changed filter conditions when receiving mail from the mail server 107. If there is an incoming mail that satisfies the filter conditions, the CPU 111 displays the screen 1610 on the operation unit 116 and accepts the selection of the incoming mail to be processed from the user in the list 1611 that includes such incoming mail (S1505). Furthermore, the CPU 111 executes the transfer process according to the user's instruction (S1515).

[0076] As described above, in this embodiment, the image forming apparatus 101 registers the sender email address as one of the filter conditions for identifying an email to be displayed as a candidate for processing among the emails received from an external device. Furthermore, the image forming apparatus 101 edits the filter conditions in accordance with a user's operation. The image forming apparatus 101 may be configured to enable the sender email address of an email to be added as a filter condition, or may be configured to enable the keyword included in the subject of an email to be added as a filter condition. This allows the user to arbitrarily edit the filter condition for identifying an email to be processed among the received emails, thereby improving user convenience.

[0077] [Other embodiments] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0078] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0079] 101: image forming device, 104: authorization server, 105: resource server, 107: mail server, 110: control unit, 111: CPU, 116: operation unit

Claims

1. An image processing device, A registration means for registering a sender's email address indicating the sender of an image of a document containing information necessary for a transfer process using a server device that provides bank services; a receiving means for displaying on a display unit a list of one or more e-mails received from a sender e-mail address registered by the registration means among the e-mails received from an external device, and receiving a user operation for selecting an e-mail to be processed from the list; an execution means for executing a transfer process using the server device based on information obtained by performing character recognition processing on an image attached to the email to be processed; An image processing device comprising:

2. The registration means registers the transfer destination information and the sender's email address in association with each other, When a user instructs execution of a transfer process based on the registered transfer destination information, the reception means displays a list of one or more e-mails received from the registered sender e-mail address in association with the transfer destination information, and receives the user operation.

2. The image processing device according to claim 1,

3. When a user instructs execution of a transfer process without using the registered transfer destination information, the receiving means displays a list of e-mails received from an external device, and receives a user operation for selecting an e-mail to be processed from the list; The registration means, after the transfer process based on the image attached to the email to be processed is completed, registers the transfer destination information used in the transfer process in association with the sender email address of the email.

3. The image processing device according to claim 2.

4. The registration means registers the transfer destination information used in the transfer process, together with position information indicating the positions where each item of the transfer destination information and the character information of the transfer amount were positioned within the image, in association with the sender email address; The execution means acquires character information of the transfer destination information and the transfer amount by the character recognition process from the position indicated by the position information registered in association with the sender email address within the image attached to the email to be processed, and executes the transfer process.

4. The image processing device according to claim 3.

5. The execution means includes: If the transfer destination information acquired by the character recognition process does not match the transfer destination information registered in association with the sender's email address, a confirmation screen is displayed on the display unit so that the user can confirm the information of the transfer destination information item where the inconsistency occurs. When the user instructs execution of the transfer process after the confirmation screen is displayed, the transfer process is executed based on the transfer destination information and the transfer amount acquired by the character recognition process.

5. The image processing device according to claim 4.

6. The registration means further adds an instruction object to an operation screen for selecting a function of the image processing device, which is displayed on the display unit, for allowing a user to instruct execution of a transfer process based on the registered transfer destination information associated with the sender email address; The receiving means, when a user performs an instruction using the instruction object displayed on the operation screen, displays a list of one or more e-mails received from the registered sender e-mail address and receives the user operation.

6. The image processing device according to claim 2, wherein the first and second inputs are input to the image processing apparatus.

7. 7. The image processing device according to claim 2, wherein the transfer destination information includes information indicating at least one of a registered name for identifying the transfer destination, a bank name, a branch name, an account type, and an account number.

8. The receiving means displays a preview screen on the display unit, the preview screen showing an image attached to the email selected from the list, When a user instructs on the preview screen to determine a processing target, the execution means executes the transfer processing based on information obtained by performing the character recognition processing on the image displayed on the preview screen.

8. The image processing device according to claim 1,

9. the registration means registers the sender email address as one of filter conditions for identifying emails to be displayed on the display unit as candidates for the processing target among emails received from an external device; The image processing device further includes an editing unit that edits the filter conditions in accordance with a user's operation.

9. The image processing device according to claim 1,

10. The editing means is capable of adding a sender email address of an email as the filtering condition.

10. The image processing device according to claim 9,

11. The editing means is capable of adding keywords contained in the subject of an e-mail as the filtering condition.

11. The image processing device according to claim 9, wherein the first and second inputs are input to the image processing device.

12. A control method for an image processing device, comprising: a registration step of registering a sender's email address indicating the sender of an image of a document containing information necessary for a transfer process using a server device that provides bank services; a receiving step of displaying on a display unit a list of one or more e-mails received from a sender e-mail address registered in the registration step among the e-mails received from the external device, and receiving a user operation of selecting an e-mail to be processed from the list; an execution step of executing the transfer process based on information obtained by performing character recognition processing on an image attached to the email to be processed; 23. A method for controlling an image processing apparatus comprising:

13. A program for causing a computer to execute each step of the method for controlling an image processing apparatus according to claim 12.

Images