Image processing system for digitizing documents, its control method and program

Abstract

To provide an image processing system and a control method thereof for enabling a user to easily grasp which text block in a preview pane is associated with a setting item in a property pane.SOLUTION: A user interface screen of an image processing system for digitizing a document has at least a preview pane 401, which is an area for displaying preview of a scanned image and a property pane 411, which is an area for entering set values for each property setting item. At least a part of a display field corresponding to the setting item displayed in the property pane is colored using a color assigned to the setting item, and when a user selects any text block of the scanned image previewed in the preview pane in a state in which any setting item is selected in the property pane, the text block is colored using a color assigned to the setting item in the selected state.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a graphical user interface (GUI) for use in digitizing documents. [Background technology]

[0002] Conventionally, optical character recognition (OCR) processing is performed on scanned images obtained by scanning documents such as order forms to extract character strings, which are then used for file names, etc. Patent Document 1 discloses a technology that extracts character strings containing a specific key from character strings contained in a document image, automatically generates a file name, and presents the file name to a user along with a preview display of the document image. This technology allows a user to easily set a file name while checking the document image. Furthermore, character strings extracted from scanned images are saved and managed as metadata, and are used for file names, etc. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-115011 A Summary of the Invention [Problem to be solved by the invention]

[0004] A common UI screen configuration for applications that use recognized text obtained through OCR processing to set file names and metadata is a two-pane configuration consisting of a preview pane and a properties pane. The preview pane displays a preview of the scanned image to be digitized, while the properties pane displays a list of properties such as file names and metadata. When digitizing documents such as purchase orders or invoices, the scanned image contains multiple text blocks, and the preview pane allows the user to visually identify each text block. The properties pane also allows the user to view various settings such as "Customer Name," "Date," and "Address." For example, in the above two-pane UI screen, a user selects the "XYZ Corporation" text block in the preview pane for the "Customer Name" setting in the properties pane. As a result, the recognized text "XYZ Corporation" in the scanned image is saved in association with the "Customer Name" setting. In this case, if there are multiple text blocks in the preview pane and multiple settings in the properties pane, it can be difficult for the user to immediately determine which text block is associated with which setting.

[0005] This invention was made in consideration of the above-mentioned problem, and aims to enable users to easily understand the correspondence between text blocks and setting items on a UI screen having a preview pane and a property pane. [Means for solving the problem]

[0006] In accordance with the present disclosure The information processing device is an information processing device comprising a control means for controlling display of a user interface screen including a document image and a plurality of fields, wherein the control means controls to input character recognition results obtained from each of a plurality of areas on the document image into a corresponding field among the plurality of fields and display the plurality of areas on the document image in a state in which the areas on the document image are distinguishable from areas other than the plurality of areas, and further, the control means controls, in response to a user selecting one area among the plurality of areas, to display the selected one area and one field corresponding to the selected one area in a color distinguishable from any other area different from the one area among the plurality of areas and any other field different from the one field among the plurality of fields, and to display a cut-out image obtained by cutting out a portion related to the selected one area from the document image. It is characterized by: [Effects of the Invention]

[0007] According to the technology of the present disclosure, a user can easily understand the correspondence between text blocks and setting items on a UI screen having a preview pane and a property pane. [Brief explanation of the drawings]

[0008] [Figure 1] A diagram showing the overall configuration of an image processing system [Figure 2] Block diagram showing the hardware configuration of an information processing device [Figure 3] Software configuration of the image processing system [Figure 4] A diagram explaining the state transitions of the UI screen [Figure 5] A diagram explaining the state transitions of the UI screen [Figure 6] 1 is a flowchart showing the flow of a UI screen control process according to the first embodiment. [Figure 7] A diagram explaining the problem of Modification 2 [Figure 8] FIG. 10 is a diagram illustrating the effect of Modification 2. [Figure 9] 10 is a flowchart showing details of a coloring control process according to Modification 2. [Figure 10] (a) is a diagram explaining the judgment of similarity of colors, and (b) is a diagram explaining the change of item colors. [Figure 11] 10 is a flowchart showing details of a coloring control process according to Modification 3. [Figure 12] FIG. 10 is a diagram illustrating the effect of Modification 3. [Figure 13] 10 is a flowchart showing the flow of a UI screen control process according to a second embodiment. [Figure 14] A diagram explaining the state transitions of the UI screen [Figure 15] A diagram explaining the state transitions of the UI screen DETAILED DESCRIPTION OF THE INVENTION

[0009] The following describes embodiments of the present invention with reference to the drawings. Note that the following embodiments do not limit the scope of the invention as claimed, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0010] [Embodiment 1] <System configuration> FIG. 1 is a diagram showing the overall configuration of an image processing system for digitizing documents according to this embodiment. The image processing system includes a scanned document processing server 111, a client terminal 121, and a business server 131. The client terminal 121 is communicatively connected to the scanned document processing server 111 and the business server 131 via a network 101 such as the Internet or an intranet. The client terminal 121 may be a personal computer, laptop computer, tablet computer, smartphone, or an MFP (Multifunction Peripheral) equipped with a scanner function. The configuration of the image processing system shown in FIG. 1 is merely an example and is not limiting. For example, the client terminal 121 may also have the functions of the scanned document processing server 111. In the following description, a scanned image obtained by scanning a document to be processed, such as an estimate, will be referred to as a "scanned document."

[0011] <Hardware configuration> FIG. 2 is a block diagram showing the hardware configuration of the scanned document processing server 111, client terminal 121, and business server 131, which are information processing devices. The user interface (UI) 201 is implemented by a display, keyboard, mouse, buttons, touch panel, etc., and performs input and output of information and signals. Computers without these hardware components can be connected and operated from other computers using remote desktop, remote shell, etc. The network interface 202 connects to the network 101 and communicates with other information processing devices and network devices (not shown). The communication method may be wired or wireless. The CPU 203 is a processing unit that controls the entire device and executes programs loaded from the ROM 204, RAM 205, secondary storage device 206, etc. The ROM 204 stores pre-installed programs and various data. The RAM 205 temporarily stores various data and also functions as a work memory for the CPU 203. The secondary storage device 206 is a large-capacity storage device such as a hard disk drive (HDD) or flash memory. The above components are interconnected via an input / output interface 207.

[0012] <Software configuration> 3 is a diagram showing the software configuration of the image processing system according to this embodiment. Each software is installed in the scanned document processing server 111, the client terminal 121, and the business server 131, and executed by the respective CPUs 203. Each software will be described below.

[0013] <Scan document processing server software> The scanned document processing application 311 is a Web application installed on the scanned document processing server 111. However, the application may be implemented in a form other than a Web application. The API 312 is an API (Application Programming Interface) provided by the scanned document processing application 311. The Web UI 313 is a Web user interface provided by the scanned document processing application 311.

[0014] The data store 321 is a storage area for saving and storing data used by the scanned document processing application 311 or the back-end application 331 (described later). The data store 321 includes a scanned document storage unit 322, a scanned document job queue 323, a property management unit 324, and a scanned document processing result storage unit 325. The scanned document storage unit 322 converts scanned document data into files and stores them as image files such as JPEG or document files such as PDF (Portable Document Format). The scanned document job queue 323 maintains a queue for managing jobs waiting for property input processing (described later). The property management unit 324 manages a list of setting items, the names and value formats (e.g., character strings, numbers, etc.) for each setting item, and item color information (described later) related to properties such as file names, folder paths, and metadata that need to be added to each scanned document. The scanned document processing result storage unit 325 stores OCR processing results and form discrimination results. The scanned document processing result storage unit 325 also stores character area information (text block information) associated with setting items, set values, and other information for each scanned document.

[0015] The back-end application 331 is an application for executing background processing. The back-end application 331 is an application responsible for various processes that can be executed sequentially in the background, specifically, OCR processing, form processing, and communication processing. The OCR processing unit 332 acquires the scanned document to be processed from the scanned document storage unit 322 and performs OCR processing on the acquired scanned document. The OCR processing extracts information identifying each text block (specifically, information on the starting coordinates, width, and height of a rectangular area) and information on the recognized character string for each text block. The form processing unit 333 uses the scanned document to be processed and information such as the text block layout pattern and recognized character string obtained by OCR processing to determine the type of the scanned document (form). This determination process may be performed using any method, such as pattern recognition or machine learning. The communication processing unit 334 transmits the scanned document and the analysis processing results to the business server 131, which is an external device. If there is no need to transmit the scanned document and the analysis processing results, the communication processing unit 334 can be omitted.

[0016] <Client terminal software> The client application 351 executes the web application of the above-mentioned scan document processing application 311. For example, the web application is executed by displaying a web UI 313 in a web browser and transmitting and receiving necessary data to and from the API 312.

[0017] <Business server software> The business application 361 is an application that supports various business operations in a company, such as file management, document management, order taking, and accounting. The business data storage 362 stores various data used by the business application 361. The business application 361 receives the results of processing by the scanned document processing server 111, performs the necessary processing on them, and stores them.

[0018] <Client application UI screen> Next, a description will be given of a user interface screen (hereinafter referred to as a "UI screen") displayed on the user interface 201 of the client terminal 121. Fig. 4(a) shows an example of a UI screen displayed on the user interface 201 by the client application 351 according to this embodiment.

[0019] The UI screen shown in FIG. 4(a) consists of two panes: a preview pane 401 and a property pane 411. A page image 402 of the scanned document "PURCHASE ORDER" is currently displayed in the preview pane 401. In the preview pane 401, the page image 402 can be scrolled and zoomed to display any position on the page image 402. The property pane 411 is a pane for displaying and editing properties to be assigned to the scanned document to be processed. The items to be set as properties include parent items such as "file name," "folder path," and "metadata," as well as child items such as "company name," "date," and "address," which are elements that make up the parent items. The following explanation will be given using an example in which "metadata" is set and registered as a parent item.

[0020] The UI screen in Figure 4(a) shows an example of a case where the document type of the scanned document being processed has been determined to be a "purchase order" in the document identification process. The screen also displays three child items, "Customer Name," "Address," and "Billing Name," as the contents of the parent item, "metadata," to be assigned to the scanned document. In the following explanation, the child items of metadata will be referred to as "metadata items."

[0021] FIG. 4(b) is a diagram illustrating a Cartesian coordinate system showing the relationship between the page image 402 and each text block. In FIG. 4(b), there are three text blocks 421, 422, and 423. Page image 402 contains many more text blocks, but these are omitted here. Text blocks detected by OCR processing are identified by their starting coordinates, width, and height, as shown by the shaded rectangles. For example, text block 423, which corresponds to the string "XYZ Corporation," is expressed in pixels as "starting coordinates (1200, 700)," "width: 720," and "height: 120."

[0022] FIG. 5 is a diagram for explaining a state in which a text block in the page image 402 is associated with a metadata item in the UI screen of FIG. 4(a). Now, a text block 501 corresponding to the string "XYZ Corporation" in the page image 402 and a metadata item "Customer Name" (the item of 510a) are in an associated state. And so that the user can easily identify, a rectangle colored with a color (in the example of FIG. 5, "red" assigned to the metadata item "Customer Name") assigned based on the item color information for each metadata item is transparently superimposed and displayed on the text block 501. Note that the coloring method is not limited to transparent superimposed display, and may be superimposed display of a colored frame line or the like. The horizontally long rectangles 510a to 510c in the property pane 411 are display fields for displaying and editing the set values etc. of metadata items which are child items for the metadata which is a major item. Also, the vertically long rectangles 511a to 511c are item color areas provided in a part of the display field for displaying the color (item color) assigned to each metadata item so that the user can identify it. And the horizontally long rectangles 512a to 512c are areas (cropped image areas) where an image (an image cropped from the page image 402) of the part corresponding to the text block selected by the user is displayed. At a stage where no text block is associated, a "+" button is displayed like the cropped image areas 512b and 512c, and further, the item colors in the item color areas 511b and 511c are not displayed. The horizontally long rectangles 513a to 513c below each cropped image area 512a to 512c are editing areas where the recognized character string included in the text block related to the user selection is displayed, and the user can edit the displayed content with a soft keyboard etc. not shown in the figure.

[0023] <UI Screen Control Process> Next, the control process of associating text blocks with each metadata item on the UI screen, performed by the client application 351 of the client terminal 121, will be described with reference to the flowchart of FIG. 6. Prior to the start of execution of the flow shown in FIG. 6, the scanned document processing application 311 of the scanned document processing server 111 first provides a queue of jobs waiting to be processed to the client application 351. When a user selects a scanned document job to be processed using the client application 351, the scanned document processing application 311 acquires necessary information corresponding to the selected scanned document from the data store 321. Specifically, the scanned document processing application 311 acquires the corresponding image file from the scanned document storage unit 322 and the corresponding OCR processing results and form discrimination processing results from the scanned document processing result storage unit 325. Furthermore, the scanned document processing application 311 acquires a list of metadata items for each form type and item color information identifying the predetermined colors assigned to each metadata item from the property management unit 324. The scanned document processing application 311 provides the acquired data to the client application 351, and the flow shown in FIG. 6 begins. In the following description, the symbol "S" indicates a step.

[0024] In S601, data of the page image and the OCR processing result (recognized character string) to be displayed in the preview pane 401 are loaded. In the following S602, a list of metadata items corresponding to the form type and item color information are loaded. Next, in S603, the contents of the data loaded in S601 and S602 are reflected on the UI screen. Specifically, the page image is displayed in the preview pane 401, and each metadata item included in the list of metadata items is displayed in the property pane 411. By performing the processing in S601 to S603 up to this point, the UI screen state shown in FIG. 4 is achieved. At this point, the coloring of each metadata item in the item color areas 511a to 511c in the property pane 411 is suspended and remains in the default state (e.g., gray). This is to visually indicate that the coloring is undecided, together with the display of the "+" button in the cropped image area 512 described above, thereby clearly indicating to the user that the metadata has not been set and prompting them to set the metadata.

[0025] In S604, an operation event of selecting the “+” button in one of the cropped image areas 512a-512c of one of the metadata items in the property pane 411 is detected. This selection operation is performed by the user touching the “+” button on the touch panel with a finger or a touch pen, or by clicking with a mouse. In this way, one metadata item selected by the user is determined from one or more metadata items included in the metadata item list. Hereinafter, the metadata item selected by the user is referred to as the “selected metadata item.” In S605, the item color area of ​​the selected metadata item determined in S604 is colored according to the item color information loaded in S602. Herein, it is assumed that the item color information associates a specific color with the order of each child item within a parent item. For example, four colors—blue, green, yellow, and red—are defined in advance as usable colors, and the item order and color are associated with each other, such as “blue” for the first setting item, “green” for the second setting item, “yellow” for the third setting item, and “red” for the fourth setting item. In this example, the three child items of the parent item "metadata" are "Customer Name," "Address," and "Billing Name," in that order, and thus "blue," "green," and "yellow" are assigned as item colors, respectively. By associating specific colors with the order of multiple setting items in this way, item color information can be standardized across various types of setting items. In the item color information, specific colors may be assigned in advance to individual specific setting items. Furthermore, the display field of the selected metadata item selected by the user may be highlighted, for example, by changing the color of the area other than the item color area within the display field, to make it easier for the user to identify the currently selected metadata item. The method for identifying the currently selected metadata item is not limited to changing the color; for example, the display field of the selected metadata item may be highlighted by surrounding it with a thick frame.

[0026] In S606, an operation event, such as a touch operation on one of the text blocks in the page image 402 in the preview pane 401, is detected. In this way, one arbitrary text block is selected from one or more text blocks included in the page image. Hereinafter, the text block selected by the user is referred to as the “selected text block.” In S607, the selected text block determined in S606 is colored in the same color as the color used in the item color area of ​​the selected metadata item colored in S605. In S608, a cutout image corresponding to the selected text block colored in S607 is displayed in the cutout image area of ​​the selected metadata item. Furthermore, the recognized character string included in the selected text block colored in S607 is displayed in the editing area of ​​the selected metadata item determined in S604. The processing in S604 to S608 results in the UI screen shown in FIG. 5. That is, the metadata item selected by the user is associated with a text block, and it is determined which text block to use to input the character string to be used for the selected metadata item.

[0027] When the above processing is completed for all metadata items in the metadata item list and the user touches the "Register" button 514, a metadata registration request is sent to the scanned document processing server 111 in S609. This metadata registration request includes information associating the edited values ​​of all metadata items with the text blocks. In the scanned document processing server 111, the scanned document processing application 311 stores the above information included in the metadata registration request received from the client terminal 121 in the scanned document processing result storage unit 325. At this time, if the metadata is to be registered by an external business server 131, the metadata registration request is transferred to the business server 131 via the communication processing unit 334. After completing all necessary processing, the scanned document processing application 311 dequeues the queue from the scanned document job queue 323. This completes the processing.

[0028] The above is the content of the UI screen control process in the client application 351 according to this embodiment. Note that in this embodiment, an item color area showing the color assigned to each setting item is provided in the display field of each setting item, but providing individual item color areas is not essential. That is, the area for highlighting indicating that a setting item has been selected by the user (the area excluding the cut image area and editing area in each display field) may also show the item color. Note that when the item color is shown in the area excluding the cut image area and editing area in each display field, highlighting indicating that the user has selected the setting item may be done, for example, by adding a thick frame around the selected display field or by changing the color density.

[0029] <Variation 1> In the flow of Fig. 6 described above, the item color area is colored at the timing of highlighting (S605) in response to the user's selection of a metadata item of interest (detection of a touch operation on the "+" button in the cropped image area). However, the timing of coloring the item color area is not limited to this. For example, the item color area in the selected metadata item may be colored at the same timing as the coloring of the selected text block in S607.

[0030] <Variation 2> In the flow shown in FIG. 6, the selected text block in the preview pane is colored the same color as the item color area of ​​the selected metadata item, allowing the user to understand the correspondence between the metadata item and the text block. However, there is a possibility that the color used for coloring and the character color or background color of the selected text block may be similar to each other. This may result in poor visibility of the color used for the selected text block. Therefore, a second variation will be described, in which the color used for coloring can be changed as needed.

[0031] FIG. 7, which corresponds to FIG. 5 and illustrates the problem with this modification, is a diagram illustrating the problem with this modification. In the page image 402′ in FIG. 7, three black character strings—“XYZ Corporation,” “1 Pumpkin Road, New York, NY,” and “(123)456-7980”—are surrounded by lines with a light blue background. A text block 701 corresponding to the character string “XYZ Corporation” in the page image 402′ is currently associated with the metadata item “Customer Name.” Since “Customer Name” is the first of the three metadata items, blue is displayed in the item color area 511a in its display field 510a. As a result, the light blue background color within the lines surrounding the character string “XYZ Corporation” and the blue used to color the text block 701 are similar to each other (similar colors). This makes it difficult for a user to understand that the text block 701 is associated with the metadata item “Customer Name.”

[0032] FIG. 8 is a diagram illustrating the effect of applying this modified example. As in FIG. 7, a text block 701 corresponding to the character string "XYZ Corporation" in page image 402' is associated with the metadata item "Customer Name." However, in FIG. 8, the item color area 511a for the metadata item "Customer Name" is displayed in red instead of the original blue. As a result, a color contrast is created between the light blue background color of the ruled line surrounding the character string "XYZ Corporation" and the red color of text block 701, making it easy for the user to understand the association.

[0033] Here, details of the process for controlling the coloring of setting items and text blocks according to this modified example will be described with reference to the flowchart of Fig. 9. The coloring control process shown in the flow of Fig. 9 is executed in place of S607 in the flow of Fig. 6 described above.

[0034] In S901, information on the character color of the character string included in the selected text block for which the operation event was detected in S606 and the background color of the text block is obtained from the scanned document processing result storage unit 325 via the scanned document processing application 311. Here, the explanation will continue assuming that the information obtained is a value indicating black as the character color (R, G, B = 0, 5, 5) and a value indicating light blue as the background color (R, G, B = 188, 213, 254).

[0035] In S902, information on the color assigned based on the item color information in S605 (i.e., the color assigned as the item color of the selected metadata item and assigned to the item color area) is acquired. In this modified example, the item color information is defined so that the default contents are "blue" to the first setting item, "green" to the second setting item, "yellow" to the third setting item, and "red" to the fourth setting item. Here, if the first of the three metadata items in the metadata pane 411, "Customer Name," is selected, the item color area will be colored "blue," and in this case, information indicating "blue" (e.g., RGB value) will be acquired.

[0036] In the next step S903, the color assigned as the item color of the selected metadata item is compared with the text color and background color in the selected text block to determine whether they are similar to each other. For example, as shown in FIG. 10A, the RGB value of the color assigned as the item color of the selected metadata item is compared with the RGB values ​​of the text color and background color acquired in S901, and a determination is made as to whether the difference is within a predetermined threshold. In this case, for example, for blue, it is desirable to set a threshold value such that colors with similar hues, such as light blue and purple, are determined to be "similar." Here, it is assumed that the blue assigned as the item color of the selected metadata item "Customer Name" is determined to be similar to the light blue background color of the selected text block. If the color assigned as the item color of the selected metadata item is determined to be similar to the text color or background color in the selected text block, the process proceeds to S904. On the other hand, if the color is determined to be dissimilar, the process proceeds to S905.

[0037] In S904, the color currently assigned as the item color of the selected metadata item is changed to a different color defined in the item color information. Specifically, the correspondence between the item order and each color in the candidate color information is changed, and a different color is assigned to the selected metadata item. In the example described above, the selected metadata item "Customer Name" is first in the order. Therefore, as shown in FIG. 10(b), the color is changed to the color associated with the last item in the order, which is the fourth item and is not used in the example described above ("red" in this case), and at the same time, "blue" is associated with the fourth item. By swapping the color correspondence between the first and fourth items in this way, the color correspondence between the second and third items can be maintained, minimizing the impact of the change. After changing the color of the selected metadata item and updating the item color information accordingly, the process returns to S903 and performs a similarity determination again. This process is repeated until a dissimilarity determination is obtained.

[0038] Then, in S905, the color display of the item color area of ​​the selected metadata item is updated as necessary. That is, if the color assigned as the item color of the selected metadata item in S904 is changed to another color, the item will be newly colored in the changed color.

[0039] In S906, the selected text block is colored using the same color as the color currently displayed in the item color area of ​​the selected metadata item. This results in the UI screen state shown in Figure 8. Once coloring is complete, the flow returns to Figure 6, and the display process of S608 is executed.

[0040] The above is the content of the coloring control process according to Modification 2. As described in the first embodiment, coloring of the item color area in the selected metadata item may be reserved until coloring of the selected text block, and both may be colored at the same time.

[0041] <Variation 3> When the number of setting items increases, it may be difficult to select and assign colors that are dissimilar to the text color or background color from the colors defined in the item color information. Therefore, we will explain a third variation in which a color is determined and assigned for each setting item without relying on the item color information.

[0042] Details of the process for controlling the coloring of setting items and text blocks according to this modified example will be described with reference to the flowchart in Fig. 11. Note that the coloring control process shown in the flow in Fig. 11 is executed in place of S602 in the flow in Fig. 6 described above. When this modified example is applied, in S605 of the flow in Fig. 6, the item color area is colored using a color determined by the process described below, rather than a color determined based on the item color information.

[0043] In S1101, a list of metadata items is loaded. In other words, in this modified example, item color information is not loaded. In the following S1102, information on the character colors of the character strings contained in all text blocks in the page image and the background colors of the text blocks is obtained from the OCR processing results loaded in S601.

[0044] In S1103, chromatic colors that appear frequently are identified from the character colors and background colors acquired in S1102. For example, suppose there are 100 text blocks in a page image, all of which use black as the character color. Meanwhile, suppose there are 80 text blocks that use white as the background color, 18 text blocks that use light blue, and 2 text blocks that use yellow-green. In this case, the chromatic colors are light blue and yellow-green, so light blue, which is used more frequently, is identified as the chromatic color that appears frequently. The reason why black and white, which are achromatic colors, are excluded is because they are generally used frequently in forms and the like, and are therefore unsuitable for use as item colors for identification purposes.

[0045] In S1104, an item color for each metadata item is determined based on the frequently occurring chromatic colors identified in S1103. An example of a specific determination method is as follows.

[0046] The first method is to use a color wheel. In this case, a frequently occurring chromatic color is used as a reference, and a color located farthest on the color wheel is determined as the item color for each metadata item. For example, if a frequently occurring chromatic color is light blue and this is used as a reference, the color located farthest on the color wheel is orange, so a color to be assigned as the item color is determined from a predetermined range (e.g., a range from yellow to purple) centered on orange. In this case, if there are multiple setting items to which item colors should be assigned and multiple colors need to be determined, the colors can be selected so that they are spaced approximately equal apart on the color wheel so that they can be distinguished from one another.

[0047] The second method is to determine, as the item color, a color dissimilar to a frequently occurring chromatic color from among a plurality of predetermined candidate colors. For example, five colors, red, green, yellow, blue, and purple, are determined in advance as candidate colors. Then, if the frequently occurring chromatic color is, for example, light blue, blue, which is similar to light blue, is excluded, and the remaining four colors (red, green, yellow, and purple) are sequentially assigned as the item color. Note that the number of candidate colors in this case may be determined depending on the format of the form expected to be processed, etc. Furthermore, when determining the colors to be excluded, a similarity determination may be performed similar to that described in S903 of Modification Example 2.

[0048] The two methods described above are merely examples, and it is sufficient if colors that are easy for the user to distinguish can be assigned as item colors based on frequently occurring chromatic colors.

[0049] 12 is a diagram illustrating the effect of applying this modified example. In this example, the background color within the ruled lines surrounding the character string "XYZ Corporation" is light blue, so three colors dissimilar to light blue are determined and assigned as item colors. That is, red, orange, and yellow are assigned as item colors to the three metadata items "Customer Name," "Address," and "Total Price," respectively. This makes the colors used to color the three text blocks 1201-1203 stand out, allowing the user to easily understand the association between each selected metadata item and its corresponding selected text block.

[0050] The above is the content of the coloring control process according to Modification 3. Note that multiple colors may be identified as frequently occurring chromatic colors. Furthermore, frequently occurring chromatic colors may be identified not from all text blocks present in the page image, but only from text blocks present in a certain area, such as the upper part of the page image. Note that, as with Modification 2, coloring of the item color area in the selected metadata item may be reserved until coloring of the selected text block, and both may be colored at the same time.

[0051] As described above, according to this embodiment, in the UI screen having a preview pane and a property pane, even if there are a plurality of text blocks in the page image and a plurality of setting items (sub-items) in the property pane, the user can easily grasp the correspondence between the two.

[0052] [Embodiment 2] In Embodiment 1, the UI screen control assuming a case where a one-to-one association of associating one text block with one metadata item is made has been described. Next, the UI screen control capable of dealing with a case where a one-to-N association of associating the same text block with a plurality of metadata items is made will be described as Embodiment 2. Regarding the contents common to Embodiment 1 such as the system configuration, the description will be omitted, and the differences will be mainly described below.

[0053] <UI Screen Control Process> The UI screen control process according to this embodiment performed by the client application 351 of the client terminal 121 will be described using the flowchart of FIG. 13. Here, according to the flow of FIG. 6 in Embodiment 1, the association with text blocks for each of the three metadata items "Customer Name", "Address", and "Billing Name" is made, and the description will be given assuming the state of the UI screen shown in FIG. 14.

[0054] In the UI screen of FIG. 14, a label 1401 indicating the number of metadata items (here, "2") with which the text block is associated is attached to the upper left of a text block 501 in the page image 402. This label is displayed when the same text block is associated with multiple metadata items (i.e., when the number of items becomes two or more), but this is not limited to this, and the label may be displayed when the text block is associated with any metadata item. Currently, the text block 501 is colored in the default color (here, gray) used when the text block is associated with multiple metadata items. In contrast, the text block 1402, which is associated with only one metadata item, "Address," is colored blue, as indicated by the corresponding item color area 511b.

[0055] With the UI screen in the state shown in FIG. 14, it is assumed that the user selects (by touching with a finger, for example) one of the display fields 510a to 510c of the metadata items in the property pane 411. In S1301, an operation event such as a touch operation by the user is detected. Here, it is assumed that an operation event for the display field 510c of "Billing Name" has been detected. In the following S1302, it is determined whether the text block associated with the metadata item selected by the user is also associated with other metadata items. If the determination result is "No," this processing ends. On the other hand, if the determination result is "Yes," the processing proceeds to S1303. Now, the text block 501 is associated with the metadata item "Billing Name" selected by the user, and further, the text block 501 is also associated with the metadata item "Customer Name." Therefore, the determination result in this case is "Yes," and the processing proceeds to S1303.

[0056] In S1303, the color of the text block associated with the selected metadata item is colored the same as the color of the item color area of ​​the selected metadata item. In this case, the color of the text block 501 changes from the default gray to the yellow indicated by the item color area 511c in the display field for the metadata item "Billing Name." The UI screen shown in FIG. 15 shows the state in which the color of the text block 501 has changed from gray to yellow in response to the selection of the metadata item "Billing Name."

[0057] In the next step S1304, a button for canceling the association between the selected metadata item and the text block is displayed. In the UI screen of Fig. 15, the "x" button 1501 that appears in the upper right corner of the text block 501 indicates the cancel button. In the following step S1305, it is determined whether or not the cancel button has been touched. If an operation event such as a touch operation on the cancel button is detected, the process proceeds to step S1306; if not, the process ends.

[0058] In S1306, the association between the selected metadata item and the text block is released, and the label indicating the number of currently associated metadata items is updated. Specifically, as a result of releasing the association with the selected metadata item, the number of metadata items indicated by the label is decremented by "1." In the example of FIG. 15, the value of label 1401 changes from the current "2" to "1." In addition, the display field corresponding to the selected metadata item is initialized to a blank state. Since the selected metadata item is now "Billing Name," the corresponding display field 510c is initialized.

[0059] The above is the content of the UI screen control process in the client application 351 according to this embodiment.

[0060] As described above, according to this embodiment, even when the same text block is associated with multiple setting items (child items), the user can accurately and easily grasp the correspondence between the child items and the text blocks.

[0061] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

Claims

1. An information processing apparatus comprising a control means for controlling display of a user interface screen including a document image and a plurality of fields, the control means controls the display of the character recognition results obtained from each of the plurality of regions on the document image by inputting the character recognition results obtained from each of the plurality of regions on the document image into corresponding fields among the plurality of fields, and controls the display of the plurality of regions on the document image in a manner that allows the plurality of regions to be distinguished from regions other than the plurality of regions, Furthermore, in response to a user selecting one area from the plurality of areas, the control means controls to display the selected area and one field corresponding to the selected area in a color that can be distinguished from other areas from the plurality of areas that are different from the one area and other fields from the plurality of fields that are different from the one field, and to display a cut-out image obtained by cutting out a portion related to the selected area from the document image.

1. An information processing device comprising:

2. The information processing device according to claim 1, characterized in that the control means controls the display of the selected area and the selected field by changing the color of the area and the selected field in a color that is distinguishable from both the other fields and the selected areas, in response to the selection of the area by the user.

3. The information processing device according to claim 1, characterized in that the control means controls the display of the one area and the one field in a color that is distinguishable from either the other areas or the other fields, in response to the user selecting the one area, so that the user can understand the correspondence between the one area and the one field.

4. 2. The information processing device according to claim 1, wherein, when the one area and the one field are displayed in a color that is distinguishable from the other areas and the other fields in response to the one area being selected by the user, the coloring of the one field is performed by coloring at least a portion of the one field.

5. The information processing device according to claim 1, characterized in that the control means controls the display of the plurality of regions on the document image in a manner that makes them distinguishable from regions other than the plurality of regions by transparently superimposing colored rectangles on each of the plurality of regions on the document image.

6. 6. The information processing apparatus according to claim 5, wherein the colored rectangles displayed in a transparent and superimposed manner on each of the plurality of regions are rectangles of different colors.

7. The information processing device described in claim 5, characterized in that the color of the colored rectangle displayed transparently and superimposed on the one area is a different color from the colored rectangle displayed transparently and superimposed on the other area, and is a color that allows the user to understand the correspondence between the one area and the one field.

8. 6. The information processing apparatus according to claim 5, wherein the color used to color the one field is the same as the color of the colored rectangle that is transparently superimposed and displayed on the one area.

9. 2. The information processing device according to claim 1, wherein the control means controls the display of the plurality of regions on the document image in a manner that makes them distinguishable from regions other than the plurality of regions by displaying a colored border for each of the plurality of regions on the document image.

10. 10. The information processing apparatus according to claim 9, wherein the colored borders displayed for the plurality of regions are borders of different colors.

11. 10. The information processing device according to claim 9, wherein the color of the colored border line displayed for the one area is different from the color of the colored border line displayed for the other areas, and is a color that allows the user to understand the correspondence between the one area and the one field.

12. 10. The information processing apparatus according to claim 9, wherein the color used to color the one field is the same as the color of the colored frame line displayed for the one area.

13. 2. The information processing apparatus according to claim 1, wherein the plurality of fields are fields for inputting values ​​corresponding to a plurality of setting items.

14. 2. The information processing device according to claim 1, wherein the color used by the control means to display the one area and the one field in a color that is distinguishable from the other areas and the other fields is a color specified based on predefined color information.

15. In the color information, colors corresponding to the order of the plurality of fields are defined in advance, the color used by the control means when displaying the one field and the one region in a color that is distinguishable from the other fields and the other regions is a color specified based on the predefined color information; 15. The information processing apparatus according to claim 14,

16. The information processing device according to claim 14, characterized in that the control means, if the color specified based on the predefined color information is similar to the character color or background color in the one area selected by the user's operation, changes the color to a dissimilar color, and uses the changed color to color and display the one area and the one field.

17. the plurality of regions are text blocks for which the character recognition results have been obtained and which have been input into any of the plurality of fields; 17. The information processing apparatus according to claim 1, wherein the area other than the plurality of areas includes a text block different from the plurality of areas.

18. 18. The information processing apparatus according to claim 1, wherein the user interface screen includes a pane for displaying the document image and a pane for displaying the plurality of fields.

19. 19. The information processing apparatus according to claim 1, wherein the document image is a document image obtained by scanning a document.

20. An information processing device described in any one of claims 1 to 19, characterized in that the cut-out image is not colored in the selected one area and the one field corresponding to that one area.

21. An information processing device described in any one of claims 1 to 20, characterized in that the character recognition result obtained from the one area entered in the one field is used as the file name of a file generated based on the document image.

22. A program for causing a computer to function as the control means of the information processing device according to any one of claims 1 to 21.

23. A control method in which a control means of an apparatus executes a program to control a display to display a user interface screen including a document image and a plurality of fields, the method comprising: the control means executes the program to input character recognition results obtained from each of a plurality of regions on the document image into corresponding fields among the plurality of fields and display the results, and to display the plurality of regions on the document image in a manner that allows them to be distinguished from regions other than the plurality of regions, Furthermore, by executing the program, the control means, in response to a user selecting one area from the plurality of areas, controls to display the selected area and one field corresponding to the selected area in a color that can be distinguished from other areas from the plurality of areas that are different from the one area and other fields from the plurality of fields that are different from the one field, and also controls to display a cut-out image obtained by cutting out a portion related to the selected area from the document image. A control method comprising:

24. 1. A system comprising at least one control means for controlling display of a user interface screen including a document image and a plurality of fields, the at least one control means controls to input character recognition results obtained from each of a plurality of regions on the document image into corresponding fields among the plurality of fields and display the results, and to display the plurality of regions on the document image in a manner that allows the regions to be distinguished from regions other than the plurality of regions, Furthermore, in response to a user selecting one area from the plurality of areas, the at least one control means controls to display the selected area and one field corresponding to the selected area in a color that is distinguishable from other areas from the plurality of areas that are different from the one area and other fields from the plurality of fields that are different from the one field, and to display a cut-out image obtained by cutting out a portion related to the selected area from the document image. A system characterized by:

25. The system described in claim 24, characterized in that the at least one control means controls to change the color of the one area and the one field in response to the one area being selected by the user, thereby displaying the selected one area and the one field in a color that is distinguishable from both the other fields and the other areas.

26. The system described in claim 24, characterized in that the at least one control means controls, in response to the user selecting the one area, to display the one area and the one field in a color that is distinguishable from both the other areas and the other fields, so that the user can understand the correspondence between the one area and the one field.

Images