Information processing apparatus, storage medium, and control method

Abstract

The invention provides an information processing apparatus, a storage medium, and a control method. In order to enable a user to easily grasp which text block within a preview pane is associated with a setting item within an attribute pane, at least a part of a display column corresponding to the setting item is colored by using a color assigned to the setting item displayed in the attribute pane. Then, when the user selects one of the text blocks in the scan image previewed and displayed in the preview pane in a state in which one of the setting items is selected in the attribute pane, the text block is colored with the color assigned to the setting item in the selected state.

Description

[0001] (This application is a divisional application of an application filed on January 21, 2021, with application number 202110081708.4, and entitled “Computerized Image Processing System for Documents, Control Method Thereof, and Storage Medium Therefor.”) Technical Field

[0002] The present invention relates to computerizing documents and graphical user interfaces (GUIs) when using computerized documents. Background Art

[0003] Conventionally, optical character recognition (OCR) processing has been performed on scanned images obtained by scanning business forms such as order forms to extract character strings, and the character strings have been used as file names, etc. Japanese Patent Application Laid-Open No. 2019-115011 discloses a technology that automatically generates file names by extracting character strings including specific keywords from character strings included in a document image, and presents the file names to the user along with a preview display of the document image. With this technology, the user can easily set the file name while checking the document image. The character strings extracted from the scanned image are then saved and managed as metadata, and the metadata is used as file names, etc.

[0004] A dual-pane configuration, consisting of a preview pane and a property pane, is a well-known configuration for an application UI screen used to set file names or metadata using a recognized character string obtained through OCR processing. The preview pane is an area for previewing and displaying a computerized scanned image of a target, while the property pane is an area for displaying property details, such as the file name and metadata, in a list. When computerizing documents such as purchase orders or invoices, the scanned image includes multiple character areas (text blocks), and the user can visually identify each text block through the preview pane. Furthermore, the property pane allows the user to check the contents of various setting items, such as "Customer Name," "Date," and "Address." For example, suppose a user selects the text block "XYZ Company" within the preview pane for the "Customer Name" setting item within the property pane on the dual-pane configuration UI screen described above. As a result, the recognized character string "XYZ Company" within the scanned image is stored in association with the "Customer Name" setting item. In this case, if multiple text blocks exist within the preview pane and multiple setting items also exist within the property pane, it becomes difficult for the user to immediately grasp which block is associated with which setting item.

[0005] The present invention has been made in view of the above-mentioned problems, and an object thereof is to enable a user to easily grasp the correspondence relationship between text blocks and setting items on a UI screen having a preview pane and a property pane. Summary of the Invention

[0006] According to the present invention, a computerized image processing system for documents includes: a display unit configured to display a user interface screen, the user interface screen being used to set properties for a scanned image obtained by scanning the document; and a control unit configured to control the display on the user interface screen, wherein on the user interface screen, there are at least a preview pane and a property pane, the preview pane being an area for previewing and displaying the scanned image, and the property pane being an area for inputting setting values ​​for each setting item of the property, and wherein at least a portion of a column of a setting item displayed in the property pane and a text block corresponding to the setting item in the scanned image previewed and displayed in the preview pane are colored using the same color or similar colors.

[0007] According to a control method of an image processing system of the present invention, the image processing system computerizes documents, and the image processing system includes a display and a controller, the display is used to display a user interface screen, the user interface screen is used to set properties for a scanned image obtained by scanning the document, and the controller is used to control the display on the user interface screen, wherein on the user interface screen, there are at least a preview pane and a property pane, the preview pane is an area for previewing and displaying the scanned image, and the property pane is an area for inputting setting values ​​of each setting item of the property, and wherein at least a portion of a column of a setting item displayed in the property pane and a text block corresponding to the setting item in the scanned image previewed and displayed in the preview pane are colored using the same color or similar colors.

[0008] According to a non-transitory computer-readable storage medium of the present invention, the program is used to cause a computer to perform the following operations: display a user interface screen, the user interface screen being used to set properties for a scanned image obtained by scanning a document, wherein at least a preview pane and a property pane are present on the user interface screen, the preview pane being an area for previewing and displaying the scanned image, and the property pane being an area for inputting setting values ​​for each setting item of the property; and control is performed to color at least a portion of a column of a setting item displayed in the property pane and a text block corresponding to the setting item in the scanned image previewed and displayed in the preview pane using the same color or similar colors.

[0009] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a diagram showing the overall configuration of an image processing system;

[0011] Figure 2 is a block diagram showing the hardware configuration of an information processing device;

[0012] Figure 3 is a diagram showing a software configuration of an image processing system;

[0013] Figure 4A This is a diagram that explains the state transition of the UI screen;

[0014] Figure 4B is a diagram illustrating an orthogonal coordinate system;

[0015] Figure 5 This is a diagram that explains the state transition of the UI screen;

[0016] Figure 6 is a flowchart showing a UI screen control process according to the first embodiment;

[0017] Figure 7 is a diagram illustrating the problem of Modification Example 2;

[0018] Figure 8 is a diagram illustrating the effect of Modification Example 2;

[0019] Figure 9 is a flowchart showing details of a coloring control process according to Modification Example 2;

[0020] Figure 10A is a graph illustrating the determination of color similarity, and Figure 10B It is a diagram illustrating the color change of the item;

[0021] Figure 11 is a flowchart showing details of a coloring control process according to Modification Example 3;

[0022] Figure 12 is a diagram illustrating the effect of Modification Example 3;

[0023] Figure 13 is a flowchart showing a UI screen control process according to the second embodiment;

[0024] Figure 14 is a diagram illustrating the state transition of the UI screen; and

[0025] Figure 15 This is a diagram explaining the state transition of the UI screen. DETAILED DESCRIPTION

[0026] Hereinafter, the present invention will be described in detail according to preferred embodiments with reference to the accompanying drawings. The configurations shown in the following embodiments are merely exemplary, and the present invention is not limited to the schematically shown configurations.

[0027] [First embodiment]

[0028] <System Configuration>

[0029] Figure 1 1 is a diagram illustrating the overall configuration of an image processing system for computerizing documents according to this embodiment. The image processing system includes a scanned document processing server 111, a client terminal 121, and a work server 131. Client terminal 121 is connected to scanned document processing server 111 and work server 131 via a network 101, such as the Internet or an intranet, to enable communication. Client terminal 121 may be an MFP (Multi-Function Peripheral) with a scanner function, in addition to a personal computer, laptop computer, tablet computer, smartphone, or the like. Figure 1 The configuration of the image processing system shown in FIG is an example, and the configuration is not limited thereto. For example, the client terminal 121 may also have the function of the scan document processing server 111. In the following description, a scanned image obtained by scanning a processing target document such as an estimation form is referred to as a "scanned document."

[0030] <Hardware Configuration>

[0031] Figure 2 1 is a block diagram showing the hardware configuration of the scan document processing server 111, the client terminal 121, and the work server 131, each of which serves as an information processing device. The user interface (UI) 201 is implemented by a display, keyboard, mouse, buttons, touch panel, etc., and inputs and outputs information and signals. A computer that does not include such hardware can also be connected and operated from another computer via a remote desktop, remote shell, etc. The network interface 202 connects to the network 101 and communicates with another information processing device and a network device not shown schematically. The communication method can be wired or wireless. The CPU 203 is a computing device responsible for controlling the entire device and executing programs read from the ROM 204, RAM 205, auxiliary storage device 206, etc. The ROM 204 stores incorporated programs and various data. The RAM 205 temporarily stores various data and also serves as working memory for the CPU 203. The auxiliary storage device 206 is a large-capacity storage device represented by an HDD and a flash memory. Each of the units described above is connected to each other via an input / output interface 207.

[0032] <Software Configuration>

[0033] Figure 3 1 is a diagram showing the software configuration of the image processing system according to this embodiment. Each software is installed in the above-mentioned scanned document processing server 111, client terminal 121 and work server 111 and executed by the CPU 203 of each device. In the following, each software is explained.

[0034] <<Scan Document Processing Server Software>>

[0035] The scanned document processing application 311 is a web application installed in the scanned document processing server 111. However, the scanned document processing application 311 may be installed in a form other than a web application. The API 312 is an API (Application Programming Interface) provided by the scanned document processing application 311. The Web UI 313 is a web user interface provided by the scanned document processing application 311.

[0036] The data memory 321 is a storage area for saving and storing data used by the scanned document processing application 311 or the backend application 331, described later. Data memory 321 contains a scanned document storage unit 322, a scanned document job queue 323, a property management unit 324, and a scanned image processing result storage unit 325. The scanned document storage unit 322 archives and saves the scanned image data itself as an image file such as a JPEG or a document file such as a PDF (Portable Document Format). The scanned document job queue 323 stores a queue that manages jobs that undergo property input processing, as described later. The property management unit 324 manages a list of setting items related to properties that need to be attached to each scanned document, such as file name, folder path, and metadata, as well as the format (string, number, etc.) of each setting item's name or value, and item color information, described later. The scanned document processing result storage unit 325 stores OCR processing results and business form determination results. Furthermore, the scanned document processing result storage unit 325 stores, for each scanned document, character area information (text block information), setting values, and other information associated with the setting items.

[0037] The backend application 331 is an application for performing background processing. The backend application 331 is responsible for various processes that can be performed sequentially in the background (specifically, OCR processing, business form processing, and communication processing). The OCR processing unit 332 obtains the processing target scanned document from the scanned document storage unit 322 and performs OCR processing on the obtained scanned document. During the OCR processing, information specifying each text block (specifically, information related to the starting coordinates, width, and height of the rectangular area) and information related to the recognized character string of each text block are extracted. The business form processing unit 333 determines the type of the scanned document (business form) by using information related to the processing target scanned document, the layout pattern of the text blocks obtained through OCR processing, and the recognized character strings. This determination process can be any method, such as pattern recognition and machine learning. The communication processing unit 334 performs processing to transmit the scanned document and its analysis processing results to the work server 131, which is an external device. If the transmission of the scanned document and its analysis processing results is not necessary, the communication processing unit 334 can be omitted.

[0038] <<Client Terminal Software>>

[0039] The client application 351 executes the web application of the above-mentioned scanned document processing application 311. For example, the client application 351 executes the web application by displaying the web UI 313 using a web browser and transmitting and receiving necessary data using the API 312.

[0040] <<Work Server Software>>

[0041] The work application 361 is an application used to assist in various company tasks, such as file management, document management, order acceptance, and accounting. The work data storage device 362 stores various data used by the work application 361. The work application 361 receives the results of processing by the scanned document processing server 111, performs necessary processing on the results, and stores the results.

[0042] <Client application UI screen>

[0043] Following the above, a user interface screen (hereinafter described as a “UI screen”) displayed on the user interface 201 of the client terminal 121 will be described. Figure 4A An example of a UI screen displayed on the user interface 201 by the client application 351 according to the present embodiment is shown.

[0044] Figure 4AThe UI screen shown in has a two-pane configuration of a preview pane 401 and a property pane 411. Here, within the preview pane 401, a page image 402 of a scanned document "PURCHASE ORDER" is preview-displayed. In the preview pane 401, any position in the page image 402 can be displayed by scrolling or zooming the page image 402. The property pane 411 is a pane for displaying and editing properties to be attached to the processing target scanned image. The items set as properties include main items such as "file name", "folder path" and "metadata", and sub-items such as "company name", "date" and "address", which are elements for configuring the main items. The following explanation is given by taking the case where "metadata" is set and registered as the main item as an example.

[0045] Figure 4A The middle UI screen is an example of a situation where, during the business form determination process, the business form type of the scanned document being processed is determined to be "PURCHASE ORDER." The state is such that three sub-items, "Customer Name," "Address," and "Check Recipient," are displayed as content attached to the main item "Metadata" of the scanned document. In the following description, sub-items of metadata are referred to as "metadata items."

[0046] Figure 4B 402 and each text block. Figure 4B In the image 402, there are three text blocks 421, 422, and 423. There are three more text blocks in the page image 402, but they are omitted here. The text blocks detected by OCR processing are identified by their starting point coordinates, width, and height, as shown in the shaded rectangles. For example, text block 423 corresponding to the string "XYZ Company" is represented in pixels as "Starting point coordinates (1200, 700)", "Width: 720", and "Height: 120".

[0047] Figure 5 It is to explain the text block within the page image 402 and Figure 4A 4. Here, the text block 501 corresponding to the character string "XYZ Company" within the page image 402 and the metadata item "Customer Name" (item 510a) are associated with each other. Then, in order to make it easy for the user to recognize, a rectangle colored in a certain color assigned to each metadata item based on the item color information is displayed in a transparent overlay manner on the text block 501 (in the figure). Figure 5In the example, "red" is assigned to the metadata item "Customer Name". The coloring method is not limited to the transparent overlay display and can be an overlay display of colored box lines, etc. The horizontally elongated rectangles 510a to 510c within the property pane 411 are display bars, in which the setting values, etc. of the metadata items as subordinate items are displayed and edited for the metadata as the main item. In addition, the vertically elongated rectangles 511a to 511c are item color areas, which are all set at a part within the display bar and all display the colors (item colors) assigned to each metadata item so that the user can recognize the colors. Then, the horizontally elongated rectangles 512a to 512c are all areas (clipped image areas) for displaying images (images clipped from the page image 402) corresponding to the text blocks selected by the user. At the stage where no text block has been associated yet, the state is such that "+" buttons are displayed as in the clipped image areas 512b and 512c, and the item colors of the item color areas 511b and 511c are not displayed. The horizontally elongated rectangles 513a to 513c located below each of the clipped image areas 512a to 512c are editing areas, in which the recognized strings included in the text blocks related to the user's selection are displayed, and the user can edit the displayed content by using a soft keyboard, etc. not shown schematically.

[0048] <UI Screen Control Processing>

[0049] Next, use Figure 6 The flowchart in to illustrate the control processing for associating text blocks with each metadata item on the above UI screen, which is executed by the client application 351 of the client terminal 121. Before starting to execute the process shown in Figure 6 first, the scanning document processing application 311 of the scanning document processing server 111 provides a job queue to be processed to the client application 351. When the user selects a job of a scanning document that the user wishes to start processing by using the client application 351, the scanning document processing application 311 obtains the necessary information corresponding to the selected scanning document from the data storage 321. Specifically, the scanning document processing application 311 obtains the corresponding image file from the scanning document storage unit 322 and obtains the corresponding OCR processing result and commercial form type determination result from the scanning document processing result storage unit 325. In addition, the scanning document processing application 311 obtains a list of metadata items for each commercial form type and item color information for identifying the predetermined colors assigned to each metadata item in advance from the attribute management unit 324. These pieces of data obtained are provided from the scanning document processing application 311 to the client application 351 and the process in Figure 6 is started. In the following description, the symbol "S" represents a step.

[0050] In S601, the data of the page image to be displayed in the preview pane 401 and the OCR processing result (recognized character string) are loaded. In the next S602, the item color information and the metadata item list corresponding to the business form type are loaded. In the next S603, the contents of the data loaded in S601 and S602 are reflected on the UI screen. Specifically, the page image is displayed in the preview pane 401, and each metadata item included in the metadata item list is displayed in the property pane 411. Through the processing of S601 to S603 executed so far, Figure 4A . At this point in time, the item color areas 511a to 511c of each metadata item within the property pane 411 have not yet been colored, and the item color areas 511a to 511c are in a default state (e.g., gray). This is because, along with the display of the "+" button in the cutout image area 512, the fact that the coloring has not yet been determined visually indicates to the user that metadata has not yet been set, and prompts the user to set metadata.

[0051] Next, at S604, an operation event is detected, indicating a selection operation of the "+" button for the cut image area 512a to 512c in one of the metadata items within the properties pane 411. This selection operation is performed by the user touching the "+" button on the touch panel with a finger or stylus, or by clicking with a mouse. In this manner, a metadata item associated with the user's selection is determined from among the one or more metadata items included in the metadata item list. Hereinafter, the metadata item selected by the user is referred to as the "selected metadata item." At S605, the item color area of ​​the selected metadata item determined in S604 is colored based on the item color information loaded in S602. It is assumed that, in the item color information, a specific color is associated with the order of each sub-item within the master item. For example, four colors, blue, green, yellow, and red, are predefined as available colors, and the item order and color are preliminarily associated with each other, such that the first setting item is associated with "blue," the second setting item is associated with "green," the third setting item is associated with "yellow," and the fourth setting item is associated with "red." In this case, the three sub-items of "Customer Name", "Address" and "Check Recipient" of the main item "Metadata" are in this order, and therefore, "blue", "green" and "yellow" are assigned as the item colors, respectively. By associating specific colors with the order of multiple setting items as described above, the item color information can be made common among multiple setting items. It is also possible to assign a specific color to each individual specific setting item in the item color information in advance. In addition, by highlighting the display bar of the selected metadata item on which the user has performed a selection operation (for example, such as changing the color of the area within the display bar other than the item color area), it is made easier for the user to identify the currently selected metadata item. Since the method of making it possible to identify the currently selected metadata item is not limited to the method of changing the color, and for example, the display bar of the metadata item can also be highlighted by surrounding the display bar with a thick-line frame.

[0052] Then, in S606, an operation event is detected, such as a touch operation on one of the text blocks present in the page image 402 within the preview pane 401. In this way, any one text block is selected from one or more text blocks included in the page image. Hereinafter, the text block selected by the user is referred to as the "selected text block". In the following S607, the selected text block determined by S606 is colored with the same color as the color used to perform coloring on the item color area within the selected metadata item colored in S605. Then, in S608, a cut image corresponding to the selected text block colored by S607 is displayed in the cut image area within the selected metadata item. In addition, the recognized character string included in the selected text block colored by S607 is displayed within the editing area of ​​the selected metadata item determined in S604. By the processing of S604 to S608 performed so far, the following has been brought about. Figure 5 That is, the metadata item and the text block selected by the user are associated with each other, and it is determined which text block is to be used to input the character string used in the selected metadata item.

[0053] When the above processing is completed for all metadata items within the metadata item list and the "Register" button 514 is touched by the user, a request to register the metadata is transmitted to the scanned document processing server 111 in S609. This metadata registration request includes information associating the edit values ​​in all metadata items with the text blocks. In the scanned document processing server 111, the scanned document processing application 311 stores the above information included in the metadata registration request received from the client terminal 121 in the scanned document processing result storage unit 325. At this time, when metadata registration is performed in the external work server 131, the metadata registration request is transmitted to the work server 131 via the communication processing unit 334. After completing all necessary processing, the scanned document processing application 311 dequeues the queue from the scanned document job queue 323. In this way, this processing is completed.

[0054] The above is the content of the UI screen control processing in the client application 351 according to the present embodiment. In the present embodiment, an item color area indicating the color assigned to each setting item is provided for each display bar of each setting item, but it is not mandatory to provide each individual item color area. That is, the highlighted area (the area other than the cut image area and the edit area in each display bar) used to indicate that the setting item is selected by the user can also display the item color. In the case where the configuration is performed so that the item color is displayed in the area other than the cut image area and the edit area within each display bar, it is sufficient to generate a highlight as an indication that the user has selected the setting item, for example, by attaching a thick line frame around the selected display bar, changing the color density, etc.

[0055] <Modification Example 1>

[0056] In the above Figure 6 In the process described above, the item color area is colored at the timing (S605) when the user selects the metadata item of interest (detection of a touch operation on the "+" button within the cropped image area) and the item is highlighted. However, the timing for coloring the item color area is not limited to this. For example, the item color area within the selected metadata item may be colored at the same timing as the coloring of the selected text block in S607.

[0057] <Modification Example 2>

[0058] In the above Figure 6 In the process described above, the selected text block within the preview pane is colored with the same color as the item color area of ​​the selected metadata item, allowing the user to understand the correspondence between the metadata item and the text block. In this case, there is a possibility that the color used for coloring is similar to the color of the characters or background color within the selected text block. In this case, the visual recognition of the color used for coloring the selected text block is degraded. Therefore, as Modification Example 2, an aspect of changing the color used for coloring as needed is described.

[0059] Figure 7 is a diagram illustrating the problem of this modification example, Figure 7 Corresponding to the aforementioned Figure 5 .exist Figure 7In the page image 402' in , three character strings "XYZ Corporation", "1 Pumpkin Road, New York, NY" and "(123)456-7890" in black character color are surrounded by ruled lines, and a light blue background color is attached to the inside thereof. Here, the state is such that the text block 701 corresponding to the character string "XYZ Corporation" within the page image 402' and the metadata item "Customer Name" are associated with each other. Then, among the three metadata items, "Customer Name" is the first setting item, and therefore, blue is displayed in the item color area 511a within its display bar 510a. As a result, the light blue that is the background color within the ruled lines surrounding the character string "XYZ Corporation" and the blue used to color the text block 701 are similar to each other (similar colors). In this situation, it is difficult for the user to grasp that the text block 701 and the metadata item "Customer Name" are associated with each other.

[0060] Figure 8 : is a diagram illustrating the effect of the case where this modification example is applied. Figure 7 As shown in , the state is such that the text block 701 corresponding to the character string "XYZ Company" within the page image 402' and the metadata item "Customer Name" are associated with each other. However, in Figure 8 In the metadata item "Customer Name," red is displayed in the item color area 511a instead of the initial blue. As a result, a color contrast is created between the light blue background color within the ruled lines surrounding the character string "XYZ Company" and the red coloring of the text block 701, allowing the user to easily grasp the association.

[0061] Here, reference Figure 9 The flowchart in FIG. 1 illustrates details of the process of controlling the coloring of setting items and text blocks according to this modification example. Figure 6 In the process of S607, execute Figure 9 The coloring control process is shown in the flow chart.

[0062] At S901, information related to the character colors of the character strings included in the selected text block for which an operation event has been detected at S606 and the background color of the text block is acquired from the scanned document processing result storage unit 325 via the scanned document processing application 311. Here, the description is continued assuming that information related to values ​​indicating black as the character color (R, G, B = 0, 5, 5) and values ​​indicating light blue as the background color (R, G, B = 188, 213, 254) is acquired.

[0063] In S902, information related to the color used for coloring based on the item color information in S605 (i.e., the color assigned as the item color of the selected metadata item and used to color the item color area) is obtained. Assume that "blue" is assigned to the first setting item, "green" is assigned to the second setting item, "yellow" is assigned to the third setting item, and "red" is assigned to the fourth setting item, which is defined as the default content in the item color information in this modified example. Here, in the case of selecting the first "customer name" among the three metadata items within the metadata pane 411, the item color area is colored with "blue", and therefore, in this case, information indicating "blue" (e.g., RGB value) is obtained.

[0064] In the next S903, the color assigned as the item color of the selected metadata item is compared with the character color and background color within the selected text block, and it is determined whether they are similar to each other. Figure 10A As shown in , the RGB values ​​of the color assigned as the item color of the selected metadata item are compared with the RGB values ​​of the character color and the background color acquired in S901, respectively, and it is determined whether the difference between them is within a predetermined threshold value. At this time, it is desirable to set a threshold value by which, for example, a light blue or purple color having a hue close to that of blue is determined to be "similar to" blue. Here, it is assumed that the blue color assigned as the item color of the selected metadata item "customer name" is determined to be similar to the light blue color that is the background color of the selected text block. In the case where it is determined in this way that the color assigned as the item color of the selected metadata item is similar to the character color or background color within the selected text block, the processing proceeds to S904. On the other hand, in the case where it is determined that they are not similar, the processing proceeds to S905.

[0065] At S904, a process is performed to change the color currently assigned to the item color of the selected metadata item to another different color defined in the item color information. Specifically, the association between the order of the items in the candidate color information and each color is changed, and a different color is assigned to the selected metadata item. In the case of the above example, the order of the selected metadata item "customer name" is the first. Therefore, if Figure 10BAs shown in , the color is changed to the color associated with the fourth selected metadata item that is last in order and not used in the above example (in this case, "red"), and processing is performed to associate "blue" with the fourth selected metadata item. By exchanging colors between the first and fourth selected metadata items in this manner, the color association between the second and third selected metadata items can be maintained, and therefore, the impact of the change can be suppressed to a minimum. After the color that is the item color of the selected metadata item is changed in this manner and the item color information is updated according to the change, the process returns to S903 and similarity determination is performed again. This process is repeated until the determination result indicates that the color assigned as the item color is no longer similar to the character color or background color within the selected text block.

[0066] Then, in S905, the color of the item color area of ​​the selected metadata item is updated as needed. That is, if the color assigned as the item color of the selected metadata item is changed to another color in S904, the newly changed color is used for coloring.

[0067] At S906, the selected text block is colored by using the same color as the color displayed in the item color area of ​​the selected metadata item. Figure 8 The state of the UI screen shown in . When the rendering is completed, the process returns Figure 6 , and execute the display processing of S608.

[0068] The above is an illustration of the contents of the coloring control processing according to Modified Example 2. As described in the first embodiment, control can also be performed so that the item color area within the selected metadata item is not colored until the selected text block is colored, and the coloring of both the item color area and the selected text block is performed at the same timing.

[0069] <Modification Example 3>

[0070] As the number of setting items increases, it may become difficult to select a color that is not similar to the character color or background color from the colors defined in the item color information and assign that color. Therefore, as Modification Example 3, an aspect of determining and assigning a color as the item color for each setting item without basing this determination and assignment on the item color information is described.

[0071] refer to Figure 11 The flowchart in FIG. 1 illustrates the details of the process for controlling the coloring of the setting items and text blocks according to this modified example. Figure 6 In the process of S602, execute Figure 11 The coloring control process shown in the flow chart is then Figure 6 In S605 of the flow in , in the case where the present modification example is applied, the item color area is colored by using a color determined by a process described later, instead of a color determined based on the item color information.

[0072] At S1101, a metadata item list is loaded. That is, in the case of this modified example, item color information is not loaded. At the next S1102, information related to the character colors of the character strings included in all text blocks within the page image and the background color of the text blocks is obtained from the OCR processing results loaded at S601.

[0073] At S1103, among the character colors and background colors obtained at S1102, a chromatic color with a high frequency of occurrence is specified. For example, it is assumed that there are 100 text blocks within the page image, and black is used as all character colors. On the one hand, for the background color, it is assumed that there are 80 text blocks using white, 18 text blocks using light blue, and two text blocks using yellow-green. In this case, the chromatic colors are light blue and yellow-green, and therefore, the light blue color with a large number of uses is specified as the chromatic color with a high frequency of occurrence. The reason for excluding black and white as non-chromatic colors is that they are not suitable for use as item colors intended for identification, because black and white are commonly used in business forms, etc.

[0074] In S1104, based on the frequently appearing chromatic colors specified in S1103, an item color is determined for each metadata item. An example of a specific determination method is as follows.

[0075] The first method is a method using a hue circle. In this case, a color located at a distant position in the hue circle is determined as an item color for each metadata item, with a frequently occurring color as a reference. For example, in a case where a frequently occurring color is light blue and this is used as a reference, the color located at the farthest position in the hue circle is orange, and therefore, the color assigned as the item color is determined within a predetermined range (e.g., a range from yellow to purple) centered on orange. At this time, in a case where there are multiple setting items to which item colors are to be assigned and it is necessary to determine multiple colors, it is sufficient to select multiple colors so that, for example, the intervals in the hue circle are substantially equal in order to distinguish between the multiple colors.

[0076] The second method is a method of determining a color that is not similar to a frequently occurring colored color from among a plurality of candidate colors determined in advance as an item color. For example, five colors, red, green, yellow, blue, and purple, are determined in advance as candidate colors. Then, in the case where the frequently occurring colored color is light blue, in addition to the blue that is similar to light blue, the color is sequentially assigned as the item color from the remaining four colors (red, green, yellow, purple). It is sufficient to determine the number of candidate colors at this time based on the format of the business form assumed to be the processing target, etc. In addition, when determining the colors to be excluded, it is sufficient to perform the same similarity determination as described at S903 of Modified Example 2.

[0077] The two methods described above are examples, and any method may be used as long as the method can assign, as an item color, a color that a user can easily distinguish from another color based on a frequently appearing chromatic color.

[0078] Figure 12 is a diagram illustrating the effects of applying this modified example. In this example, the background color within the grid surrounding the character string "XYZ Company" is light blue. Therefore, three colors dissimilar to light blue are determined and assigned as item colors. Specifically, red, orange, and yellow are assigned as item colors to the three metadata items "Customer Name," "Address," and "Total Price," respectively. Consequently, the colors used to color the three text blocks 1201 to 1203 are eye-catching, allowing the user to easily grasp the association between the colors and the selected text block corresponding to each selected metadata item.

[0079] The above is an illustration of the contents of the coloring control processing according to Modification Example 3. A plurality of colors can be designated as frequently appearing chromatic colors. In addition, frequently appearing chromatic colors can be designated by taking only the text blocks existing in a partial area such as the top of the page image as a target instead of all the text blocks existing within the page image. As in Modification Example 2, control can also be performed so that the project color area within the selected metadata area is not colored until the selected text block is colored, and coloring of both the project color area and the selected text block is performed at the same timing.

[0080] As described above, according to this embodiment, even when there are multiple text blocks within a page image and multiple setting items (sub-items) within a property pane on a UI screen having a preview pane and a property pane, the user can easily grasp the correspondence therebetween.

[0081] [Second embodiment]

[0082] In the first embodiment, the UI screen control in the case of assuming a one-to-one association where one text block is associated with one metadata item was described. Next, as the second embodiment, the UI screen control that can also handle the case of performing a one-to-N association will be described, in which the same text block is associated with multiple metadata items. The description of those contents (such as the system configuration) common to the first embodiment is omitted, and hereinafter, the differences will be mainly described.

[0083] <UI Screen Control Process>

[0084] Use Figure 13 The flowchart in [ ] is used to illustrate the UI screen control process according to this embodiment, and this process is executed by the client application 351 of the client terminal 121. Here, the description is given on the premise of the following assumption: Along the process in the first embodiment, an association has been made between each of the three metadata items "Customer Name", "Address", and "Check Receiver" and the text block, and the state of the UI screen shown in [ ] has been brought about. Figure 6 In [ ], an association has been made between each of the three metadata items "Customer Name", "Address", and "Check Receiver" and the text block along the process in the first embodiment, and the state of the UI screen shown in [ ] has been brought about. Figure 14 shown in [ ]

[0085] On the Figure 14 UI screen in [ ], at the upper left of the text block 501 within the page image 402, a label of the metadata item associated with the text block (here, "two") is attached. This label is displayed at the timing of associating the same text block with multiple metadata items (that is, at the time point when the number of metadata items becomes two or more), but this is not limited, and the label may also be displayed at the timing of associating the text block with one of the metadata items. Then, here, in the case of associating with multiple metadata items, the text block 501 is colored in the default color (here, gray). In contrast, the text block 1402 associated with only one metadata item "Address" is colored blue as indicated by the corresponding item color area 511b.

[0086] Assume that the UI screen is in Figure 14In the case of the state shown in , the user has selected (using a touch operation of a finger, etc.) one of the display columns 510a to 510c of each metadata item within the property pane 411. At S1301, an operation event such as a touch operation of the user is detected. Here, it is assumed that an operation event for the display column 510c "Check Recipient" is detected. In the following S1302, it is determined whether the text block associated with the metadata item selected by the user is also associated with another metadata item. In the case where the determination result is "No", this processing is completed. On the other hand, in the case where the determination result is "Yes", the processing proceeds to S1303. Here, the text block 501 is associated with the metadata item "Check Recipient" selected by the user, and in addition, the text block 501 is also associated with the metadata item "Customer Name". Therefore, the determination result in this case is "Yes", and therefore the processing proceeds to S1303.

[0087] At step S1303, the text block associated with the selected metadata item is colored with the same color as the item color area of ​​the selected metadata item. Here, the color of the text block 501 is changed from gray, which is the default color, to yellow, as indicated by the item color area 511c within the display column of the metadata item "Check Recipient." Figure 15 The UI screen shown in shows a state in which the color of the text block 501 has been changed from gray to yellow in conjunction with the selection of the metadata item "check recipient."

[0088] In the next S1304, a button for canceling the association between the selected metadata item and the text block is displayed. Figure 15 On the UI screen in FIG. 5 , the "×" mark button 1501 that appears in the upper right corner of the text block 501 indicates a cancel button. In the next step S1305, it is determined whether a touch operation or the like has been performed on the cancel button. If an operation event such as a touch operation on the cancel button is detected, the process proceeds to S1306, and if no operation event is detected, the process is exited.

[0089] In S1306, the association between the selected metadata item and the text block is canceled, and the label indicating the number of currently associated metadata items is updated. Specifically, as a result of canceling the association with the selected metadata item, "one" is subtracted from the number of metadata items indicated by the label. Figure 15 In the example shown in FIG, the value of label 1401 changes from "two" to "one." In addition, the display field corresponding to the selected metadata item is initialized and brought to a blank state. Here, the selected metadata item is "Check Recipient," and therefore, the corresponding display field 510c is initialized.

[0090] The above is the content of the UI screen control process in the client application 351 according to this embodiment.

[0091] As described above, according to the present embodiment, even in a case where the same text block is associated with a plurality of setting items (sub-items), the user can accurately and easily grasp the correspondence relationship between the sub-items and the text block.

[0092] [Other embodiments]

[0093] The embodiments of the present invention can also be implemented by the following method, that is, providing software (program) that performs the functions of the above-mentioned embodiments to a system or device through a network or various storage media, and the computer or central processing unit (CPU) or microprocessing unit (MPU) of the system or device reads and executes the program.

[0094] According to the technology of the present invention, on a UI screen having a preview pane and a property pane, a user can easily grasp the correspondence relationship between text blocks and setting items.

[0095] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An information processing device comprising a control unit configured to display a user interface screen including a document image and a plurality of columns, in, The control unit displays character recognition results obtained from each of a plurality of regions in the document image in a corresponding column among the plurality of columns, and displays the plurality of regions in such a manner that the plurality of regions can be distinguished from any region other than the plurality of regions, and In which, the control unit colors the column selected by the user from the multiple columns and the area selected by the user in the multiple areas in a manner such that both the column and the area can be distinguished from other columns in the multiple columns except the column and other areas in the multiple areas except the area, so that the user can grasp the correspondence between the column and the area, wherein the user selects the area to input the character recognition result obtained from the area into the column.

2. The information processing device according to claim 1, wherein The column is colored by coloring at least a portion of the column.

3. The information processing device according to claim 1, wherein The control unit displays the plurality of regions in such a manner that the plurality of regions can be distinguished from any regions other than the plurality of regions by displaying a colored rectangle on each of the plurality of regions in a transparent overlay manner.

4. The information processing device according to claim 3, wherein The colored rectangles displayed on the plurality of areas in a transparent overlay manner are different in color from one another.

5. The information processing device according to claim 3, wherein The colored rectangle displayed on the area in a transparent overlay manner has a color that is different from the color of the colored rectangles displayed on other areas among the multiple areas in a transparent overlay manner except the area, and enables the user to grasp the correspondence between the column and the area. The information processing device according to claim 5 , wherein: The color used to color the column so that the user can grasp the correspondence between the column and the area is the same as the color of the colored rectangle displayed in a transparent overlay manner on the area.

7. The information processing device according to claim 1, wherein The control unit displays the plurality of regions in such a manner that the plurality of regions can be distinguished from any regions other than the plurality of regions by displaying a colored frame for each of the plurality of regions.

8. The information processing device according to claim 7, wherein The colored frames displayed for the plurality of areas are different in color from one another.

9. The information processing device according to claim 7, wherein The colored frame displayed for the area has a color that is different from a color of the colored frames displayed for other areas among the plurality of areas except for the area and enables a user to grasp the correspondence between the column and the area.

10. The information processing device according to claim 9, wherein The color used to color the column so that the user can grasp the correspondence between the column and the area is the same as the color of the colored frame displayed for the area.

11. The information processing device according to claim 1, wherein The plurality of fields are fields into which values ​​corresponding to a plurality of setting items are respectively input.

12. The information processing device according to claim 1, wherein The control unit colors the column at a timing when an operation event for selecting the column is detected.

13. The information processing device according to claim 1, wherein The control unit colors the column and the region so as to be distinguishable from the other columns and the other regions at a timing when the user selects the region.

14. The information processing device according to claim 1, wherein The color used by the control unit to color the column and the area in a manner that can be distinguished from the other columns and the other areas is a color that is specified based on predefined color information and enables the user to grasp the correspondence between the column and the area.

15. The information processing device according to claim 14, wherein The color information predefines colors corresponding to the order of the plurality of columns, and The color used by the control unit to color the column and the area in a manner that can be distinguished from the other columns and the other areas is a color that is specified based on the predefined color information and enables the user to grasp the correspondence between the column and the area.

16. The information processing device according to claim 14, wherein When the color specified based on the predefined color information is similar to the character color or background color in the area selected by the user's operation, the control unit changes the color to the following color, and uses the changed color to color the column and the area, where the color is not similar to the character color or the background color and enables the user to grasp the correspondence between the column and the area.

17. The information processing device according to claim 1, wherein The plurality of areas are text boxes from which character recognition results input into respective ones of the plurality of fields are obtained, and Any area other than the plurality of areas includes a text box different from the plurality of areas.

18. The information processing device according to claim 1, wherein The user interface screen includes a pane for displaying the document image and a pane for displaying the plurality of columns.

19. The information processing device according to claim 1, wherein The document image is a document image obtained by scanning a document.

20. A computer-readable storage medium storing a program for causing a computer to function as a control unit configured to display a user interface screen including a document image and a plurality of columns, in, The control unit displays character recognition results obtained from each of a plurality of regions in the document image in a corresponding column among the plurality of columns, and displays the plurality of regions in such a manner that the plurality of regions can be distinguished from any region other than the plurality of regions, and In which, the control unit colors the column selected by the user from the multiple columns and the area selected by the user in the multiple areas in a manner such that both the column and the area can be distinguished from other columns in the multiple columns except the column and other areas in the multiple areas except the area, so that the user can grasp the correspondence between the column and the area, wherein the user selects the area to input the character recognition result obtained from the area into the column.

21. The computer-readable storage medium of claim 20, wherein: The column is colored by coloring at least a portion of the column.

22. The computer-readable storage medium of claim 20, wherein: The control unit displays the plurality of regions in such a manner that the plurality of regions can be distinguished from any regions other than the plurality of regions by displaying a colored rectangle on each of the plurality of regions in a transparent overlay manner.

23. The computer-readable storage medium of claim 22, wherein: The colored rectangles displayed on the plurality of areas in a transparent overlay manner are different in color from one another.

24. The computer-readable storage medium of claim 22, wherein: The colored rectangle displayed on the area in a transparent overlay manner has a color that is different from the color of the colored rectangles displayed on other areas among the multiple areas in a transparent overlay manner except the area, and enables the user to grasp the correspondence between the column and the area.

25. The computer-readable storage medium of claim 24, wherein: The color used to color the column so that the user can grasp the correspondence between the column and the area is the same as the color of the colored rectangle displayed in a transparent overlay manner on the area.

26. The computer-readable storage medium of claim 20, wherein: The control unit displays the plurality of regions in such a manner that the plurality of regions can be distinguished from any regions other than the plurality of regions by displaying a colored frame for each of the plurality of regions.

27. The computer-readable storage medium of claim 26, wherein: The colored frames displayed for the plurality of areas are different in color from one another.

28. The computer-readable storage medium of claim 26, wherein: The colored frame displayed for the area has a color that is different from a color of the colored frames displayed for other areas among the plurality of areas except for the area and enables a user to grasp the correspondence between the column and the area.

29. The computer-readable storage medium of claim 28, wherein: The color used to color the column so that the user can grasp the correspondence between the column and the area is the same as the color of the colored frame displayed for the area.

30. The computer-readable storage medium of claim 20, wherein: The plurality of fields are fields into which values ​​corresponding to a plurality of setting items are respectively input.

31. The computer-readable storage medium of claim 20, wherein: The control unit colors the column at a timing when an operation event for selecting the column is detected.

32. The computer-readable storage medium of claim 20, wherein: The control unit colors the column and the region so as to be distinguishable from the other columns and the other regions at a timing when the user selects the region.

33. The computer-readable storage medium of claim 20, wherein: The color used by the control unit to color the column and the area in a manner that can be distinguished from the other columns and the other areas is a color that is specified based on predefined color information and enables the user to grasp the correspondence between the column and the area.

34. The computer-readable storage medium of claim 33, wherein: The color information predefines colors corresponding to the order of the plurality of columns, and The color used by the control unit to color the column and the area in a manner that can be distinguished from the other columns and the other areas is a color that is specified based on the predefined color information and enables the user to grasp the correspondence between the column and the area.

35. The computer-readable storage medium of claim 33, wherein: When the color specified based on the predefined color information is similar to the character color or background color in the area selected by the user's operation, the control unit changes the color to the following color, and uses the changed color to color the column and the area, where the color is not similar to the character color or the background color and enables the user to grasp the correspondence between the column and the area.

36. The computer-readable storage medium of claim 20, wherein: The plurality of areas are text boxes from which character recognition results input into respective ones of the plurality of fields are obtained, and Any area other than the plurality of areas includes a text box different from the plurality of areas.

37. The computer-readable storage medium of claim 20, wherein: The user interface screen includes a pane for displaying the document image and a pane for displaying the plurality of columns.

38. The computer-readable storage medium of claim 20, wherein: The document image is a document image obtained by scanning a document.

39. A control method for a control unit of a device to execute a program to display a user interface screen, the user interface screen including a document image and a plurality of columns, in, The control unit displays character recognition results obtained from each of a plurality of regions in the document image in a corresponding column among the plurality of columns, and displays the plurality of regions in such a manner that the plurality of regions can be distinguished from any region other than the plurality of regions, and In which, the control unit colors the column selected by the user from the multiple columns and the area selected by the user in the multiple areas in a manner such that both the column and the area can be distinguished from other columns in the multiple columns except the column and other areas in the multiple areas except the area, so that the user can grasp the correspondence between the column and the area, wherein the user selects the area to input the character recognition result obtained from the area into the column.

Images