Image processing system for digitizing document, control method thereof and program

The image processing system addresses the challenge of associating text blocks with setting items by coloring the display field in the property pane to match the selected text block color, improving user comprehension and efficiency.

JP2024164222AActive Publication Date: 2024-11-26CANON KK
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
JP2024145994
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2020-01-21
Filing Date
2024-08-27
Publication Date
2024-11-26
Estimated Expiration
2040-02-27

AI Technical Summary

Technical Problem

Users face difficulty in understanding the correspondence between text blocks and setting items on a UI screen with a preview pane and a property pane during document digitization, especially when there are multiple text blocks and setting items.

Method used

An image processing system that colors part of the display field corresponding to a setting item in the property pane to match the assigned color of the selected text block in the preview pane, facilitating easy association.

Benefits of technology

Enhances user understanding of the correspondence between text blocks and setting items on the UI screen, making the association process more intuitive and efficient.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024164222000001_ABST
    Figure 2024164222000001_ABST
Patent Text Reader

Abstract

To provide an image processing system and a control method thereof for enabling a user to easily grasp which text block in a preview pane is associated with a setting item in a property pane.SOLUTION: A user interface screen of an image processing system for digitizing a document has at least a preview pane 401, which is an area for displaying preview of a scanned image and a property pane 411, which is an area for entering set values for each property setting item. At least a part of a display field corresponding to the setting item displayed in the property pane is colored using a color assigned to the setting item, and when a user selects any text block of the scanned image previewed in the preview pane in a state in which any setting item is selected in the property pane, the text block is colored using a color assigned to the setting item in the selected state.SELECTED DRAWING: Figure 5
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a graphical user interface (GUI) for use in digitizing and utilizing documents. [Background technology]

[0002] Conventionally, optical character recognition (OCR) processing is performed on a scanned image obtained by scanning a document such as an order form to extract character strings, which are then used for file names, etc. Patent Document 1 discloses a technology for extracting character strings containing a specific key from character strings contained in a document image, automatically generating a file name, and presenting the file name to a user together with a preview display of the document image. This technology allows a user to easily set a file name while checking the document image. Character strings extracted from a scanned image are also stored and managed as metadata, and used for file names, etc. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-115011 A Summary of the Invention [Problem to be solved by the invention]

[0004] A two-pane configuration consisting of a preview pane and a property pane is well known as a UI screen configuration of an application when setting a file name and metadata using a recognized character string obtained by OCR processing. Here, the preview pane is an area for previewing the scanned image to be digitized, and the property pane is an area for displaying a list of the contents of properties such as the file name and metadata. When digitizing documents such as purchase orders and invoices, the scanned image contains multiple character areas (text blocks), and the preview pane allows the user to visually check each text block. In addition, the property pane allows the user to check the contents of various setting items such as "Customer Name", "Date", and "Address". For example, in the above two-pane UI screen, a user selects the text block "XYZ Corporation" in the preview pane for the setting item "Customer Name" in the property pane. As a result, the recognized character string "XYZ Corporation" in the scanned image is saved in association with the setting item "Customer Name". In this case, if there are multiple text blocks in the preview pane and multiple setting items in the property pane, it is difficult for the user to immediately understand which text block is associated with which setting item.

[0005] This invention has been made in consideration of the above-mentioned problems, and aims to enable users to easily understand the correspondence between text blocks and setting items on a UI screen having a preview pane and a property pane. [Means for solving the problem]

[0006] The image processing system for digitizing documents according to the present disclosure comprises a display means for displaying a user interface screen for setting properties for a scanned image obtained by scanning the document, and a control means for controlling the display on the user interface screen, wherein the user interface screen has at least a preview pane which is an area for previewing the scanned image, and a property pane which is an area for inputting setting values ​​for each setting item of the properties, and wherein the control means colors at least a portion of a display field corresponding to a setting item displayed in the property pane using a color assigned to the setting item, and when a setting item is selected in the property pane, when a user selects a text block of the scanned image previewed in the preview pane, the control means colors the text block using the color assigned to the selected setting item. Effect of the Invention

[0007] According to the technology of the present disclosure, a user can easily grasp the correspondence between text blocks and setting items on a UI screen having a preview pane and a property pane. [Brief description of the drawings]

[0008] [Figure 1] A diagram showing the overall configuration of an image processing system. [Diagram 2] A block diagram showing a hardware configuration of an information processing device. [Diagram 3] A diagram showing the software configuration of the image processing system. [Figure 4] A diagram explaining the state transitions of the UI screen [Diagram 5] A diagram explaining the state transitions of the UI screen [Figure 6] 1 is a flowchart showing the flow of a UI screen control process according to the first embodiment. [Figure 7] FIG. 1 is a diagram explaining the problem of the second modification example. [Figure 8]FIG. 13 is a diagram for explaining the effect of the second modified example. [Figure 9] 11 is a flowchart showing details of a coloring control process according to Modification 2. [Figure 10] (a) is a diagram explaining the judgment of similarity of colors, and (b) is a diagram explaining the change of item colors. [Figure 11] 11 is a flowchart showing details of a coloring control process according to Modification 3. [Figure 12] FIG. 13 is a diagram for explaining the effect of Modification 3. [Figure 13] 11 is a flowchart showing the flow of a UI screen control process according to the second embodiment. [Figure 14] A diagram explaining the state transitions of the UI screen [Figure 15] A diagram explaining the state transitions of the UI screen DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments of the present invention will be described with reference to the drawings. Note that the following embodiments do not limit the invention according to the claims, and all of the combinations of features described in the embodiments are not necessarily essential to the solution of the invention.

[0010] [Embodiment 1] <System configuration> FIG. 1 is a diagram showing the overall configuration of an image processing system for digitizing a document according to the present embodiment. The image processing system includes a scanned document processing server 111, a client terminal 121, and a business server 131. The client terminal 121 is communicably connected to the scanned document processing server 111 and the business server 131 via a network 101 such as the Internet or an intranet. The client terminal 121 may be a personal computer, a laptop computer, a tablet computer, a smartphone, or an MFP (Multifunction Peripheral) equipped with a scanner function. The configuration of the image processing system shown in FIG. 1 is an example, and is not limited thereto. For example, the client terminal 121 may also have the function of the scanned document processing server 111. In the following description, a scanned image obtained by scanning a document to be processed such as an estimate is called a "scanned document."

[0011] <Hardware configuration> FIG. 2 is a block diagram showing the hardware configuration of the scan document processing server 111, the client terminal 121, and the business server 131 as information processing devices. The user interface (UI) 201 is realized by a display, a keyboard, a mouse, buttons, a touch panel, etc., and inputs and outputs information and signals. A computer that does not have these hardware can be connected and operated from another computer by remote desktop, remote shell, etc. The network interface 202 connects to the network 101 and communicates with other information processing devices and network devices (not shown). The communication method may be either wired or wireless. The CPU 203 is a calculation device that controls the entire device, and executes programs read from the ROM 204, RAM 205, secondary storage device 206, etc. The ROM 204 records the embedded programs and various data. The RAM 205 temporarily stores various data and also functions as a work memory for the CPU 203. The secondary storage device 206 is a large-capacity storage device represented by a HDD or a flash memory. The above-mentioned parts are mutually connected via an input / output interface 207.

[0012] <Software configuration> 3 is a diagram showing the software configuration of the image processing system according to this embodiment. Each software is installed in the above-mentioned scanned document processing server 111, client terminal 121, and business server 131, and executed by each CPU 203. Each software will be described below.

[0013] <Scan document processing server software> The scanned document processing application 311 is a Web application installed in the scanned document processing server 111. However, the application may be implemented in a form other than a Web application. The API 312 is an API (Application Programming Interface) provided by the scanned document processing application 311. The Web UI 313 is a Web user interface provided by the scanned document processing application 311.

[0014] The data store 321 is a storage area for saving and storing data used by the scanned document processing application 311 or the back-end application 331 described later. The data store 321 includes a scanned document storage unit 322, a scanned document job queue 323, a property management unit 324, and a scanned document processing result storage unit 325. The scanned document storage unit 322 converts the data of the scanned document itself into a file and stores it as an image file such as JPEG or a document file such as PDF (Portable Document Format). The scanned document job queue 323 holds a queue for managing jobs waiting for property input processing described later. The property management unit 324 manages a list of setting items, the names and value formats (character strings, numbers, etc.) for each setting item, and item color information described later, related to properties such as file names, folder paths, and metadata that need to be added for each scanned document. The scanned document processing result storage unit 325 stores the OCR processing results and the form discrimination results. In addition, the scanned document processing result storage unit 325 stores character area information (text block information) associated with the setting items, set values, and the like, for each scanned document.

[0015] The back-end application 331 is an application for executing background processing. The back-end application 331 is an application in charge of various processes that may be executed sequentially in the background, specifically, OCR processing, form processing, and communication processing. The OCR processing unit 332 acquires a scanned document to be processed from the scanned document storage unit 322, and executes OCR processing on the acquired scanned document. In the OCR processing, information for identifying each text block (specifically, information on the starting point coordinates, width, and height of a rectangular area) and information on the recognized character string for each text block are extracted. The form processing unit 333 uses the scanned document to be processed and information such as the arrangement pattern of the text block and the recognized character string obtained by the OCR processing to determine the type of the document (form) that was the subject of the scan. This determination processing may be any method such as pattern recognition or machine learning. The communication processing unit 334 performs a process of transmitting the scanned document and the analysis processing result thereof to the business server 131, which is an external device. When it is not necessary to transmit the scanned document and the analysis processing result thereof, the communication processing unit 334 can be omitted.

[0016] <Client terminal software> The client application 351 executes the Web application of the above-mentioned scan document processing application 311. For example, the Web application is executed by displaying the Web UI 313 in a Web browser and transmitting / receiving necessary data to / from the API 312.

[0017] <Business server software> The business application 361 is an application for supporting various business operations in a company, such as file management, document management, order taking, and accounting. The business data storage 362 stores various data used by the business application 361. The business application 361 receives the results of processing by the scanned document processing server 111, performs necessary processing on them, and stores them.

[0018] <Client application UI screen> Next, a description will be given of a user interface screen (hereinafter, referred to as a "UI screen") displayed on the user interface 201 of the client terminal 121. Fig. 4(a) shows an example of a UI screen displayed on the user interface 201 by the client application 351 according to this embodiment.

[0019] The UI screen shown in FIG. 4(a) has two panes: a preview pane 401 and a property pane 411. A page image 402 of a scanned document of "PURCHASE ORDER" is currently displayed in the preview pane 401. In the preview pane 401, the page image 402 can be scrolled and zoomed to display any position in the page image 402. The property pane 411 is a pane for displaying and editing properties to be assigned to the scanned document to be processed. Items to be set as properties include parent items such as "file name", "folder path", and "metadata", and child items such as "company name", "date", and "address", which are elements that make up the parent items. In the following explanation, an example will be given in which "metadata" is set and registered as a parent item.

[0020] The UI screen in Fig. 4(a) is an example of a case where the form type of the scanned document to be processed has been determined to be a "purchase order" in the form discrimination process. Three child items, "Customer Name," "Address," and "Billing Name," are displayed as the contents of the parent item, "metadata," to be assigned to the scanned document. In the following explanation, the child items of metadata will be referred to as "metadata items."

[0021] FIG. 4(b) is a diagram for explaining an orthogonal coordinate system showing the relationship between the page image 402 and each text block. In FIG. 4(b), there are three text blocks 421, 422, and 423. There are many more text blocks in the page image 402, but they are omitted here. The text blocks detected by the OCR process are identified by their starting coordinates, width, and height, as shown by the shaded rectangles. For example, the text block 423 corresponding to the character string "XYZ Corporation" is expressed in pixels as "start coordinates (1200, 700)," "width: 720," and "height: 120."

[0022] FIG. 5 is a diagram for explaining a state in which a text block in the page image 402 is associated with a metadata item in the UI screen of FIG. 4(a). Now, a text block 501 corresponding to the character string "XYZ Corporation" in the page image 402 and a metadata item "Customer Name" (the item of 510a) are in an associated state. And, so that the user can easily identify, a rectangle colored with a color (in the example of FIG. 5, "red" assigned to the metadata item "Customer Name") assigned based on the item color information for each metadata item is transparently superimposed and displayed on the text block 501. Note that the coloring method is not limited to the transparent superimposed display, and a superimposed display of a colored frame line or the like may be used. The horizontally long rectangles 510a to 510c in the property pane 411 are display fields for displaying and editing the set values and the like of the metadata items that are child items for the metadata that is the major item. Also, the vertically long rectangles 511a to 511c are item color areas provided in a part of the display field for displaying the colors (item colors) assigned to each metadata item so that the user can identify them. And, the horizontally long rectangles 512a to 512c are areas (cropped image areas) where an image (an image cropped from the page image 402) of the part corresponding to the text block selected by the user is displayed. At a stage where no text block is associated, a "+" button is displayed as in the cropped image areas 512b and 512c, and further, the item colors in the item color areas 511b and 511c are not displayed. The horizontally long rectangles 513a to 513c below each of the cropped image areas 512a to 512c are editing areas where the recognized character strings included in the text block related to the user selection are displayed, and the user can edit the displayed content using a soft keyboard or the like (not shown).

[0023] <UI Screen Control Process> Next, a control process of associating text blocks with each metadata item on the UI screen, which is performed by the client application 351 of the client terminal 121, will be described with reference to the flowchart of FIG. 6. Prior to the start of execution of the flow shown in FIG. 6, first, a queue of items waiting to be processed is provided to the client application 351 by the scan document processing application 311 of the scan document processing server 111. When a user selects a job of a scan document to be processed using the client application 351, the scan document processing application 311 acquires necessary information corresponding to the selected scan document from the data store 321. Specifically, the scan document processing application 311 acquires a corresponding image file from the scan document storage unit 322, and an associated OCR processing result and a form discrimination processing result from the scan document processing result storage unit 325. Furthermore, the scan document processing application 311 acquires a list of metadata items for each form type and item color information for identifying a predetermined color assigned in advance to each metadata item from the property management unit 324. The scan document processing application 311 provides the acquired data to the client application 351, and the flow shown in FIG. 6 starts. In the following description, the symbol "S" represents a step.

[0024] In S601, data of a page image and an OCR processing result (recognized character string) to be displayed in the preview pane 401 are loaded. In the following S602, a metadata item list and item color information corresponding to the form type are loaded. In the next S603, the contents of the data loaded in S601 and S602 are reflected on the UI screen. Specifically, the page image is displayed in the preview pane 401, and each metadata item included in the metadata item list is displayed in the property pane 411. The processing of S601 to S603 up to this point results in the UI screen state shown in FIG. 4. At this point, coloring of the item color areas 511a to 511c of each metadata item in the property pane 411 is reserved and is in a default state (for example, gray). This is to visually express the fact that the coloring is undecided together with the "+" button display in the cut image area 512 described above, thereby clearly indicating to the user that the metadata has not been set, and to urge the user to set the metadata.

[0025] Then, in S604, an operation event of a selection operation of the "+" button in the cut image area 512a to 512c in any of the metadata items in the property pane 411 is detected. This selection operation is an operation in which the user touches the "+" button on the touch panel with a finger or a touch pen, or a click operation with a mouse. In this way, one metadata item related to the user's selection is determined from one or more metadata items included in the metadata item list. Hereinafter, the metadata item selected by the user is referred to as a "selected metadata item." In the following S605, the item color area of ​​the selected metadata item determined in S604 is colored according to the item color information loaded in S602. Here, it is assumed that a specific color is associated with the order of each child item existing in the parent item in the item color information. For example, four colors, blue, green, yellow, and red, are defined in advance as available colors, and the item order and color are associated with each other, such as "blue" for the first setting item, "green" for the second setting item, "yellow" for the third setting item, and "red" for the fourth setting item. In this case, the three child items of the parent item "metadata" are "Customer Name", "Address", and "Billing Name" in this order, so that "blue", "green", and "yellow" are assigned as item colors to each of them. In this way, by associating specific colors with the order of a plurality of setting items, item color information can be made common among various types of setting items. In addition, in the item color information, specific colors may be assigned in advance to individual specific setting items. In addition, the display field of the selected metadata item selected by the user is highlighted by, for example, changing the color of the area other than the item color area in the display field, so that the user can easily identify the currently selected metadata item. In addition, the method of identifying the currently selected metadata item is not limited to changing the color, and for example, the display field of the metadata item may be highlighted by surrounding it with a thick frame.

[0026] Then, in S606, an operation event such as a touch operation on any of the text blocks present in the page image 402 in the preview pane 401 is detected. In this way, any one of one or more text blocks included in the page image is selected. Hereinafter, the text block selected by the user is referred to as a "selected text block." In the following S607, the selected text block determined in S606 is colored in the same color as the color of the item color area in the selected metadata item colored in S605. Then, in S608, a cut image corresponding to the selected text block colored in S607 is displayed in the cut image area in the selected metadata item. Furthermore, the recognized character string included in the selected text block colored in S607 is displayed in the editing area of ​​the selected metadata item determined in S604. By the processes in S604 to S608 up to this point, the UI screen is in the state shown in FIG. 5. That is, the metadata item selected by the user is associated with a text block, and it is determined which text block is used to input the character string to be used in the selected metadata item.

[0027] When the above process is completed for all metadata items in the metadata item list and the user touches the "Register" button 514, a metadata registration request is transmitted to the scanned document processing server 111 in S609. This metadata registration request includes information associating edited values ​​and text blocks for all metadata items. In the scanned document processing server 111, the scanned document processing application 311 stores the above information included in the metadata registration request received from the client terminal 121 in the scanned document processing result storage unit 325. At this time, if the metadata is to be registered in an external business server 131, the metadata registration request is transferred to the business server 131 via the communication processing unit 334. After completing all necessary processes, the scanned document processing application 311 dequeues the queue from the scanned document job queue 323. This ends the process.

[0028] The above is the content of the UI screen control process in the client application 351 according to this embodiment. In this embodiment, an item color area showing the color assigned to each setting item is provided in the display field of each setting item, but providing an individual item color area is not essential. That is, the area for highlighting indicating that the setting item has been selected by the user (area excluding the cut image area and editing area in each display field) may also show the item color. In addition, when the item color is shown in the area excluding the cut image area and editing area in each display field, the highlighting indicating that the user has selected the setting item may be shown, for example, by adding a thick frame surrounding the selected display field or by changing the color density.

[0029] <Variation 1> In the flow of Fig. 6 described above, the item color area is colored at the timing of highlighting (S605) associated with the selection of a metadata item of interest by the user (detection of a touch operation on the "+" button in the cropped image area). However, the timing of coloring the item color area is not limited to this. For example, the item color area in the selected metadata item may be colored at the same timing as the coloring of the selected text block in S607.

[0030] <Variation 2> In the flow of FIG. 6 described above, the selected text block in the preview pane is colored in the same color as the item color area of ​​the selected metadata item, so that the user can understand the correspondence between the metadata item and the text block. In this case, the color used for coloring and the character color or background color in the selected text block may be similar to each other. If this happens, the visibility of the color applied to the selected text block will be reduced. Therefore, an aspect in which the color used for coloring is changed as necessary will be described as Modification 2.

[0031] FIG. 7 is a diagram for explaining the problem of this modified example, which corresponds to FIG. 5 described above. In the page image 402′ in FIG. 7, three character strings in black, “XYZ Corporation”, “1 Pumpkin Road, New York, NY”, and “(123)456-7980”, are surrounded by lines, and the inside of the lines is colored light blue as a background. Now, a text block 701 corresponding to the character string “XYZ Corporation” in the page image 402′ is associated with the metadata item “Customer Name”. And, since “Customer Name” is the first setting item among the three metadata items, blue is displayed in the item color area 511a in the display field 510a. As a result, the light blue background color in the lines surrounding the character string “XYZ Corporation” and the blue color used to color the text block 701 are similar to each other (similar colors). This makes it difficult for the user to understand that the text block 701 is associated with the metadata item “Customer Name”.

[0032] FIG. 8 is a diagram for explaining the effect of applying this modified example. As in FIG. 7, a text block 701 corresponding to the character string "XYZ Corporation" in a page image 402' is associated with a metadata item "Customer Name". However, in FIG. 8, the item color area 511a of the metadata item "Customer Name" is displayed in red, not the original blue. As a result, a color contrast is created between the light blue background color of the ruled line surrounding the character string "XYZ Corporation" and the red color of the text block 701, making it easy for the user to understand the association.

[0033] Here, details of the process for controlling the coloring of setting items and text blocks according to this modified example will be described with reference to the flowchart of Fig. 9. Note that the coloring control process shown in the flow of Fig. 9 is executed instead of S607 in the flow of Fig. 6 described above.

[0034] In S901, information on the character color of the character string included in the selected text block in which the operation event was detected in S606 and the background color of the text block is obtained from the scanned document processing result storage unit 325 via the scanned document processing application 311. Here, the explanation will be continued assuming that information on a value (R, G, B = 0, 5, 5) indicating black as the character color and a value (R, G, B = 188, 213, 254) indicating light blue as the background color is obtained.

[0035] In S902, information on the color assigned based on the item color information in S605 (i.e., the color assigned as the item color of the selected metadata item and assigned to the item color area) is acquired. In this modified example, it is assumed that the item color information is defined to assign, as default contents, "blue" to the first setting item, "green" to the second setting item, "yellow" to the third setting item, and "red" to the fourth setting item. Here, if the first of the three metadata items in the metadata pane 411, "Customer Name," is selected, the item color area will be colored in "blue," and in this case, information indicating "blue" (e.g., RGB value) is acquired.

[0036] In the next step S903, the color assigned as the item color of the selected metadata item is compared with the character color and background color in the selected text block to determine whether they are similar to each other. For example, as shown in FIG. 10(a), the RGB value of the color assigned as the item color of the selected metadata item is compared with the RGB values ​​of the character color and background color acquired in S901, and it is determined whether the difference is within a predetermined threshold. At this time, for example, for blue, it is desirable to set a threshold value such that light blue or purple, which has a similar hue, is determined to be "similar." Here, it is assumed that the blue assigned as the item color of the selected metadata item "Customer Name" is determined to be similar to the light blue background color of the selected text block. In this way, if it is determined that the color assigned as the item color of the selected metadata item is similar to the character color or background color in the selected text block, the process proceeds to S904. On the other hand, if it is determined that they are not similar, the process proceeds to S905.

[0037] In S904, a process is performed to change the color currently assigned as the item color of the selected metadata item to another different color defined in the item color information. Specifically, the correspondence between the item order and each color in the candidate color information is changed, and a different color is assigned to the selected metadata item. In the above example, the order of the selected metadata item "Customer Name" is the first. Therefore, as shown in FIG. 10(b), the color is changed to the color (in this case, "red") associated with the fourth item, which is the last in the order and is not used in the above example, and at the same time, "blue" is associated with the fourth item. By exchanging the color correspondence between the first and fourth items in this way, the color correspondence between the second and third items can be maintained, and the impact of the change can be minimized. In this way, the color that is the item color of the selected metadata item is changed, and the item color information is updated accordingly, and then the process returns to S903 to perform a similarity determination again. This process is repeated until a dissimilar determination result is obtained.

[0038] Then, in S905, the color display of the item color area of ​​the selected metadata item is updated as necessary. That is, if the color assigned as the item color of the selected metadata item in S904 is changed to another color, the item is newly colored in the changed color.

[0039] In S906, the selected text block is colored using the same color as the color being displayed in the item color area of ​​the selected metadata item. This results in the UI screen state shown in Fig. 8. When coloring is complete, the flow returns to Fig. 6, and the display process of S608 is executed.

[0040] The above is the content of the coloring control process according to Modification 2. As described in the first embodiment, coloring of the item color area in the selected metadata item may be reserved until coloring of the selected text block, and both may be colored at the same time.

[0041] <Variation 3> When the number of setting items increases, it may be difficult to select and assign colors that are dissimilar to the text color or background color from the colors defined in the item color information. Therefore, a third modification will be described below, which determines and assigns a color to be the item color for each setting item without using the item color information.

[0042] Details of the process for controlling the coloring of setting items and text blocks according to this modified example will be described with reference to the flowchart in Fig. 11. Note that the coloring control process shown in the flow in Fig. 11 is executed in place of S602 in the flow in Fig. 6 described above. In S605 of the flow in Fig. 6 when this modified example is applied, the item color area is colored using a color determined by the process described below, rather than a color determined based on the item color information.

[0043] In S1101, a list of metadata items is loaded. In other words, in this modified example, item color information is not loaded. In the following S1102, information on the character colors of character strings included in all text blocks in the page image and the background colors of the text blocks is obtained from the OCR processing results loaded in S601.

[0044] In S1103, chromatic colors that appear frequently are identified from the character colors and background colors acquired in S1102. For example, assume that there are 100 text blocks in a page image, and all of them use black as the character color. Meanwhile, assume that there are 80 text blocks that use white as the background color, 18 text blocks that use light blue, and 2 text blocks that use yellow-green. In this case, the chromatic colors are light blue and yellow-green, so light blue, which is used more frequently, is identified as the chromatic color that appears frequently. The reason why black and white, which are achromatic colors, are excluded is that they are generally used widely in forms and the like, and are not suitable for use as item colors for the purpose of identification.

[0045] In S1104, an item color for each metadata item is determined based on the frequently occurring chromatic colors identified in S1103. An example of a specific determination method is as follows.

[0046] The first method is to use a hue wheel. In this case, a frequently occurring chromatic color is used as a reference, and a color located at a distant position on the hue wheel is determined as the item color for each metadata item. For example, if a frequently occurring chromatic color is light blue and this is used as a reference, the color located at the farthest position on the hue wheel is orange, so a color to be assigned as the item color is determined from a predetermined range (e.g., a range from yellow to purple) centered on orange. In this case, if there are multiple setting items to which item colors should be assigned and multiple colors need to be determined, the colors may be selected so that the intervals between the colors on the hue wheel are approximately equal, for example, so that the multiple colors can be distinguished from each other.

[0047] The second method is a method of determining, as item colors, colors dissimilar to frequently occurring chromatic colors from among a plurality of predetermined candidate colors. For example, five colors, red, green, yellow, blue, and purple, are determined in advance as candidate colors. Then, if the frequently occurring chromatic color is, for example, light blue, the remaining four colors (red, green, yellow, and purple) are sequentially assigned as item colors excluding blue, which is similar to light blue. Note that the number of candidate colors in this case may be determined according to the format of the form expected to be processed, etc. In addition, when determining colors to be excluded, a similarity determination may be performed similar to that described in S903 of the second modified example.

[0048] The two methods described above are merely examples, and it is sufficient if colors that are easy for the user to distinguish can be assigned as item colors based on frequently occurring chromatic colors.

[0049] Fig. 12 is a diagram for explaining the effect of applying this modified example. In this example, since the background color within the ruled lines surrounding the character string "XYZ Corporation" is light blue, three colors dissimilar to light blue are determined and assigned as item colors. That is, red, orange, and yellow are assigned as item colors to the three metadata items "Customer Name," "Address," and "Total Price," respectively. This makes the colors used to color the three text blocks 1201 to 1203 stand out, allowing the user to easily understand the association between each selected metadata item and the corresponding selected text block.

[0050] The above is the content of the coloring control process according to the modified example 3. Note that a plurality of colors may be identified as frequently occurring chromatic colors. Also, frequently occurring chromatic colors may be identified not from all text blocks present in a page image, but only from text blocks present in a partial area, such as the upper part of the page image. Note that, as in the modified example 2, coloring of the item color area in the selected metadata item may be reserved until coloring of the selected text block, and both may be colored at the same time.

[0051] As described above, according to this embodiment, in the UI screen having a preview pane and a property pane, even if there are a plurality of text blocks in the page image and a plurality of setting items (sub-items) in the property pane, the user can easily grasp the correspondence between the two.

[0052] [Embodiment 2] In Embodiment 1, the UI screen control assuming a one-to-one association in which one text block is associated with one metadata item was described. Next, the UI screen control capable of handling the case where a one-to-N association in which the same text block is associated with a plurality of metadata items will be described as Embodiment 2. Note that the description of the content common to Embodiment 1 such as the system configuration will be omitted, and the following will focus on the differences.

[0053] <UI Screen Control Process> The UI screen control process according to this embodiment performed by the client application 351 of the client terminal 121 will be described with reference to the flowchart of FIG. 13. Here, according to the flow of FIG. 6 of Embodiment 1, the association with the text blocks for the three metadata items "Customer Name", "Address", and "Billing Name" is made, and the description will be made assuming the state of the UI screen shown in FIG. 14.

[0054] In the UI screen of FIG. 14, a label 1401 indicating the number of metadata items (here, "2") with which the text block is associated is attached to the upper left of a text block 501 in a page image 402. This label is displayed when the same text block is associated with multiple metadata items (i.e., when the number of items becomes two or more), but this is not limited to this, and the label may be displayed when the text block is associated with any metadata item. Now, the text block 501 is colored in the default color (here, gray) when associated with multiple metadata items. In contrast, the text block 1402 associated with only one metadata item "Address" is colored in the blue indicated by the corresponding item color area 511b.

[0055] With the UI screen in the state shown in FIG. 14, it is assumed that one of the display fields 510a to 510c of the metadata items in the property pane 411 is selected by the user (by touch operation with a finger, etc.). In S1301, an operation event such as a touch operation by the user is detected. Here, it is assumed that an operation event for the display field 510c of "Billing Name" is detected. In the following S1302, it is determined whether the text block associated with the metadata item selected by the user is also associated with other metadata items. If the determination result is "No", this process is terminated. On the other hand, if the determination result is "Yes", the process proceeds to S1303. Now, the text block 501 is associated with the metadata item "Billing Name" selected by the user, and further, the text block 501 is also associated with the metadata item "Customer Name". Therefore, the determination result in this case is "Yes", and the process proceeds to S1303.

[0056] In S1303, the color of the text block associated with the selected metadata item is colored the same as the color of the item color area of ​​the selected metadata item. Here, the color of the text block 501 changes from the default color, gray, to the yellow indicated by the item color area 511c in the display field of the metadata item "Billing Name." The UI screen shown in FIG. 15 shows a state in which the color of the text block 501 has changed from gray to yellow as a result of the metadata item "Billing Name" being selected.

[0057] In the next step S1304, a button for releasing the association between the selected metadata item and the text block is displayed. In the UI screen in Fig. 15, button 1501 with an "x" mark that appears in the upper right corner of text block 501 indicates the release button. In the next step S1305, it is determined whether or not the release button has been touched or otherwise operated. If an operation event such as a touch operation on the release button is detected, the process proceeds to S1306, and if not, the process ends.

[0058] In S1306, the association between the selected metadata item and the text block is released, and the label indicating the number of currently associated metadata items is updated. Specifically, as a result of releasing the association with the selected metadata item, the number of metadata items indicated by the label is subtracted by "1." In the example of FIG. 15, the value of label 1401 changes from the current "2" to "1." In addition, the display field corresponding to the selected metadata item is initialized to a blank state. Currently, the selected metadata item is "Billing Name," so the corresponding display field 510c is initialized.

[0059] The above is the content of the UI screen control process in the client application 351 according to this embodiment.

[0060] As described above, according to this embodiment, even when the same text block is associated with a plurality of setting items (child items), the user can accurately and easily grasp the correspondence between the child items and the text block.

[0061] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

Claims

1. An image processing system for digitizing a document, comprising: a display means for displaying a user interface screen for setting properties for a scanned image obtained by scanning the document; a control means for controlling a display on the user interface screen; Equipped with the user interface screen has at least a preview pane that is an area for displaying a preview of the scanned image, and a property pane that is an area for inputting setting values ​​for each setting item of the property; The control means For a setting item displayed in the property pane, coloring at least a part of a display field corresponding to the setting item using a color assigned to the setting item; When a user selects any of the text blocks of the scanned image previewed in the preview pane while any of the setting items is selected in the property pane, the text block is colored using the color assigned to the selected setting item. An image processing system comprising:

2. The image processing system according to claim 1, characterized in that the control means performs the coloring of at least a portion of the display field corresponding to the setting item displayed in the property pane at the timing when an operation event for selecting the setting item displayed in the property pane is detected.

3. The image processing system according to claim 1, characterized in that the control means performs the coloring of at least a portion of the display field corresponding to the setting item displayed in the property pane at the same timing as the coloring of the text block.

4. 4. The image processing system according to claim 1, wherein at least a portion of the display field is an item color area provided within the display field.

5. 5. The image processing system according to claim 1, wherein the control means assigns a color to the setting item selected by the user that is not similar to the character color or background color of the text block included in the scanned image.

6. 5. The image processing system according to claim 1, wherein the control means assigns a specific color to a setting item selected by a user based on item color information that defines a plurality of assignable colors.

7. In the item color information, colors are associated with the plurality of setting items according to their order, the control means refers to the item color information and assigns a color corresponding to the order of the setting items selected by the user; 7. The image processing system according to claim 6.

8. The image processing system of claim 6, wherein the control means changes the color assigned in accordance with the item color information to a dissimilar color if the color is similar to the character color or background color of the text block selected by the user, and then performs the coloring.

9. 5. The image processing system according to claim 1, wherein the control means identifies a frequently occurring chromatic color among the character colors or background colors of a text block included in the scanned image, and assigns a color that is not similar to the frequently occurring chromatic color to the setting item selected by the user.

10. 10. The image processing system according to claim 9, wherein the control means assigns a color located at a position distant from the chromatic color on a color wheel to the setting item selected by the user.

11. 10. The image processing system according to claim 9, wherein the control means assigns a color that is not similar to the chromatic color from among a plurality of predetermined candidate colors to the setting item selected by the user.

12. The image processing system of any one of claims 1 to 4, characterized in that when a text block of the scanned image previewed in the preview pane is associated with a setting item displayed in the properties pane, the control means displays a label indicating the number of the setting item for the text block.

13. The image processing system according to claim 12, wherein the control means displays the label when a text block of the scanned image previewed in the preview pane is associated with a plurality of setting items displayed in the property pane.

14. The control means If a text block associated with a setting item selected by the user among any of the setting items displayed in the property pane is also associated with another setting item, a button for canceling the association is displayed; When an operation event of a selection operation on the release button is detected, the number of setting items indicated by the label is subtracted.

14. The image processing system according to claim 13.

15. A method for controlling an image processing system for digitizing a document, comprising: The image processing system includes: a display means for displaying a user interface screen for setting properties for a scanned image obtained by scanning the document; a control means for controlling a display on the user interface screen; Equipped with the user interface screen has at least a preview pane that is an area for displaying a preview of the scanned image, and a property pane that is an area for inputting setting values ​​for each setting item of the property; The control means For a setting item displayed in the property pane, coloring at least a part of a display field corresponding to the setting item using a color assigned to the setting item; When a user selects any of the text blocks of the scanned image previewed in the preview pane while any of the setting items is selected in the property pane, the text block is colored using the color assigned to the selected setting item. The control method is characterized by controlling as follows.

16. A program for causing a computer to function as the means of the image processing system according to any one of claims 1 to 14.

Citation Information

Patent Citations

  • Document attribute input device and method

    JP2005322082A

  • Content display device and method

    JP2006113976A

  • Program and apparatus for supporting input of information

    JP2011248592A

  • Information processing system, information processing program, and information processing method

    JP2013196479A

  • Image processing program, image processing method, and image processing apparatus

    JP2016126356A