CLIENT TERMINAL, CONTROL METHOD AND PROGRAM FOR CLIENT TERMINAL
The client terminal dynamically adjusts the display of character strings based on user interactions, addressing the challenge of maintaining accurate string presentation during image resizing or movement within the displayed area.
Patent Information
- Application Number
- JP2022192636
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-12-01
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-12-01
AI Technical Summary
Existing systems fail to appropriately display a list of character strings obtained from image data in response to changes in the displayed area, such as enlargement or reduction, within one page of the image data.
A client terminal that performs character recognition on image data and controls the display area based on user instructions, ensuring the list of character strings is updated accordingly as the image data is resized or moved within the page.
Enables appropriate display of character strings in response to changes in the image data's display area, providing a mechanism for efficient information presentation.
Smart Images

Figure 0007581306000006 
Figure 0007581306000007 
Figure 0007581306000008
Abstract
Description
[Technical field]
[0001] The present invention relates to a user interface that displays a list of character strings contained in image data. [Background technology]
[0002] In recent years, as part of the digital transformation trend, there has been an increasing need to digitize paper documents and use IT systems to streamline subsequent document processing operations. First, the paper document is scanned or photographed to create an electronic file of the image. There are various types of documents, such as purchase orders, invoices, and application forms, and these various document processing operations can be made more efficient by extracting and using character strings written on the document from the image file using optical character recognition (OCR). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2020-086717 A Summary of the Invention [Problem to be solved by the invention]
[0004] Patent Document 1 discloses a technology that displays a list of character strings obtained by performing character recognition processing on image data, and uses a character string selected from the list as the destination folder name for a file that contains the image data. The list is displayed on each page together with the corresponding image data.
[0005] However, there is no disclosure of changing the displayed area within one page of image data. For example, it is conceivable to change the displayed area by enlarging or reducing the image data or moving within the image data. Patent Document 1 is unable to appropriately display a list of character strings in accordance with the change in the displayed area.
[0006] An object of the present invention is to appropriately display a list of character strings acquired by performing character recognition processing on image data in accordance with changes in the area in which the image data is displayed. [Means for solving the problem]
[0007] In order to achieve the above object, the client terminal of the present invention is read Obtained R Image data and, The image data do Character Recognition Processing and acquiring one or more character strings recognized from the image data by Means and ,before At least one of the image data Department, And the above 1 or more String Consists of at least a portion of A display control means for controlling display of the list, the display control means is capable of controlling, based on an instruction from a user, to change a displayed area of the image data within one page of the image data; before Note The strike is ,before Note Displayed When the area changes, Of the image data The Displayed The area changes so that the recognized character string is displayed. Effect of the Invention
[0008] The client terminal of the present invention can provide a mechanism for appropriately displaying a list of character strings obtained by performing character recognition processing on image data in accordance with changes in the area in which the image data is displayed, based on instructions such as enlargement or reduction. [Brief description of the drawings]
[0009] [Figure 1] 1 is a diagram showing a system configuration and a network configuration for implementing the present invention; [Diagram 2] 1 is a hardware configuration diagram relating to an information processing function of an information processing device according to the present invention; [Diagram 3] FIG. 2 is a diagram illustrating the software and hardware configuration of the system of the present invention. [Figure 4]2 is a flowchart showing a process according to the present invention. [Diagram 5] FIG. 2 is a diagram illustrating an original image in the present invention. [Figure 6] FIG. 2 is a diagram showing an example of a UI according to the present invention. [Figure 7] FIG. 2 is a diagram showing an example of a UI according to the present invention. [Figure 8] FIG. 2 is a diagram showing an example of a UI according to the present invention. [Figure 9] FIG. 2 is a diagram showing an example of a UI according to the present invention. [Figure 10] FIG. 2 is a diagram showing an example of a UI according to the present invention. [Figure 11] FIG. 2 is a diagram showing an example of a UI according to the present invention. [Figure 12] FIG. 2 is a diagram illustrating an original image in the present invention. [Figure 13] FIG. 2 is a diagram showing an example of a UI according to the present invention. [Figure 14] 4 is a flowchart showing a UI control process according to the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] <First embodiment> Hereinafter, the embodiments of the present invention will be described with reference to the drawings. Note that the following embodiments do not limit the invention according to the claims, and all of the combinations of features described in the embodiments are not necessarily essential to the solution of the invention.
[0011] FIG. 1 shows an example of a system configuration and a network configuration for carrying out the present invention. Reference numeral 101 denotes a network such as the Internet or an intranet. Reference numeral 111 denotes a client terminal. Reference numeral 121 denotes a scanner terminal. Reference numeral 131 denotes an application server. The client terminal 111 may be of any type or form, such as a personal computer, a laptop computer, a tablet computer, or a smartphone. The scanner terminal 121 may be of any type or form, such as an office multifunction device, an inkjet multifunction device, or a dedicated scan terminal. The application server 131 may be provided in any form, such as an on-premise server, a virtual machine server hosted in the cloud, or SaaS (Software as a Service).
[0012] FIG. 2 shows a module configuration diagram of the information processing function of the client terminal 111, the scanner terminal 121, and the application server 131. 202 is a network interface, which is connected to a network such as a LAN and communicates with other computers and network devices. The communication method may be either wired or wireless. 204 is a ROM in which embedded programs and data are recorded. 205 is a RAM for temporary memory area. 206 is a secondary storage device such as a HDD or flash memory. 203 is a CPU, which executes programs read from the ROM 204, the RAM 205, the secondary storage device 206, etc. 207 is a GPU. The GPU 207 is a processor specialized for image processing, and is responsible for image processing, drawing output images to a display, and large-scale parallel arithmetic processing such as machine learning. The GPU 207 is not necessarily provided. 201 is a user interface, which inputs and outputs information and signals using a display, a keyboard, a mouse, buttons, a touch panel, etc. A computer that does not have these hardware components can be connected and operated from another computer using a remote desktop, a remote shell, etc. Each unit is connected via an input / output interface 208.
[0013] 3 is a diagram showing the software and hardware configuration of this system. The software installed on each piece of hardware is executed by the respective CPUs 203, and as shown by the network connection arrows, the configuration allows mutual communication. In addition, in hardware equipped with a GPU 207, it is also possible to configure the GPU 207 to execute image processing.
[0014] 301 is a client application executed by the client terminal 111. The client application 301 often takes the form of a native application that is installed and executed in the OS of the client terminal 111. This is because it is possible to create an application that can access all functions such as the camera and files provided by the OS. However, if the OS provides APIs that can be used from a browser for those necessary functions, a web application consisting of HTML, JavaScript, etc. may be executed in the browser. In that case, the client application 301 becomes a browser. In this embodiment, the following description will be given on the assumption that the client application 301 is used in the client terminal 111 such as a so-called general-purpose smartphone or tablet PC. However, if the scanner terminal 121 has a configuration such as an OS and a touch panel UI required to execute the client application 301, the client application 301 may be executed in the scanner terminal 121.
[0015] 311 is a file storage for saving and managing files. In this way, the client terminal 111 may be configured to include a storage server. The file storage 311 is made up of a file storage unit 312 and a metadata management unit 313. The file storage unit 312 stores and manages binary data of the files themselves. The metadata management unit 313 stores and manages metadata for each file. Typical file metadata include the creation date and time, size, and the name of the user who created the file, but any metadata may be handled.
[0016] Reference numeral 321 denotes an image processing unit that executes image processing including character recognition processing such as OCR (Optical Character Recognition). The image processing unit 321 is composed of a temporary file storage unit 322 and an image processing execution unit 323. The temporary file storage unit 322 stores files that are the subject of image processing by the image processing execution unit 323 (described later) and the results of the execution. The image processing execution unit 323 reads out files that are the subject of processing from the temporary file storage unit 322, executes character recognition processing by OCR, etc., and saves the execution result file in the temporary file storage unit 322. Reference numeral 324 denotes a camera. The client application 301 may be configured to acquire images captured by the camera 324.
[0017] Reference numeral 331 denotes a scanner equipped in the scanner terminal 121. The scanner 331 scans the surface of a document using an optical sensor to read reflected light and color. Reference numeral 332 denotes an image forming unit, which forms a page image of the document from the optical measurement values read by the scanner 331. Reference numeral 333 denotes a communication interface. The communication interface 333 controls communication with external devices using a wired or wireless communication protocol such as LAN, USB, WiFi, or Bluetooth. The client application 301 can connect to the communication interface 333 to start scanning a document using the scanner 331 and obtain an image generated by the image forming unit 332.
[0018] The file storage 311 and the image processing unit 321 may be arranged in the application server 131 as a file storage 341 and an image processing unit 351. Since 341 to 353 are similar to the above-mentioned 311 to 323, the description thereof will be omitted.
[0019] Fig. 4 is a flowchart explaining the overall processing flow. The processing performed by the client application 301 shown in Fig. 4 is executed as arithmetic processing by the CPU 203 after a program recorded in the secondary storage device 206 of the client terminal 111 is loaded into the RAM 205. As described above, image processing is executed by the GPU 207. Note that, in this embodiment, a configuration is described in which the client application 301 of the client terminal 111 performs the subsequent processing, but this is not limiting. As described above, the scanner terminal may be configured to perform the subsequent processing.
[0020] The client application 301 acquires an original image from one of the camera 324, the scanner terminal 333, and the file storages 311 and 341 (S401). In the present invention, image data is acquired based on one original. Specifically, the client application 301 detects a press of a camera selection button 612 on a screen 610 shown in FIG. 6 described later. When the client application 301 detects the press, it displays a screen 620 on the user interface 201. The user photographs the original using the camera 324 provided in the client terminal 111. When the client application 301 detects a press of a capture button 623 on the screen 620, it acquires the captured image data. In addition, when acquiring image data from the scanner terminal 333, the user presses the scanner selection button 611. When the client application 301 detects the press, it displays a screen 630 on the user interface 201. The user performs settings related to scanning on the screen 630. When the client application 301 detects that the user has pressed the scan button 635, it transmits information indicating a scan execution instruction to the scanner terminal 121. The scanner terminal 121 that has received the information executes a scan using the scanner 331. Image data obtained by the scan is transmitted to the client application 301 via the communication interface 333, and the client application 301 acquires the image data. When acquiring image data from the file storage 311, the user presses the local file button 613 on the screen 610. When the client application 301 detects the pressing, it displays a screen 640 on the user interface 201. Then, it accepts the user's selection of a file from among the files displayed on the screen 640. The client application 301 acquires the file selected by the user from the file storage 311. The process of acquiring image data from the file storage 341 is the same as that of acquiring image data from the file storage 311, except that the source of the image data is the file storage 341 of the application server.
[0021] The client application 301 inputs an image to the image processing unit 321, and the image processing unit 321 stores the image in the temporary file storage unit 322. The image processing execution unit 323 reads the image file in the temporary file storage unit 322, executes character recognition processing (OCR), and outputs the image processing result to the temporary file storage unit 322 (S402). The client application 301 acquires the image processing result from the image processing unit 321 (S403). Even when the image processing unit 351 of the application server 131 is used, the procedure is the same except that the access destination is 351. When the client terminal 111 does not have a CPU or GPU with sufficient performance, this configuration can be used to offload image processing to the application server 131. The client application 301 displays the image and the OCR character string of the image processing result on the UI, and accepts the selection of the OCR character string (S404). The client application 301 uses the selected OCR string for at least one of a folder name, a file name, and metadata and saves the image file in the destination file storage 311 or 341 (S405).
[0022] Hereinafter, the processes in S403, S404, and S405 will be described in detail with reference to FIGS.
[0023] FIG. 5 shows an entire page image 500 of the document image acquired in S401. In this embodiment, the following description will be given using an image of a PURCHASE ORDER document in English. The document may be in any language or type. The document image 500 contains various information (date, number, company name, address, telephone number, product name, price, etc.) written on the purchase order as character strings. 501 is a diagram for explaining an orthogonal coordinate system showing the relationship between the document image 500 and the OCR character string area 502 obtained by image processing. In S402, image processing is performed using the document image 500 as input, and the image processing result is acquired in S403. The following information is obtained as the image processing result: the start coordinates, width, and height of the OCR character string "PURCHASE ORDER" and the OCR character string area 502. In addition, additional analysis information such as whether it is a so-called Key-Value type character string pair or whether it is a Key and Value character string can be included in the image processing result from the position and size information between the acquired OCR character string and the OCR character string area. For example, the OCR result string can be parsed and the key and value can be detected based on the detection of a character that is often used after the key, such as a colon (:). Alternatively, the specification of the key string can be accepted from the user in advance, and when the string is detected, the corresponding value string can be specified. In the original image 500, examples of key-value type string pairs include the pair "PO Number:" and "2022-P001-07525" and the pair "Company Name:" and "XYZ Corporation". Table 1 is a table format of the OCR string array, which is the image processing result obtained from the original image 500. Table 1 also includes the position and size information of the OCR string area described above, but is omitted from the table to avoid complication. The following explanation will be continued on the assumption that the position and size information of the OCR string area is always present for each string array.
[0024] [Table 1]
[0025] 6, 7, 8, 9, 10, 11, and 13 are diagrams illustrating the UI of the client application 301. These UIs are displayed, for example, on the user interface 201 of the client terminal 111. For example, they are displayed on a touch panel of a smartphone or the like. Fig. 12 is a diagram illustrating an area within a document image to assist in the explanation of Fig. 11. Fig. 14 is a flowchart illustrating a UI processing flow.
[0026] Hereinafter, the UI control process of the client application 301 will be explained in order using these figures.
[0027] Reference numeral 610 denotes an image reading screen. This screen allows the user to select the source of the document image from the scanner terminal 121, the camera 324, or the file storage 311 or 341. When the client application 301 detects that the user has pressed the scanner selection button 611, it acquires image data from the scanner terminal 121 as the source of the document image. When the client application 301 detects that the user has pressed the camera selection button 612, it acquires image data from the camera 324 as the source of the document image. When the client application 301 detects that the user has pressed the local file button 613, it acquires image data from the file storage 311 as the source of the document image. When the client application 301 detects that the user has pressed the application server button 614, it acquires image data from the file storage 341 as the source of the document image.
[0028] When the camera 324 is set as the acquisition source of the document image, the client application 301 displays a camera shooting screen 620 on the user interface 201. 621 is a shooting area display section of the camera 324, and the user presses a shooting button 623 to shoot a document 622. When the client application 301 detects the pressing of the back button 624, it displays the previous screen (screen 610 in this case). The operations of the back buttons are all similar, so a description thereof will be omitted below.
[0029] When the scanner terminal 121 is set as the acquisition source of the original image, the client application 301 displays a scan setting screen 630. Reference numerals 631, 632, and 633 are drop-down controls for selecting a single-sided or double-sided scan setting, a color mode (color or black and white), and a resolution, respectively. When the client application 301 detects that the user has pressed a scan button 635, it scans the original placed in the scanner terminal 121 via the communication interface 333 and acquires the generated image data.
[0030] When the file storage 311 or 341 is set as the source of the original image, the client application 301 displays an image file read screen 640. 641 is a control that displays the folder path in the file storage. 642 is a list view control for folder navigation. The user presses 642 to move to the target folder. To return to the upper folder, the user presses control 641. 643 is a list view control for selecting a file in the file storage. After accepting the selection of the image file to be acquired as the original image from the user in 643, when detecting the pressing of the Next button 644, the client application determines that the file to be read is the file indicated by 643, and displays the next screen (here, screen 710). The operations of the Next buttons are all similar, so a description thereof will be omitted hereafter. Up to this point corresponds to the processing of S401.
[0031] When the client application 301 acquires an original image from the scanner terminal 121, the camera 324, or the file storage 311 or 341, it displays an image confirmation screen 710. 711 is a preview of the original image. When the client application 301 detects that the user has pressed the Next button, it inputs the original image and executes image processing in the image processing unit 321 or 351. The above corresponds to the processing in S402.
[0032] The client application 301 displays a destination folder designation screen 720 while waiting for image processing execution to finish in the background. 721 displays the folder path of either the destination file storage 311 or 341. 722 is a list view control that displays a list of subfolders below the current folder. When the client application 301 detects that 722 is pressed, it displays screen 730 and displays the current folder in 731. After the user navigates through the screens to the desired destination folder, he or she presses the Next button. The client application 301 detects the pressing of the Next button and determines the destination folder.
[0033] The subsequent processing will be described with reference to the flowchart in Fig. 14. The processing shown in Fig. 14 is a detailed description of S403 and S404. In the subsequent processing, S403 and S404, i.e., S1401 to S1406, are repeated four times. The first time is for determining the name of the subfolder in which the file containing image data is to be saved, the second time is for determining the file name, the third time is for determining the metadata to be set in the file, and the fourth time is for determining the tags to be set in the file.
[0034] First, the process of determining a subfolder name will be described. The process of determining a subfolder name is started when image data is acquired and image processing is executed. Client application 301 acquires an original image and an OCR character string array shown in Table 1 as image processing results from image processing unit 321 or 351 (S1401). The process up to this point corresponds to S403. Client application 301 creates a list view array shown in Table 2 for the next list view to be used, and imports the OCR character string array in Table 1 (S1402). Everything other than the list view array number in Table 2 is the OCR character string array data itself in Table 1. Rearranging the list view array numbers changes which OCR character string data is displayed in the list view and in what order.
[0035] [Table 2]
[0036] When the client application 301 detects pressing of the Next button on screen 730 and identifies the destination folder, it displays a subfolder designation screen 810. Note that the destination folder may be identified before S1402. Also, in the process of determining the subfolder name, subflow 1 and subflow 2 are not described, and a configuration in which no processing is performed is described, but this is not limited to this. Subflow 1 and subflow 2 may also be performed in the process of determining the subfolder name. Detailed processing of subflow 1 and subflow 2 will be described in the process of determining metadata and the process of determining tags.
[0037] Reference numeral 811 denotes a preview area for the original image. In the preview area 811, it is possible to move the original image 500 up and down and left and right within the page by swiping. Only about the top third of the original image 500 can be displayed in 811, but it is possible to move to the bottom of the page and change the display area by scrolling. It is also possible to change the display area by zooming in and out of the image by pinching. Reference numeral 812 denotes a list view control for enumerating and selecting OCR character strings extracted from the image. Reference numeral 813 denotes a scroll indicator that is displayed when there are too many items in the list view control 812 to display them all.
[0038] The client application 301 displays the acquired document image in the preview area 811 (S1403). Then, the OCR character string "PURCHASE ORDER" displayed at the top of the preview area 811 is set to the top of the list view array (S1404). That is, in table 2, the list view array number 1 is set to be the top of the display. Then, the OCR character string of the list view array number at the top of the display is set to the top of the list view control 812, and the list view array of table 2 is displayed in the list view control 812 (S1405). In this embodiment, the list view control 812 can be scrolled up and down by a swipe operation, and items are displayed in the order of the list view array number of table 2. Here, in a scrollable list, there is a limit to the character strings that can be displayed on the screen. In this embodiment, the list is scrolled and displayed so that the character string set at the top of the display is displayed on the screen. 820 is the same subfolder designation screen, but shows a screen resulting from scrolling the document image 500 downward by swiping the preview area 811. The client application 301 detects that at least one of the operations of moving and zooming has been performed on the preview area as shown in 811 to 821 (S1406). When detected, the client application 301 updates and displays a preview of the document image 500 after moving and zooming in the preview area (S1403). In response to the OCR character string "Phone:" becoming the top in the preview area 821, the client application 301 sets the display top of the list view array of table 2 to list view array number 10 (S1404). The client application 301 updates the display of the list view control 822 so that the "Phone:" character string of list view array number 10 comes to the top (S1405). Specifically, the client application 301 scrolls the list view control 822 down to a predetermined position and controls so that the OCR character string "Phone:" is displayed at the top in the display area of the list view control 822.Thereafter, when movement or zooming of preview area 811 is detected by repeating this process, the display of preview area 821 is updated, and in conjunction with this, the display top of list view control 822 is changed and the display is updated. Note that in this embodiment, the display of the list is controlled to match the display range of the image displayed in preview area 821 by scrolling list view control 822. Alternatively, the character strings included in the list may be only character strings included in a partial area of the image data displayed in preview area 821. In this way, the display of the list may be controlled so that the OCR character string displayed at the top of preview area 821 is also displayed at the top of the display area of the list view control.
[0039] In this embodiment, the use case is to select and specify "XCamera Company" with the character string array number 13 as the subfolder name. When the display area of the preview area 821 and the top display of the list view 822 are linked and the target "XCamera Company" character string is displayed in the list view 822, the user taps 822 to select it. When the client application 301 accepts the selection of a character string in the list by the user, it highlights 822 to indicate a selected state. In addition, it highlights the "XCamera Company" character string area 823 in the preview area 821 in response to the selected character string in 822. This allows the user to confirm which character string area in the document image has been selected. When the client application 301 detects the pressing of the Next button, it transitions to the next screen and ends this flow.
[0040] Next, the second processing of S1401 to S1406 will be described. This is processing for determining the file name of the image data. S1401 and S1402 are the same as the processing for determining the subfolder name. Note that S1401 and S1402 may be omitted in the second and subsequent processing. In that case, the document image, OCR character string array, and list view array acquired in S1401 and S1402 for the first time are also used in the subsequent processing. Also, in the second processing, subflow 1 is omitted. In S1403, the client application 301 displays a screen 910. 910 is a file name designation screen. Here, a use case in which multiple OCR character strings are selected from within a document image will be described. The preview area 911 and the list view 912 are the same as 811 and 812. The user long-presses and taps the OCR character string 922 in the list view to select it. When the client application 301 detects the long-press tap, it displays a check mark and highlights the list view control 922 to indicate that it is in a selected state. Also, an OCR character string area 923 corresponding to the OCR character string 922 is highlighted in the preview area 911. The user further presses and taps the OCR character string 931 in the list view to select it. When the client application 301 detects the long press tap on 931, it displays a check mark and highlights the list view control 931 to indicate that it is in a selected state. Also, an OCR character string area 932 corresponding to the OCR character string 931 is highlighted in the preview area 911. 933 is a control for rearranging the order when there are multiple selected OCR character strings. The user presses and holds 933 to drag the selected OCR character string to a desired position. As a result, the result after rearranging the selected OCR character strings 922 and 931 becomes 941. When the client application 301 detects the pressing of the Next button, it transitions to the next screen and ends this flow.
[0041] Next, the third processing of S1401 to S1406 will be described. This is processing for determining metadata to be set in a file containing image data. S1401 and S1402 are similar to the processing for determining a subfolder name. Here, a case where the processing of subflow 1 is also performed will be described.
[0042] In S1411, the client application 301 judges whether or not to display the Key string at the bottom of the list. Specifically, it is confirmed whether or not the toggle switch 1021 shown on the screen 1020 is ON. If it is not ON, that is, if it is OFF, the process proceeds to S1403, and the client application 301 displays the metadata specification screen 1010. The preview area 1011 and the list view 1012 are the same as 811 and 812. 1013 is a setting button of the client application 301. When the setting button 1013 is pressed, the setting screen 1020 opens. 1021 is a toggle switch for whether or not to display the Key string of the Key-Value type string pair at the bottom of the list. 1022 is a toggle switch for whether or not to display the Key string at the bottom of the list when only a part of the string is displayed in the preview. 1022 will be described later in another figure. The screen 1010 shows a case where the toggle switch 1021 is OFF, and the list view control 1012 is displayed in the display order as shown in the list view array of Table 2.
[0043] When the toggle switch 1021 is ON (S1411), the client application 301 rearranges the list view array so that the OCR character strings whose key-value type is Key are placed at the bottom, as shown in Table 3 (S1412).
[0044] [Table 3]
[0045] Then, the client application 301 displays the list view in the order sorted in table 3, as shown in list view control 1031. This makes it possible to move character strings that are unlikely to be used to input folder names, file names, metadata, etc., such as "PO Number:" and "Company Name" from among the character strings present in the original image 500 to the bottom of the list. Since there is a limit to the number of items that can be displayed in the list view control 1031, it is effective to remove character string candidates that are unlikely to be used from the items to be displayed.
[0046] Subflow 1 may also be performed after displaying the document image preview in S1403. Specifically, when detecting pressing of the Set button 1013 while the screen 1010 is displayed, the client application 301 displays the screen 1020. When detecting that the toggle switch 1021 has been switched from OFF to ON and the OK button has been pressed on the screen 1020, the client application 301 moves the Key character string to a lower position in the list view array and redisplays the updated list view. The client application 301 also detects that the toggle switch 1021 has been switched from ON to OFF and the OK button has been pressed. Then, the client application 301 updates the display of the list view according to the original list view array before the Key character string was moved to a lower position in the list view array.
[0047] Here, a use case will be described in which the OCR character string "XYZ Corporation" is selected and input as the value of the metadata "Company Name." When the toggle switch 1021 is OFF, the target OCR character string "XYZ Corporation" is not displayed in the displayable range of the list view control 1012. When the toggle switch 1021 is ON, the key character string is moved lower in the list as shown in Table 3, and the target OCR character string "XYZ Corporation" can be displayed in the list view control 1031. When the user taps the target OCR character string "XYZ Corporation" in the list view control 1041 to select it, the client application 301 highlights the OCR character string. Also, the OCR character string area 1042 is highlighted in the preview area. When the client application 301 detects the pressing of the Next button, it transitions to the next screen and ends this flow.
[0048] Next, the fourth processing of S1401 to S1406 will be described. This is processing for determining a tag to be set to a file including image data. S1401 and S1402 are the same as the processing for determining a subfolder name. Subflow 1 will be omitted. Here, a flow for performing subflow 2 instead of S1404 will be described. In S1403, the client application 301 displays a screen 1110. 1110 is a tag specification screen. A tag is a type of metadata, and generally does not require a strict type definition like Key-Value type metadata, and one or more additional data can be freely added. It is also called a label. The preview area 1111 and the list view 1112 are the same as 811 and 812. As shown in FIG. 12, the preview area 1111 is in a state where a part of the original image 500, an area 1201, is enlarged and displayed. At this time, if a list view array is configured by focusing on the character string included in the area 1202 including 1201, it will be as shown in Table 4. A column indicating the preview display state is added to the right end.
[0049] [Table 4]
[0050] The preview display state indicates whether the OCR character string is fully displayed, partially displayed, or hidden in the preview area 1111. When the toggle switch 1022 on the setting screen 1020 is OFF, the characters are displayed in the list view control 1112 in the order of the list view arrangement in Table 4.
[0051] In S1421, the client application 301 determines whether the toggle switch 1022 is set to ON. Specifically, first, when the client application 301 detects pressing of the setting button 1113 on the screen 1110 displayed in S1403, the client application 301 displays the screen 1020. If the toggle switch 1022 is set to ON on the screen 1020 and pressing of the OK button is detected, the client application 301 determines that the toggle switch 1022 is set to ON. If the toggle switch 1022 is set to OFF and pressing of the OK button is detected, the client application 301 determines that the toggle switch 1022 is not set to ON. If the toggle switch 1022 is set to ON (S1421), the client application 301 enumerates the OCR character strings in the page area 1202 that is the same height as the preview area, and classifies the preview display state into full display, partial display, or non-display (S1422). The client application 301 sorts the list view array in the preview display state column in the order of full display, partial display, and non-display, as shown in Table 5 (S1423).
[0052] [Table 5]
[0053] The client application 301 sets the smallest list view array number of the OCR strings listed in S1422 after sorting in S1423 to the top of the list view array (S1424). In this embodiment, this is the OCR string "Description" of list view array number 9 in Table 5. The list view array sorted in the preview display state of Table 5 is displayed in the list view control 1121 (S1405). This process allows the strings that are fully or partially displayed among the OCR strings in the area 1202 in the original image 500 to be displayed in the list view control 1121 as higher-ranked candidates. Note that the above-mentioned process may be performed between S1402 and S1403 as in subflow 1. That is, the client application 301 may check the setting of the toggle switch 1022 before displaying the screen 1110 in S1403, and display the list view based on the list view array according to the setting.
[0054] The user long-presses and taps on OCR character strings 1131, 1132 to be used for tags from the list view control 1121 to select them. When the client application 301 detects the long-press tap, it displays a check mark and highlights the list view controls 1131, 1132 to indicate that they have been selected. It also highlights the selected OCR character string areas 1133, 1134 in the preview area 1111. When the client application 301 detects the pressing of the Next button, it transitions to the next screen and ends this flow. The next screen is screen 1300, which will be described later.
[0055] In this embodiment, an example of controlling the display of the list to be switched when the user moves or zooms in the image in the preview area 821 is shown. For example, the screen 820 shown in FIG. 8 is an example in which the display area is moved downward from the preview area 811 of the screen 810. Also, the screen 1110 shown in FIG. 11 is an example in which zooming in is performed to enlarge and display a part of the area in the preview area according to the user's instruction. Not limited to this, the control of switching the display of the list may be performed even when the user issues an instruction to zoom out in the preview area and reduces the display of the image data. For example, as in the screen 1110 shown in FIG. 11, the user issues an instruction to zoom out in the preview area 1111 from a state in which the OCR character string "Part No." is displayed at the top of the display area of the list view control 1112. In this way, when the image is reduced so that the entire document image 500 is displayed, the client application 301 sets the OCR character string "PURCHASE ORDER" at the top of the list view array. Then, control is performed to display the list in the same manner as the list view control 812.
[0056] Reference numeral 1300 denotes a screen for saving to a destination. Reference numeral 1301 denotes a text control showing the destination name of the destination file storage 311 or 341. Reference numeral 1302 denotes a text control showing the folder path specified and configured in FIGS. 7 and 8. Reference numeral 1303 denotes a text control showing the file name specified and configured in FIG. 9. Reference numeral 1304 denotes a text control showing the file name specified and configured in FIG. 10. Reference numeral 1305 denotes a text control showing one or more tags specified and configured in FIG. 11. When a save button 1306 is pressed, the client application 301 saves the original image in the destination storage 311 or 314 with the specified folder path and file name. Also, if metadata (including tags, etc.) is specified, the metadata is saved in the destination storage 311 or 314 in addition to the original image. This corresponds to S405 in the overall flow.
[0057] In this embodiment, a UI control method for selecting a desired OCR character string from an image in a client application has been described. Even if the client terminal does not have a sufficiently large display and is constrained by tap, swipe, and pinch operations specific to touch panel UIs, it becomes possible to easily and quickly select a desired OCR character string in an image. This solves the problem described at the beginning, which is to increase the efficiency of the task of selecting a desired OCR character string in an image in a touch panel UI for selecting an OCR character string in an image.
[0058] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of each of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC or FPGA) for implementing one or more of the functions. [Explanation of symbols]
[0059] 111 Client terminal 301 Client Application
Claims
1. an acquiring means for acquiring image data obtained by scanning a document and one or more character strings recognized from the image data by character recognition processing of the image data; a display control means for controlling display of a list composed of at least a part of the image data and at least a part of the one or more character strings; having the display control means is capable of controlling, based on an instruction from a user, to change a displayed area of the image data within one page of the image data; The client terminal according to claim 1, wherein the list changes when the displayed area changes so that a character string recognized in the displayed area of the image data is displayed.
2. The client terminal according to claim 1, characterized in that the user instruction is at least one of an instruction to enlarge the display of at least a portion of the image data and an instruction to reduce the display of at least a portion of the image data, and the displayed area of the image data changes based on the instruction.
3. The list is displayed in a scrollable format; 2. The client terminal according to claim 1, wherein the display control means scrolls and displays the list to a predetermined position so that the recognized character string is displayed in the displayed area.
4. 2. The client terminal according to claim 1, wherein the display control means causes only character strings included in the displayed area to be included in the list so that the recognized character string is displayed in the displayed area.
5. the display control means controls display of the list such that, among the character strings recognized in the displayed area, a character string only partially included in the area is arranged in the list below a character string whose entirety is included in the area; 2. The client terminal according to claim 1, wherein when there are a plurality of character strings, only a portion of which is included in the region, a character string that is higher in the image data is arranged higher in the list.
6. 2. The client terminal according to claim 1, wherein the display control means does not include in the list, among the character strings recognized in the displayed area, a character string of which only a part is included in the displayed area.
7. The character recognition process further includes recognizing a combination of a character string that is a Key and a character string that is a Value that corresponds to the Key; the display control means controls display of the list so that a character string that is the Key among character strings recognized in the displayed area is placed at the bottom of the list; 2. The client terminal according to claim 1, wherein when there are a plurality of character strings that serve as the key, a character string that is located higher in the image data is arranged higher in the list.
8. a receiving means for receiving a selection of a displayed character string included in the list; 2. The client terminal according to claim 1, further comprising a setting unit for setting information about the image data to the image data.
9. 9. The client terminal according to claim 8, wherein the display control means controls so that the character string selected by the accepting means is extracted from the list and displayed.
10. 9. The client terminal according to claim 8, wherein the setting means sets the information using a character string selected by the accepting means.
11. 9. The client terminal according to claim 8, wherein the information is at least one of a file name of a file including the image data, a folder name in which the file is saved, and metadata set to the file.
12. The client terminal includes a camera; 2. The client terminal according to claim 1, wherein said acquiring means acquires said image data by photographing an original with said camera.
13. The client terminal includes a scanner; 2. The client terminal according to claim 1, wherein the acquiring means acquires the image data by scanning an original with a scanner.
14. The client terminal includes a storage server, 2. The client terminal according to claim 1, wherein the acquiring means acquires the image data from the storage server.
15. the client terminal includes a touch panel; 2. The client terminal according to claim 1, wherein the display control means controls so that at least a part of the image data and the list are displayed on the touch panel.
16. A character recognition means for carrying out the character recognition process. The client terminal of claim 1 further comprising:
17. The client terminal is capable of communicating with a server; The character recognition process is performed on the server; The acquiring means acquires the one or more character strings from the server.
2. The client terminal according to claim 1.
18. In the list, at least a part of the one or more character strings is displayed in a vertical arrangement.
2. The client terminal according to claim 1.
19. The list is displayed in a format that is scrollable in the vertical direction; The display control means scrolls and displays the list to a predetermined position so that a character string located at the top of the display area, among the one or more character strings recognized from the image data, is displayed at the top of a display area in which the list is displayed.
20. A client terminal according to claim 18.
20. an acquiring step of acquiring image data obtained by scanning a document and one or more character strings recognized from the image data by character recognition processing of the image data; a display control step of controlling display of a list configured of at least a portion of the image data and at least a portion of the one or more character strings; having In the display control step, it is possible to control so that a displayed area of the image data is changed within one page of the image data based on an instruction from a user, A method for controlling a client terminal, characterized in that, when the displayed area changes, the list changes so that a character string recognized in the displayed area of the image data is displayed.
21. On the computer, an acquisition step of acquiring image data obtained by scanning a document and one or more character strings recognized from the image data by character recognition processing of the image data; a display control step of controlling display of a list configured of at least a portion of the image data and at least a portion of the one or more character strings; A program for executing In the display control step, it is possible to control so that a displayed area of the image data is changed within one page of the image data based on an instruction from a user; The program, wherein the list changes when the displayed area changes so that a character string recognized in the displayed area of the image data is displayed.
Citation Information
Patent Citations
Medical information management device
JP2013229007A
Image processing device and control method thereof, and program
JP2020086717A
Image processing device, method for controlling the same, and program
JP2020140564A
