Information processing apparatus, method of controlling information processing apparatus, and program

Abstract

To provide an image data management method for extracting a character string from image data while reducing false determination and processing load.SOLUTION: An information processing apparatus includes: image analysis means which detects a character string region existing image data, and recognizes a character string block described in the character string region; search character string generation means which connects to the character string block to generate a search character string for searching; and recording means which records the image data and information on the search character string in storage means. The search character string generation means generates a plurality of search character strings based on multiple reading orders of the character strings, and excludes portions having the same order in the generated search character strings.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to an information processing apparatus that records image data of a document in a format that allows full-text search, and a method and program for controlling the information processing apparatus. [Background technology]

[0002] Conventionally, a technique has been proposed in which character recognition processing (OCR (Optical Character Recognition)) is performed on image data of a document, and image data and OCR character strings are linked and recorded to enable character search. In addition, many techniques have been proposed to estimate the connection order of character strings (hereinafter referred to as "reading order"). However, when a unique reading order is specified, there is a problem that if the specified reading order is incorrect, character searches may miss some characters. In recent years, in order to improve the accuracy of estimating reading orders, technology has been proposed that interprets the meaning of character strings and determines whether they are linguistically related. However, the language processing that determines the linguistic relatedness of each character string imposes a large processing load. There is also a method to estimate the reading order from the relative positions of character strings. However, low-load reading order estimation techniques are prone to misjudgment of the reading order. The connection relationships of the writing positions are subject to a variety of rules, and it is difficult to apply a specific connection algorithm to all possibilities. As a technique for addressing this issue, for example, the technique described in Patent Document 1 has been proposed. The technology described in Patent Document 1 improves character search accuracy by generating search strings corresponding to multiple reading orders (writing directions) from intermediate OCR results, thereby absorbing the effects of erroneous determination of the writing direction by OCR. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2009-98777 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the method of Patent Document 1 has the problem that it generates search character strings for each column in a document, and cannot search for keywords that are separated outside the columns due to line breaks, etc. In addition, it generates reading order candidates for all character strings in a document, and the search time increases as the amount of data to be searched increases. In order to solve the above problems, an object of the present invention is to provide a mechanism for extracting character strings from image data and managing image data with reduced erroneous determinations and low processing load. [Means for solving the problem]

[0005] The present invention is an information processing device comprising: image analysis means for detecting character string areas present in image data and recognizing character string blocks written in the character string areas; search string generation means for connecting the character string blocks to generate a search character string to be searched; and recording means for recording the image data and information on the search string together in storage means, wherein the search string generation means generates a plurality of search strings based on a plurality of reading orders of the character strings and excludes portions of the generated plurality of search strings that have the same order. Effect of the Invention

[0006] According to the present invention, it is possible to provide a mechanism for extracting character strings from image data and managing image data with reduced erroneous determinations and low processing load. [Brief description of the drawings]

[0007] [Figure 1] 1 is a diagram showing the overall configuration of a print log management system according to an embodiment; [Diagram 2] FIG. 2 is a diagram illustrating an example of the hardware configuration of a print log management server according to the embodiment. [Diagram 3] FIG. 2 is a diagram illustrating an example of the software configuration of a print log management server according to the embodiment. [Figure 4]FIG. 4 is a diagram showing an overall process flow executed by the print log management server of the embodiment. [Diagram 5] 5A to 5C are views showing examples of a print image to be processed and a character recognition result in the embodiment. [Figure 6] FIG. 4 is a diagram showing a process flow of a search string generation process in the embodiment. [Figure 7A] FIG. 13 is a diagram showing an example of reading order processing (priority given to the area of ​​the ruled lines) in the embodiment. [Figure 7B] FIG. 13 is a diagram showing an example of reading order processing (prioritizing the same line) in the embodiment. [Figure 7C] FIG. 13 is a diagram showing an example of reading order processing (preferentially giving priority to adjacent line breaks) in the embodiment. [Figure 8A] FIG. 4 is a diagram showing a print log search screen in the embodiment. [Figure 8B] FIG. 13 is a diagram showing a search result confirmation screen in the embodiment. [Figure 9A] FIG. 13 is a diagram showing a print log search screen according to another embodiment. [Figure 9B] FIG. 13 is a diagram showing a search result confirmation screen in another embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, the embodiments of the present invention will be described with reference to the drawings. Note that the following embodiments do not limit the invention according to the claims, and all of the combinations of features described in the embodiments are not necessarily essential to the solution of the invention. [Embodiment 1]

[0009] Overall system configuration 1 is a diagram showing an example of the configuration of a print log management system in this embodiment. This system is composed of a printer 101, a print log management server 102 which is an information processing device, and a user terminal 103, which are connected to each other so as to be able to communicate with each other via a network 104. As an example of this network 104, the Internet, a LAN (Local Area Network), or the like is applied. The user terminal 103 is a terminal device having an application, a printer driver, etc., and into which a user inputs a print instruction for document data. Examples of the terminal device include a general-purpose computer such as a personal computer, and a portable terminal device such as a smartphone or a tablet terminal. When a print instruction for document data edited by an application is input into the user terminal 103, the document data is converted into print data such as PDL data that can be interpreted by the printer 101 by the printer driver. PDL stands for Page Description Language. The converted print data is then transmitted to the printer 101 and the print log management server 102. The print data includes print attributes such as the destination printer name, the print job name, identification information (host name or IP address, etc.) of the user terminal 103 that made the print request, the user name of the print requester, a time stamp, print setting information (2-in-1, double-sided, enlarged, reduced, etc.), and the number of copies to be printed. The printer 101 receives print data from the user terminal 103, and prints the document data on paper in accordance with the print setting information. The print log management server 102, which is an information processing device, generates and analyzes image data from print data transmitted by the user terminal 103, converts the printed document into a format that allows full-text search, and records and manages it as a print log. It also accepts a search operation of the print log performed by an administrator (e.g., a security administrator) who checks whether the printed document is appropriate, and displays a confirmation screen. When an incident of information leakage occurs, the administrator can perform a keyword search of the print log specifying the attributes of the print data or the written character string, and check the printed image of the document that is found in the search. The search and confirmation of the print log may be performed by providing an administrator terminal separate from the print log management server 102. There are no particular limitations as long as the print log stored in the print log management server 102 can be accessed.

[0010] Print log management server hardware configuration 2 is a diagram illustrating an example of a hardware configuration of the print log management server 102, which is an information processing device according to the present embodiment. As illustrated in FIG. 2, the print log management server 102 includes a control unit 220, a communication unit 225, a display unit 226, an operation unit 227, and a storage unit 228. The control unit 220 includes a CPU 221, a ROM 222, a RAM 223, and an input / output interface (I / O) 224, and these units are connected to each other via a bus. A communication unit 225, a display unit 226, an operation unit 227, and a storage unit 228 are connected to the I / O 224. Each of these functional units is capable of communicating with the CPU 221 via the I / O 224. The communication unit 225 communicatively connects the print log management server 102 to the network 104 . As an example, a liquid crystal display (LCD) or an organic EL (Electro Luminescence) display is used as the display unit 226. The display unit 226 displays information such as the print log stored therein so that the administrator can check it. As an example, the operation unit 227 is implemented by an operation input device such as a keyword or a mouse. The operation unit 227 accepts various instructions from the administrator to the print log management server 102. The display unit 226 and the operation unit 227 may be integrated into one device such as a touch panel. The display unit 226 and the operation unit 227 may be displayed and operated directly by the administrator, or may be displayed and operated via a separately provided administrator terminal. The storage unit 228 may, for example, be a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. The storage unit 228 stores an information processing program 229, a print log, etc.

[0011] Print log management server software configuration 3 is a diagram showing an example of the software configuration of the print log management server 102, which is an information processing device in this embodiment. The information processing program 229 is made up of a main processing unit 300, a print image generation unit 301, an image analysis unit 302, a search string generation unit 303, a print log management unit 304, and a display processing unit 310. The main processing unit 300 uses a print image generating unit 301 to a print log management unit 304 to generate image data from print data, convert the printed document into a format that allows full text search, and record and manage the data as a print log. The print image generating unit 301 performs rendering on the print data sent from the user terminal 103, and generates image data (print image) to be printed. In this embodiment, the print image is generated by the print log management server 102, but this is not limited to this. A print image generated by a printer driver or a print image generated by the printer 101 may be acquired. The image analysis unit 302 detects character string areas present in a print image, and recognizes character strings written in the character string areas. A search string generation unit 303 estimates a plurality of connection orders (reading orders) between the character strings recognized by the image analysis unit 302, and generates a search string to be used for a character search. The print log management unit 304 stores the print data received from the user terminal 103, the print image, the generated search character string, and the like in the storage unit 228 together. The display processing unit 310 displays a UI (User Interface) screen that allows the administrator to perform a search operation on the print log. The administrator can use the UI screen displayed here to perform a keyword search on the search string and check the print data of the document that is found in the keyword search. Fig. 8 is an example of a UI screen displayed by the display processing unit 310.

[0012] Overall process flow 4 is a flowchart showing the overall flow of processing executed by the main processing unit 300 in this embodiment. The processing of this flowchart is realized by the information processing program 229 stored in the storage unit 228 being loaded into the RAM 223 and executed by the CPU 221. Each processing step is hereinafter represented by "S". This process starts when the print log management server 102 receives print data from the user terminal 103 . In S401, the print image generation unit 301 performs rendering on the print data transmitted from the user terminal 103, and generates image data to be printed (print image). In S402, the image analysis unit 302 performs area analysis for each attribute of an object in the image, and extracts an area in which a character string is written (character string area). First, the print image that has been binarized to black and white is contour-traced to extract clusters of pixels surrounded by black pixel contours. Then, for clusters of black pixels larger than a certain area, the contours of the white pixels inside are also traced to extract clusters of white pixels, and clusters of black pixels are recursively extracted from inside clusters of white pixels that have an area of ​​at least a certain size. The clusters of black pixels thus obtained are classified by size and shape, and are then classified into regions with different attributes. For example, pixel clusters with an aspect ratio close to 1 and within a certain range of size are extracted as character regions (CHARACTER). In addition to character regions, flat pixel clusters are extracted as ruled lines (LINE), regions with scattered clusters of irregular pixels are extracted as photo regions (PHOTO), and other pixel clusters of arbitrary shapes are extracted as picture regions (PICTURE). Next, each character extracted as a character area (CHARACTER) is connected to adjacent characters, and grouped as a single-line character string area (TEXT). In this embodiment, adjacent character strings are grouped under the condition that they are aligned as one line and do not cross ruled lines. In specific grouping processing, characters on the same line are first collected, and then the character group is separated from the group of characters on the same line at parts where the character size is significantly different or the character spacing exceeds a predetermined value or at ruled lines. Finally, the area occupied by the separated group of characters is regarded as a character string area. A method other than the method of this embodiment may be used for the character string area as long as it is a single line and does not cross ruled lines. In S403, the image analysis unit 302 performs OCR processing on the character string area detected in S402. OCR is a technology for recognizing characters written in a region of interest in an image and converting them into character string data. Note that the method for extracting character string data is not limited to the method of this embodiment. For example, if the print data electronically holds text information from a source application (word processing software, spreadsheet software, etc.), the character string data may be obtained from that information.

[0013] (Extracting character string regions) FIG. 5 shows the result of extracting character string regions from a printed image. FIG. 5(a) shows a print image 500 generated in S401, and FIG. 5(b) shows each character string area extracted in S402 with a dotted line block. The numbers in the dotted line blocks are area IDs assigned to each character string area. Each character string area is subjected to OCR processing in S403. The character string data obtained by the OCR processing is linked to the character string area and stored as a character string block to be used in the subsequent search character string generation process. The character string block is specifically acquired in the format of character string block information 502 in FIG. 5(b). The character string block information 502 stores an area ID (id), a character string (text), a character string area (rect), and further character information (chars) for each character included in the character string block. The character information (chars) for each character stores information on the writing area of ​​each character, and is referred to later to highlight character strings hit in a full-text search performed on the document. As long as the character string area and character string can be stored as information on the character string block, the data format is not limited to the above.

[0014] In S404, the search string generation unit 303 estimates a plurality of connection orders (reading orders) between the character strings extracted in S403, and generates a search string to be subjected to a character search. Details of the process in S404 will be described later. In S405, the print log management unit 304 records the print log in the storage unit 228, and this processing flow ends. The print log includes the print data, search strings generated from the print image, etc., and can be viewed, displayed, and searched by the administrator. In this embodiment, the search strings generated in S404 are stored in a database as shown in Table 1 so that they can be associated with information on the writing area of ​​each character. Based on the position of the character in the search string (how many characters it corresponds to from the beginning), the corresponding character string block is matched with the corresponding character, and the character string block information is read, making it possible to obtain information on the writing area of ​​each character.

[0015] [Table 1]

[0016] As shown in Table 1, by storing the information in a database so that it can be associated with the information on the writing area of ​​each character, it is possible to highlight character strings that are found in a full-text search performed on the document later, but this is not limited to the recording method of this embodiment. For example, a searchable PDF may be created by generating a transparent text layer (search layer) based on the search string in accordance with the information in the writing area, combining it with a print image (raster image), and converting it into a PDF format.

[0017] <Search string generation process details> 6 is a flowchart showing a detailed flow of the process of generating a search string in S404. This process starts when the OCR process in S403 is completed for the print image and all the information of the string blocks in the print image (string areas and strings) is collected. Note that the method of extracting the string blocks is not limited to the method of this embodiment. For example, if the print data electronically holds text information from a source application (word processing software, spreadsheet software, etc.), the string blocks may be created based on that information. In S601, the search string generating unit 303 acquires information (string area and string) of a string block included in a print image. In S602, the search string generation unit 303 estimates a plurality of connection orders (reading orders) for the string blocks acquired in S601. In this embodiment, the estimated reading order is stored in the form of an array in which the region IDs of the string blocks are arranged (e.g., 70a, 70b, and 70c in Figs. 7A, 7B, and 70C). In this embodiment, a highly accurate reading order estimation that imposes a large processing load, such as interpreting meanings and determining the relationship between languages, is not performed, but a low-load method of estimating the reading order from the relative positions of character strings is applied.

[0018] 7A, 7B, and 7C show three examples of the reading order estimation process and the results. (The area separated by lines takes precedence) FIG. 7A is a flowchart showing a flow of estimating the reading order based on the result of rearranging the characters so that the characters within the area (frame) separated by the ruled lines are connected with priority. In S701, character string area information of each character string block is obtained. In S702, line information is obtained. In S703, a group is created of the character string blocks that belong to the area (frame) separated by the ruled lines obtained in S702. The grouping results in S703 are shown as frames for printed image 500 in Fig. 7A. Referring to printed image 500 in Fig. 7A, it can be seen that character string blocks 513, 516, etc. are grouped together as an in-frame group. In S704, the storage order of the string blocks is rearranged based on the grouping result in S703, and the result is output. In S704, the string blocks are rearranged from top to bottom and left to right based on the center of gravity of the string regions in each group, and the groups are connected and output. In this embodiment, as shown in the reading order estimation result 70a, the output result is a reading order array that retains grouping information.

[0019] (Same line takes precedence) FIG. 7B is a flowchart showing the flow of estimating the reading order based on the result of rearranging the characters so that the same lines are connected with priority. In S711, character string area information of each character string block is obtained. In S712, each character string block is determined to be written vertically or horizontally and then sorted. In this embodiment, the determination of vertical or horizontal writing is performed using the aspect ratio of the character string area, and a character string block that cannot be determined to be written vertically or horizontally, such as a single character area, is sorted into both vertical and horizontal writing. In S713, a group of identical lines is created for each of the vertically written and horizontally written lines sorted in S712. In this embodiment, identical lines are determined based on whether the heights of the lines overlap each other by a specific percentage or more. The results of grouping in S713 for printed image 500 in Fig. 7B are shown by frames. Referring to printed image 500 in Fig. 7B, it can be seen that character string blocks 510, 511, 512, 513, 514, etc. are grouped together as in-line groups. Note that in printed image 500, all character strings are written horizontally. In S714, the storage order of the character string blocks is rearranged based on the grouping result in S713, and the result is output. In S714, rearrangement is performed within each group so that horizontal writing is arranged from left to right, and vertical writing is arranged from top to bottom, and each group is connected and output. In this embodiment, as shown in the reading order estimation result 70b, the output result is a reading order arrangement that retains grouping information.

[0020] (Prefer nearby line breaks) FIG. 7C is a flowchart showing the flow of estimating the reading order based on the result of rearranging the characters so that adjacent line breaks are connected with priority. In S721, character string area information of each character string block is obtained. In S722, each character string block is determined to be written vertically or horizontally and then sorted. In this embodiment, the determination of vertical or horizontal writing is performed using the aspect ratio of the character string area, and a character string block that cannot be determined to be written vertically or horizontally, such as a single character area, is sorted into both vertical and horizontal writing. In S723, groups that can be connected by nearby line breaks are created for each of the vertical and horizontal writings sorted in S722. In this embodiment, nearby line breaks are determined based on whether the line spacing between the two lines is within a specific ratio and whether the line widths overlap by more than a specific ratio. The results of grouping in S723 for the printed image 500 in FIG. 7C are shown in frames. Referring to the printed image 500 in FIG. 7C, it can be seen that character string blocks 510, 513, 516, 511, 514, etc. are grouped together as nearby line break groups. Note that all character strings in the printed image 500 are sorted into horizontal writing. In S724, the storage order of the character string blocks is rearranged based on the grouping result in S723, and the result is output. In S724, rearrangement is performed within each group so that horizontal writing is arranged from top to bottom, and vertical writing is arranged from right to left, and each group is connected and output. In this embodiment, as shown in reading order 70c, the output result is a reading order arrangement that retains grouping information. 7A, 7B, and 7C are merely examples, and are not limited to the method of this embodiment. For example, reading order estimation may be performed within a column or with the same font as a priority connection, or more reading order patterns may be added.

[0021] In S603 of Fig. 6, the search string generation unit 303 selects one of the multiple reading orders estimated in S602 as the main reading order. In this embodiment, the reading order that prioritizes intra-box connection (reading order 70a in Fig. 7A) is set as the main reading order. In S604, the search string generation unit 303 generates a main search string based on the standard reading order. The search string is generated by connecting character strings corresponding to the area IDs of the reading order result with line breaks (\n). In the example of the print image 500 in FIG. 5, the main search string is "Report (\n) Case name (\n) Visit date...ABC project (\n)...Keyword S (\n) AXXX...". In S605, the search string generation unit 303 selects one sub-reading order other than the main reading order from among the multiple reading orders estimated in S602. In this embodiment, the reading order that prioritizes same-line connections or adjacent line break connections (70b and 70c in Figs. 7B and 7C) is selected. In S606, the search string generation unit 303 compares the sub-reading order selected in S605 with the standard reading order to determine whether there is a part with the same order. In this embodiment, the order of area IDs of the string blocks is compared. From the standard reading order, string blocks are connected until there are at least two blocks worth of characters and the number of characters reaches a predetermined number, and it is determined whether the order exists in the sub-reading order. If there is a part with the same order (Yes in S606), the process proceeds to S607, and if there is no part with the same order (No in S606), the process proceeds to S608. For example, in the reading order 70b in FIG. 7B, the order of area IDs "2→3→4", "6→7", etc. are common to the main reading order. In this embodiment, the reading order is determined based on grouping, and an individual character string block that has not been grouped is determined to have the same order as the main reading order for that block alone. For example, the character string in the character string block 501 is "report", which is not grouped with other character strings in either the same line priority reading order 70b or the adjacent line break priority reading order 70c, so the character string "report" may be searched for in the main reading order. In S607, the search string generation unit 303 excludes the parts that are determined to be in the same order as the main in S606 from the sub reading order. For example, in the reading order 70b in Fig. 7B, the parts with area IDs "1 → 2 → 3 → 4" and "6 → 7" are excluded, and "5 → 6 → 9 → 10 → 11 → 12..." remains as the reading order result. In addition, in reading order 70c of FIG. 7C, area IDs such as "5→8" and "9→11→13→15" are excluded, and "2→5→3→6→4→7→15→10..." remains as the reading order result. In S608, the search string generation unit 303 generates a sub search string based on the sub reading order. The search string is a string corresponding to the area ID of the reading order result, separated by a line break (\n). In the example of the print image 500 in FIG. 5, the sub search string has the following two patterns. "ABC Project (\n) 2022 / 09 / 16 (\n) ··· Keyword S (\n) BXXX ···", "Case name (\n) ABC Project (\n) Visit date (\n) 2022 / 09 / 16 ··· Keyword S (\n) CXXX ···". In this embodiment, two patterns of sub search strings remain, but multiple sub search strings may be generated or may disappear. They may also be connected and held. In S609, the search string generation unit 303 determines whether the processes of S603 to S604 or S605 to S608 have been completed for all of the multiple reading orders estimated in S602. If the processes have been completed for all of the reading orders (Yes in S609), this process ends. On the other hand, if the processes have not been completed for all of the reading orders (No in S609), the process proceeds to S605 to S608. In this embodiment, one main reading order is selected as the reference reading order and compared with the non-main reading orders, but after completing this flow, the reference can be further selected from the non-main reading orders, and the non-main reading orders can be compared with each other, and any parts with the same order can be excluded.

[0022] <Print log search and result display screen (UI)> The administrator can view, display, and search the print log information stored in the storage unit 228 in S405. The display processing unit 310 displays various information included in the stored print log on a UI (User Interface) screen in response to a request from the administrator. FIG. 8 is a diagram showing an example of a UI screen displayed by the display processing unit 310 in this embodiment. 8A includes a search condition specification area 1010 and a print log list area 1020. On the print log search screen 1000, an administrator specifies search conditions and checks corresponding print logs. The search condition specification area 1010 is a condition specification area for filtering the print logs to be displayed in a list by specifying print attribute information or keywords. In this embodiment, the search conditions are "date," "login name," "file name," etc., but are not limited to these. Other attributes may be specified, and multiple conditions may be added. Furthermore, the keyword specification area 1011 accepts input of keywords for full-text search. The print log list area 1020 displays a list of print logs in reverse chronological order based on the search results specified in the search condition specification area 1010. If no search conditions are specified, all print logs are displayed. In this embodiment, the contents displayed in the list are job names, transmission dates and times, etc., but are not limited to these. The search result confirmation screen 1100 in FIG. 8B includes a preview area 1030 and a print log information area 1040. In the search result confirmation screen 1100, the administrator confirms the image and operation information of the selected print log. In this embodiment, when the administrator selects (clicks) any line in the print log list area 1020, the search result confirmation screen 1100 is displayed. Then, the administrator can confirm the details of the corresponding print log, such as the image of the print log and operation information, in the preview area 1030 and the print log information area 1040. The preview area 1030 displays a preview of a print image of the print log selected in the print log list area 1020. In this embodiment, character string areas 1031, 1032 in which the keyword specified in the keyword specification area 1011 is hit are highlighted. In this embodiment, the background color of the highlighted character string area is differentiated depending on whether the reading order of the hit character string has been confirmed (whether there are other candidate reading orders). By displaying the parts 1031, 1032 that have been hit in character string areas with confirmed reading orders more conspicuously, it is possible to guide the administrator's gaze to check character strings that are unlikely to be erroneously hit with priority. The highlighting method is not limited to applying a background color to the character string area, and may be a method of changing the character color or character font. The print log information area 1040 displays detailed information about the print log selected in the print log list area 1020 . The pop-up area 1050 is displayed on the front of the search result confirmation screen 1100 when the mouse is placed on a character string area (here, character string area 1031) with a reading order candidate among the highlighted character string areas. The pop-up area 11050 displays character strings that are multiple reading order candidates. By displaying the pop-up area 11050, the administrator can easily check the cause of a mishit, such as in the case of a mishit. In this embodiment, the reading order candidates are displayed and the administrator only checks them, but some operation by the administrator may be accepted separately. For example, an instruction to confirm the reading order from the reading order candidates may be accepted, and search character strings with reading orders other than the confirmed one may be deleted from the log. Also, an instruction to search for another document with the same layout as the document with the mishit may be accepted, allowing editing of the search character string. As described above, by applying the present invention, it is possible to determine parts where the reading order cannot be identified from multiple reading order candidates, and to use multiple possibilities as search strings only for parts where the reading order cannot be identified, thereby preventing missed searches and reducing search time.

[0023] [Embodiment 2] In the first embodiment, a method was described for preventing missed searches and reducing search time by determining the parts where the reading order cannot be specified from the multiple reading order candidates, and setting multiple possibilities as search strings only for the parts where the reading order cannot be specified. However, a character search does not necessarily hit any one of the multiple reading orders, and if there are duplicate hits in multiple search strings, it is necessary to narrow down the hit strings to one and display them. Therefore, in this embodiment, when a fuzzy search for a string results in duplicate hits for multiple search strings, an example is described in which the hit strings are narrowed down to one. Fuzzy search is a search method that allows similar strings other than exact matches, such as partial matches of strings. Note that only the differences from the first embodiment are described below. 9A and 9B are diagrams showing an example of a UI screen displayed by the display processing unit 310 in this embodiment. 9A, a character string "keyword SABC" is specified as a keyword in a keyword specification area 1011, and it is specified that fuzzy search is permitted in a "fuzzy search" permission specification area 1014. Then, the search results of the print logs are displayed in a list in a print log list area 1020. 9B, a preview image 1030 and detailed information 1040 of the print log selected (clicked) by the administrator in the print log list area 1020 are displayed. In this embodiment, a preview image of the results of a fuzzy search performed by specifying the keyword "keyword SABC" for the print image 500 in FIG. 5 is displayed. From the three reading order estimation results shown in Figs. 7A, 7B, and 7C, a search string is generated as a succeeding string of the string "keyword S" 1101, in which three patterns of "AXXX..." 1102, "BXXX..." 1103, and "CXXX..." 1104 are connected. Then, as a result of the fuzzy search, the keyword is hit in all three patterns of the search string. That is, in the string in which "keyword S" 1101 and "AXXX..." 1102 are connected in the first reading order (main), "keyword SA" is the hit string. In the string in which "keyword S" 1101 and "BXXX..." 1103 are connected in the second reading order (sub), "keyword SB" is the hit string. In the string in which "keyword S" 1101 and "CXXX..." 1104 are connected in the third reading order (sub), "keyword SC" is the hit string. In this embodiment, a screen is provided that highlights character strings that match a specified keyword on a preview image, but highlighting all of the duplicated hit character strings would make the screen cluttered. Therefore, duplicate hit character strings are narrowed down to one by selecting one of them, and then highlighted. In this embodiment, the hit character string that has the highest similarity to the original keyword is selected from among the hit character strings. In this embodiment, the similarity is calculated based on the Levenshtein distance between the character strings. Here, the Levenshtein distance is a type of distance that indicates how different two strings are, and is defined as the minimum number of steps required to transform one string into another by inserting, deleting, or substituting one character. Here, the first “Keyword SA” that is most similar to the original keyword “Keyword SABC” is selected, and the selected hit strings are highlighted on the preview UI screen as shown in “Keyword S” 1101 and “AXXX...” 1102 in FIG. 9B. When multiple hits are found in multiple reading orders, the method of selecting a hit character string in which reading order is not limited to the method of this embodiment. For example, the one with the closest distance between the connected character string regions may be selected, or a reading order estimated using a more advanced logic may be selected as the main reading order. Furthermore, these may be combined. As described above, by applying the present invention, in a character search performed on multiple search character strings, if duplicate hits occur in multiple search character strings, it is possible to narrow down the hit character strings to one and display the search results.

[0024] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) An information processing device, an image analysis means for detecting a character string area present in image data and recognizing a character string block written in the character string area; a search string generating means for generating a search string to be searched by connecting the string blocks; a recording means for recording the image data and the information on the search character string together in a storage means; The search character string generating means generating a plurality of search strings based on a plurality of reading orders of the character strings; The generated search strings are excluded from the same sequence. 23. An information processing apparatus comprising: (Configuration 2) The search string generating means excludes a portion where connected string blocks are arranged in the same order. 2. The information processing device according to configuration 1. (Configuration 3) The reading order is to rearrange the character string blocks so that they are connected within the area separated by the ruled lines. 2. The information processing device according to configuration 1. (Configuration 4) The reading order is to rearrange the same lines so that they are connected. 4. The information processing device according to any one of configurations 1 to 3. (Configuration 5) The same lines are lines whose heights overlap by a certain percentage or more. 5. The information processing device according to configuration 4. (Configuration 6) The reading order is rearranged so that adjacent line breaks are connected. 4. The information processing device according to any one of configurations 1 to 3. (Configuration 7) The adjacent line breaks are those where the line spacing is within a specific percentage and the line width overlaps by more than a specific percentage. 7. The information processing device according to configuration 6. (Configuration 8) The information on the string block is a string region and a string. 8. The information processing device according to any one of configurations 1 to 7. (Configuration 9) a search means for searching and displaying the image data by a keyword; The search means displays an image in which the keyword is relevant in an emphasized manner. 9. The information processing device according to any one of configurations 1 to 8. (Configuration 10) The search means has a function of fuzzy search, and has a narrowing-down means for narrowing down the search results when a plurality of character strings are found. 10. The information processing device according to any one of configurations 1 to 9. (Configuration 11) The image data is print image data of a print log. 11. The information processing device according to claim 1, wherein the information processing device is a computer. (Method 1) A method for controlling an information processing device, comprising: an image analysis step of detecting a character string area present in the image data and recognizing a character string block written in the character string area; a search string generating step of generating a search string to be searched by connecting the string blocks; a recording step of recording the image data and the information on the search character string together in a storage means; The step of generating a search string includes: generating a plurality of search strings based on a plurality of reading orders of the character strings; The generated search strings are excluded from the same sequence. 23. A method for controlling an information processing apparatus comprising: (Program 1) A program for causing a computer to execute the control method for an information processing device according to the first method. [Explanation of symbols]

[0025] 101 Printer 102 Printing log management server 103 User terminal

Claims

1. An information processing device, an image analysis means for detecting a character string area present in image data and recognizing a character string block written in the character string area; a search string generating means for generating a search string to be searched by connecting the string blocks; a recording means for recording the image data and the information on the search character string together in a storage means; The search character string generating means generating a plurality of search strings based on a plurality of reading orders of the character strings; The generated search strings are excluded from the same sequence.

23. An information processing apparatus comprising:

2. The search string generating means excludes a portion where connected string blocks are arranged in the same order.

2. The information processing apparatus according to claim 1,

3. The reading order is to rearrange the character string blocks so that they are connected within the area separated by the ruled lines.

2. The information processing apparatus according to claim 1,

4. The reading order is to rearrange the same lines so that they are connected.

4. The information processing device according to claim 1, wherein the information processing device is a computer.

5. The same lines are lines whose heights overlap by a certain percentage or more.

5. The information processing apparatus according to claim 4.

6. The reading order is rearranged so that adjacent line breaks are connected.

4. The information processing device according to claim 1, wherein the information processing device is a computer.

7. The adjacent line breaks are those where the line spacing is within a specific percentage and the line width overlaps by more than a specific percentage.

7. The information processing apparatus according to claim 6,

8. The information on the string block is a string region and a string.

4. The information processing device according to claim 1, wherein the information processing device is a computer.

9. a search means for searching and displaying the image data by a keyword; The search means displays an image in which the keyword is relevant in an emphasized manner.

4. The information processing device according to claim 1, wherein:

10. The search means has a function of fuzzy search, and has a narrowing-down means for narrowing down the search results when a plurality of character strings are found.

4. The information processing device according to claim 1, wherein the information processing device is a computer.

11. The image data is print image data of a print log.

4. The information processing device according to claim 1, wherein the information processing device is a computer.

12. A method for controlling an information processing device, comprising: an image analysis step of detecting a character string area present in the image data and recognizing a character string block written in the character string area; a search string generating step of generating a search string to be searched by connecting the string blocks; a recording step of recording the image data and the information on the search character string together in a storage means; The step of generating a search string includes: generating a plurality of search strings based on a plurality of reading orders of the character strings; The generated search strings are excluded from the same sequence.

23. A method for controlling an information processing apparatus comprising:

13. A program for causing a computer to execute the method for controlling an information processing device according to claim 12.

Images