Image processing device, image processing method, and program

Abstract

For example, the present invention provides an image processing device that can appropriately identify tablespaces contained in image data and process the data. [Solution] The system comprises an acquisition unit for acquiring image data and a control unit. The control unit generates a first character map from the image data at a first resolution, which shows the presence or absence of characters in two dimensions. The control unit calculates the difference in placement information between a row of interest and rows adjacent to that row of interest, and identifies the region where the calculated difference is less than or equal to a threshold as a table region.

Description

Technical Field

[0006] ,

[0007] , ,

[0001] The present disclosure relates to an image processing apparatus and the like.

Background Art

[0002] For example, as shown in Patent Document 1, a technique for performing character recognition on an image including a table format and converting it into character data is known.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] One object of the present disclosure is to provide, for example, an image processing apparatus or the like that can appropriately identify a table area included in image data and perform data processing.

Means for Solving the Problems

[0005] In view of the above problems, the image processing apparatus in the present disclosure includes an acquisition unit that acquires image data and a control unit. The control unit generates a first character map that two-dimensionally indicates the presence or absence of character arrangement from the image data at a first resolution, calculates the difference in arrangement information between a target row and a row adjacent to the target row, and identifies an area where the calculated difference is less than or equal to a threshold value as a table area.

[0006] The image processing method in the present disclosure includes a step of acquiring image data, a step of generating a first character map that two-dimensionally indicates the presence or absence of character arrangement from the image data at a first resolution, and a step of calculating the difference in arrangement information between a target row and a row adjacent to the target row and identifying an area where the calculated difference is less than or equal to a threshold value as a table area.

[0007] The program in this disclosure provides a computer that stores image data with a function to generate a first character map from the image data at a first resolution, which shows the presence or absence of characters in two dimensions, and a function to calculate the difference in placement information between a row of interest and rows adjacent to that row of interest, and to identify a region where the calculated difference is less than or equal to a threshold as a table region. [Effects of the Invention]

[0008] According to this disclosure, for example, it becomes possible to provide an image processing device that can appropriately identify table regions included in image data and process the data. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram illustrating the system overview in the first embodiment. [Figure 2] This figure illustrates the hardware configuration of the image forming apparatus in the first embodiment. [Figure 3] This is a diagram illustrating the software configuration in the first embodiment. [Figure 4] This figure illustrates an example of the first character map in the first embodiment. [Figure 5] This is a flowchart illustrating the operation of the table data output process in the first embodiment. [Figure 6] This is a flowchart illustrating the operation of the table space estimation process in the first embodiment. [Figure 7] This is a flowchart illustrating the operation of the cell identification process in the first embodiment. [Figure 8] This is a diagram illustrating the operation in the first embodiment. [Figure 9] This is a diagram illustrating the operation in the first embodiment. [Figure 10] This is a diagram illustrating the operation in the first embodiment. [Figure 11] This is a diagram illustrating the operation in the first embodiment. [Figure 12] This is a diagram illustrating the operation in the first embodiment. [Figure 13] This is a diagram for explaining the operation in the first embodiment. [Figure 14] This is a diagram for explaining the operation in the first embodiment. [Figure 15] This is a diagram for explaining the operation in the second embodiment. [Figure 16] This is a diagram for explaining the operation in the third embodiment. [Figure 17] This is a diagram for explaining the operation in the third embodiment.

Modes for Carrying Out the Invention

[0010] Hereinafter, modes for carrying out the present disclosure will be described with reference to the drawings. Note that the embodiments described below are one of the embodiments for providing the present disclosure, and the content of the present disclosure is not limitedly interpreted based on the following description.

[0011] Generally, a technique for recognizing characters from image data generated based on contents such as a text (for example, OCR: Optical Character Recognition) is known. At this time, if the image data contains only a text (hereinafter, when simply referred to as a text, it includes a character or a character string), there is no problem. However, for example, when it has an area of a table, a problem occurs in handling the character data included in this table area (hereinafter, referred to as a table area).

[0012] For example, when the table includes grid lines, the image processing apparatus can obtain the coordinate positions of each cell by extracting the grid lines and acquire characters for each cell. However, there has been a problem that the image processing apparatus cannot correctly recognize the table area for a table without grid lines or a table having only some grid lines.

[0013] In addition, by pre-adding the attributes of the table area to the image data in advance or by pre-defining the rules of the table, the image processing apparatus can appropriately recognize the table area. However, in this case, there is a problem that the user needs to define the attributes and rules in advance, resulting in an increased workload. Also, when the user does not define the attributes and rules, there is a problem that the image processing apparatus cannot correctly recognize the table area from the image data.

[0014] To solve such one or more problems, in the following embodiments, an image processing apparatus and the like that can recognize a table area by a simple method from image data and execute appropriate processing will be described. In the following embodiments, the case where the image processing apparatus is applied to an image forming apparatus will be described. However, for example, it may be applied to an information processing apparatus (for example, a smartphone, a tablet, a computer capable of processing images).

[0015] [1. First Embodiment] [1.1 Overall System] FIG. 1 is a diagram for explaining the outline of system 1. System 1 includes an image forming apparatus 10, a terminal apparatus 20, and a server apparatus 30. In system 1, for example, the image forming apparatus 10, the terminal apparatus 20, and the server apparatus 30 are connected via a network NW.

[0016] Here, any of the apparatuses can be realized as an image processing apparatus. For example, the image forming apparatus 10 generates image data from a read document and executes image processing on the image data.

[0017] In addition, the terminal apparatus 20 executes image processing on the image data stored in the terminal apparatus 20, such as the image data read and transmitted by the image forming apparatus 10 or the image data received from the network NW.

[0018] In addition, the server apparatus 30 executes image processing on the image data read and transmitted by the image forming apparatus 10 or the image data received from the terminal apparatus 20.

[0019] Thus, the image processing device can be implemented using any of the devices. In the following embodiment, the case in which it is applied to the image forming apparatus 10 will be described.

[0020] The image forming apparatus 10 is, for example, a device called a multifunction printer or an MFP (Multifunction Peripheral / Printer / Product). For example, when the image forming apparatus 10 executes a job (print job), it can form an image on paper, which is a recording medium. The image forming apparatus 10 can execute jobs for multiple processes, such as copying, faxing, scanning, and printing.

[0021] The terminal device 20 is a device used by a user or administrator. For example, a user sends a print job to the image forming apparatus 10 from the terminal device 20. The image forming apparatus 10 executes the received print job, for example, printing on recording paper based on the print data. The administrator can also access the image forming apparatus 10 from the terminal device 20 to configure the image forming apparatus 10, for example.

[0022] The server device 30 may receive image data from the image forming apparatus 10 or the terminal device 20. The server device 30 may also transmit the processed image data to the image forming apparatus 10 or the terminal device 20.

[0023] System 1 may be equipped with the devices shown in Figure 1 as needed. For example, if the image processing device is implemented as an image forming apparatus 10, the terminal device 20 and server device 30 may be provided in System 1 as needed. Also, for example, the terminal device 20 may be one unit or multiple units may be connected. Furthermore, the server device 30 may utilize an external service (for example, a service provided on the cloud).

[0024] Furthermore, the network NW connecting the image forming apparatus 10, terminal device 20, and server device 30 can be any communication line or communication system, and may utilize wired or wireless LAN (Local Area Network), the Internet, public telephone networks, mobile communications (e.g., 4G / 5G / 6G mobile communications), next-generation telephone networks, and other communication systems.

[0025] [1.2 Hardware Configuration] In this embodiment, the hardware configuration of the image forming apparatus 10 will be described with reference to Figure 2. Figure 2 is a diagram showing an example of the image forming apparatus 10.

[0026] As shown in Figure 2, the image forming apparatus 10 has one or more of the following: a control unit 100 as a control device, a storage unit 110 (storage 112, ROM (Read-only memory) 114, and RAM (Random Access Memory) 116) as a storage device, a display unit 130 as a display device, an operation unit 140 as an operation device, an image reading unit 150 as a reading device, an image forming unit 160 as a printing device, and a communication unit 170 as a communication device.

[0027] The control unit 100 controls the entire image forming apparatus 10. The control unit 100 realizes various functions by reading and executing various programs stored in the memory unit 110 (for example, storage 112 or ROM 1114). The control unit 100 may be realized by one or more control devices / arithmetic units (CPU (Central Processing Unit), SoC (System on a Chip)). Alternatively, the control unit 100 may be composed of one or more control circuits.

[0028] The storage unit 110 stores programs, data, etc. The storage unit 110 may be composed of, for example, a storage device 112, a ROM 114, a RAM 116, etc. Alternatively, the storage unit 110 may include, for example, a cache memory included in another functional unit (for example, a control unit 100, a communication unit 170, etc.).

[0029] Storage 112 is a non-volatile storage device capable of storing programs and data. For example, it may consist of a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). Alternatively, storage 112 may be configured as an externally connectable USB memory device. Furthermore, storage 112 may be, for example, a storage area located in the cloud.

[0030] ROM114 is a non-volatile memory that can retain programs and data even when the power is turned off.

[0031] RAM116 is the main memory primarily used by the control unit 100 during processing. RAM116 is a rewritable memory that temporarily holds data including programs read from storage 112 and ROM 114, as well as execution results.

[0032] The display unit 130 is a display device capable of displaying various information and execution screens. The display unit 130 may be, for example, a liquid crystal display (LCD), an organic electroluminescence (OLED) display, or an electrophoretic display. The display unit 130 also includes an interface to which a display device can be connected. For example, it may consist of an external display device connected via HDMI® (High-Definition Multimedia Interface), DVI (Digital Visual Interface), or DisplayPort.

[0033] The operation unit 140 is an operating device that allows user input. For example, it may be a touch panel integrated with the display unit 130, or an operating device such as operation buttons. Alternatively, the operation unit 140 may be an operating device such as a keyboard or a mouse. Furthermore, the operation unit 140 may include an interface to which an operating device can be connected (for example, USB (Universal Serial Bus)). For example, the image forming apparatus 10 may connect different operating devices (operating devices having touch panels).

[0034] The image reading unit 150 reads the original document (image) and outputs it as image data. The image reading unit 150 may be, for example, a scanner, or a reading device utilizing a CCD (Charge Coupled Device) or CIS (Contact Image Sensor). The image reading unit 150 may also read image data from a recording medium such as a USB memory or an SD card (registered trademark). Furthermore, the image reading unit 150 may read image data from a terminal device such as a smartphone connected to the image forming apparatus 10 via the communication unit 170. The image reading unit 150 also functions as an image acquisition unit.

[0035] The image forming unit 160 forms an image, for example, on recording paper. The image forming unit 160 includes, for example, an image carrier, forms a toner image on the image carrier, and forms an image by transferring the image on the image carrier onto the recording paper. The image forming unit 160 may be configured as an image forming device such as a printer. Alternatively, the image forming unit 160 may form an image electronically as an image file.

[0036] The communication unit 170 is a communication interface for communicating with other devices. For example, it may be a network interface capable of providing wired connections such as Ethernet (registered trademark) or wireless connections such as IEEE 802.11a / b / g / n. The communication unit 170 may also provide a function as a base station for other devices via wireless communication. Furthermore, the communication unit 170 may function as an acquisition unit, for example, to acquire images from a network or from an external storage device.

[0037] Furthermore, in this embodiment, the communication unit 170 is capable of communicating with other devices via a network NW. The communication unit 170 may also be equipped with a communication method for connecting with terminal devices such as smartphones. For example, the communication unit 170 may communicate with terminal devices using a short-range wireless communication method such as Bluetooth® or NFC (Near Field Communication).

[0038] Note that the terminal device 20 and the server device 30 may be general-purpose information processing devices. For example, they may have one or more functions such as a control unit, a storage unit (storage, ROM, RAM), a display unit, an operation unit, and a communication unit. The configuration of the terminal device 20 and the server device 30 is self-explanatory, so a detailed explanation is omitted.

[0039] For example, the terminal device 20 and the server device 30 each have at least a control unit and a memory unit. The control unit of the terminal device controls the entire terminal device. The control unit of the terminal device implements various functions by reading and executing various programs stored in the memory unit of the terminal device (e.g., storage or ROM). This control unit of the terminal device may be implemented by one or more control devices / arithmetic units (CPU (Central Processing Unit), SoC (System on a Chip)). Alternatively, the control unit of the terminal device may consist of one or more control circuits.

[0040] Furthermore, the control unit of the server device 30 controls the entire server device. The control unit of the server device realizes various functions by reading and executing various programs stored in the storage unit of the server device (e.g., storage or ROM). This control unit of the server device may be realized by one or more control devices / arithmetic units (CPU (Central Processing Unit), SoC (System on a Chip)). Alternatively, the control unit of the server device may be composed of one or more control circuits.

[0041] [1.3 Software Configuration] The main software configuration of the image forming apparatus 10 in System 1 will be described with reference to Figure 3. Note that the software configuration shown in Figure 3 mainly describes the configuration necessary for this embodiment, and other configurations are omitted. For example, in the case of the image forming apparatus 10, it may further include configurations that provide necessary functions in the image forming apparatus 10, such as an image processing unit and a power control unit.

[0042] The control unit 100 of the image forming apparatus 10 functions as follows by executing a program (application) stored in the storage unit 110.

[0043] The character identification unit 1010 identifies characters contained in the image data stored in the image data storage area 1110. The character identification unit 1010 stores the results of the character identification in the identification data storage area 1120. The character identification unit 1010 may, for example, divide the image data into one or more areas before identifying the characters.

[0044] Here, the character identification unit 1010 can consider the following methods for identifying characters.

[0045] (1) Method for identifying the location of a character For example, the character identification unit 1010 identifies a shape that is identified as a character (for example, a predetermined shape or rectangular area) and determines the position where the character included in the image data is located. In this case, it is sufficient for the character identification unit 1010 to identify that it is a character, and it does not need to recognize what kind of character it is.

[0046] (2) Method of recognizing characters For example, the character identification unit 1010 identifies characters by recognizing characters contained in the image data through character recognition processing. As a method for recognizing characters, for example, any OCR (Optical Character Recognition / Reader) technology can be used.

[0047] Furthermore, the character identification unit 1010 may perform character recognition after dividing the image data into areas such as a character area, a photograph area, and an image area. The character identification unit 1010 performs character recognition processing on the image of the character area among the divided areas, and outputs the recognized characters as specific data. Alternatively, the character identification unit 1010 may perform character recognition from the image data by, for example, performing pattern recognition, or by using a convolutional neural network or the like.

[0048] The table data output unit 1020 outputs table data from the table area contained in the image data. Here, table data is, for example, table-formatted data consisting of rows and columns. The table data output unit 1020 may output the table data, for example, each cell along with its cell position, or it may output it in CSV (Comma Separated Value) format. Alternatively, the table data output unit 1020 may output the table data in a markup language, for example, in HTML.

[0049]

[0050]

[0051]

[0052]

[0053]

[0054]

[0055]

[0056]

[0057]

[0058]

[0059]

[0060]

[0061]

[0062]

[0063]

[0064]

[0065]

[0066]

[0067]

[0068]

[0069]

[0070]

[0071]

[0072]

[0073]

[0074]

[0075]

[0076]

[0077]

[0078]

[0079]

[0080]

[0081]

[0082]

[0083]

[0084]

[0085]

[0086]

[0087]

[0088]

[0089]

[0090]

[0091]

[0092]

[0093]

[0094]

[0095]

[0096]

[0097]

[0098]

[0099]

[0100]

[0101]

[0102]

[0103]

[0104]

[0105]

[0106]

[0107]

[0108]

[0109]

[0110]

[0111]

[0112]

[0113]

[0114]

[0115]

[0116]

[0117]

[0118]

[0119]

[0120]

[0121]

[0122]

[0123]

[0124]

[0125]

[0126]

[0127]

[0128]

[0129]

[0130]

[0131]

[0132]

[0133]

[0134] The output may also use tags. The table data output unit 1020 may be called and executed from the character identification unit 1010, or it may be executed independently of other processing units. The table area estimation unit 1030 estimates the table area from the image data using the first character map. The processing of the table area estimation unit 1030 will be described later. The cell identification unit 1040 identifies cells from the table area using the second character map. The processing of the cell identification unit 1040 will be described later. The character map generation unit 1050 generates a character map. The character map is a two-dimensional array data consisting of 0s and 1s, indicating whether or not the image data contains characters. The character map generation unit 1050 indicates the position information in the image data where characters are contained. The sentence map also indicates the arrangement of characters in the image data, and the information indicated by "0" and "1" is called arrangement information. The character map may also be stored in the character map storage area 1130. The character map storage area 1130 may store the character map as a two-dimensional array, or as text data or binary data. The control unit 100 may generate a character map each time based on the image data, or it may generate and store it in advance. For example, Figure 4 shows an example of a character map. Here, the character map is, for example, 16 rows x 16 columns of data. As will be described in detail later, when the image data contains characters, "1" is stored as placement information. The character map also has character maps of multiple resolutions. Here, the resolution of the character map may refer to the number of rows and columns when the character map is shown in two dimensions. For example, the character map may store a first character map with a first resolution of 16 rows and 16 columns, and a second character map with a second resolution of 32 rows and 32 columns. The storage unit 110 reserves areas for image data storage area 1110 for storing image data, specific data storage area 1120 for storing specific data, and character map storage area 1130 for storing character maps. The specific data storage area 1120 stores information about the character data identified by the character identification unit 1010 as specific data. The specific data may, for example, store the results of character recognition performed by the character identification unit 1010 (characters, layout, etc.). Alternatively, the specific data may store the position of characters included in the image data.Note that when the control unit 100 generates a character map directly from image data, it does not need to generate specific data. [1.4 Processing Flow] The processing flow of this embodiment will be described below with reference to the figures. The following processing will be described focusing on points that clarify the features of the present invention. The following processing may be performed by the functional units shown in Figure 3, but for the sake of explanation, it will be described as being performed by the control unit 100. [1.4.1 Table Data Output Processing] The table data output processing realized by the table data output unit 1020 will be described with reference to Figure 5. First, the control unit 100 acquires image data (S102). The control unit 100 acquires image data, for example, by reading it from the image data storage area 1110. Next, the control unit 100 determines the first character map (S104). The first character map is smaller in size than the second character map. For example, the control unit 100 determines the first character map by generating a 16x16 character map. The control unit 100 may also read a first character map that has been pre-stored in the storage unit 110. Based on the information in the first character map, the control unit 100 generates a character map suitable for the image data. For example, from the size of the image data, it determines the size of one area in the row direction (horizontal direction) and column direction (vertical direction) so that the first character map covers it. For example, if the image data is A4 size and 2894 x 4093 dots, then when the size of the first character map is 16 x 16, the size of one area indicating the arrangement will be approximately 180 x 256 dots. The control unit 100 then performs table area estimation processing using the first character map (S106). When the control unit 100 performs table area estimation processing, it can determine the approximate location of the table area within the image data. The table area estimation processing will be explained in detail in the following figure. If there is no table area in the image data, the control unit 100 terminates this process (S108; No). If a tablespace exists, the control unit 100 determines a second character map corresponding to the tablespace (S110). Here, the second character map has a higher resolution (finer resolution) than the first character map. That is, if the first character map is composed of F in the column direction and F in the row direction, a map composed of G in the column direction and G in the row direction with a value greater than F is used (F <G)。Then, the control unit 100 performs cell identification processing using the second character map for the area identified as a table area (S112). As a result, the cell in the table area included in the image data is identified by the control unit 100. Details of the cell identification processing will be described later. The control unit 100 then acquires table data from the part identified as a table area (S114). That is, since the cells included in the table area have been identified in S112, table data is output based on the data of those cells. For example, the control unit 100 may perform the following processing on the area identified as a table: (1) Perform character recognition on a part of the table area The control unit 100 performs character recognition processing on a part of the area recognized as a table area. For example, from the identified cells, the first row of a row or the first column of a column is recognized as an item name, etc., and character recognition is performed on a part of the table area such as the first row of a row or the first column of a column. In this case, the control unit 100 can acquire the necessary items and information from the table area. For example, the control unit 100 may acquire the characters recognized for each column as attribute names such as product, unit price, and quantity. (2) The control unit 100 performs character recognition on the entire table space and executes character recognition processing on the entire area recognized as a table space. For example, by performing character recognition on all cells included in the table space, it outputs table data corresponding to the entire table. At this time, the control unit 100 may output with HTML tags attached so that cells can be identified. The control unit 100 may also output in a format where cells are separated by commas (CSV format). The control unit 100 may also output with attributes attached to each cell depending on the application. For example, the control unit 100 may output attributes such as product name and price for each column. (3) The control unit 100 outputs the character data of the table space as table data. If character recognition has already been performed on the image data, the control unit 100 outputs the characters corresponding to the part recognized as a table space as table data. At this time, as explained in (2), the table data may be output in a format that allows cells to be identified. [1.4.2 Table Space Estimation Processing] The table space estimation processing will be explained with reference to Figure 6. Figure 6 shows an example of the processing performed by the control unit 100. First, the control unit 100 sets the initial values ​​of variables and parameters (S202).The variables to be judged, m and n, are variables that indicate the row to be judged in the character map. The control unit 100 sets m to an initial value of "1" and n to an initial value of m+1 to indicate the next row. The area judgment variable h is a variable that determines the size to be estimated as a table area. The control unit 100 sets the initial value of h to "0". The difference judgment threshold d is a threshold used in the character map to determine whether the arrangement information is the same when comparing each row. The control unit 100 sets the initial value of d to "3" as an example. The area judgment threshold R stores the threshold for the size to be estimated as a table. The control unit 100 sets the initial value of R to "3" as an example. That is, the control unit 100 does not need to estimate a table area smaller than R as being too small. The maximum row max is a constant that indicates the maximum number of rows in the character map. For example, in this embodiment, the first character map is a 16-row, 16-column character map, so max is set to "16". Next, the control unit 100 calculates the difference between row m, which is the row of interest to be judged, and its adjacent row n, and determines whether this difference is d or less (S204). Specifically, the control unit 100 compares the arrangement information of row m with the arrangement information of row n and counts how many arrangement information items differ. Then, if the differing arrangement information is d or less, the control unit 100 estimates it to be a table space and moves the process to S206, and if it is greater than d, it estimates it to be something other than a table space and moves the process to S216. If the difference in arrangement information between row m and row n is d or less (S204; Yes), the control unit 100 adds 1 to h because it is highly likely to be a table space (S206). Then, if the entire first character map has not been judged (n is not max), the control unit 100 adds 1 to m and determines the next row as the row of interest (S208; No → S210 → S204). Here, when n reaches its maximum value, the control unit 100 determines whether h is greater than or equal to R (S212). This is because the control unit 100 does not recognize anything as a table if it does not have a predetermined number of rows, even if the difference in the arrangement information is less than or equal to d. Here, if h is greater than or equal to R, the control unit 100 estimates that region as a table region (S214). For example, the control unit 100 estimates the area between row nh and row n as a table region.Furthermore, if h is less than R, the table space is considered too small and does not need to be estimated as a table space (S212; No). Also, in S204, the control unit 100 determines whether h is R or greater if the difference between row m and row n is not less than or equal to d (S204; No). That is, if the table space extends up to the row of interest, and the row after the row of interest is no longer part of the table space, the control unit 100 estimates the table space up to the row of interest. Here, if h is R or greater, the control unit 100 estimates the table space up to the row of interest (S216; Yes → S218). As a specific example, the control unit 100 estimates the area between row mh and row m as a table space. Also, if h is less than R, the table space is considered too small and does not need to be estimated as a table space (S216; No). Next, the control unit 100 initializes the value of h to "0" (S220), and terminates the process when n reaches its maximum (S222; Yes). If n is not at its maximum, the control unit 100 proceeds to S210 (S222; No → S210). That is, the control unit 100 selects the next row as the row of interest, calculates the difference between the row of interest and its adjacent rows, and compares them. The control unit 100 does not necessarily have to compare h and R. For example, a tablespace may be identified regardless of its size. The control unit 100 may also estimate multiple tablespaces. In this case, the control unit 100 may estimate the first estimated tablespace as the first tablespace, and the last estimated tablespace as the second tablespace. The control unit 100 may also terminate the process once a tablespace has been identified. In this case, the control unit 100 may terminate the process after S218. Furthermore, although the control unit 100 focuses on rows in Figure 6, it may also focus on columns. That is, in S204, it may compare the difference between column m and column n. Also, although the control unit 100 describes d and h as fixed values, they may be changed according to, for example, the size of the image data or the size of the first character map. For example, if the size of the first character map increases, the values ​​of d and h may be increased. [1.4.3 Cell Identification Processing] Next, the cell identification processing will be explained with reference to Figure 7. The cell identification processing applies the second character map to the table area. The control unit 100 determines whether the arrangement information of the second character map contains rows or columns with different distributions.Here, if a table area contains rows or columns with different distributions, it is determined that the table area is not a table area and is canceled (S316). When a table area is canceled, that area becomes a non-table area. Therefore, the control unit 100 cannot acquire table data from the table area in the table data acquisition process in Figure 5 (S114 in Figure 5). If there is an arrangement of consecutive "0"s in the row direction of the second character map, the control unit 100 identifies the arrangement information as a cell boundary in the row direction (S304; Yes → S306). Also, if there is an arrangement of consecutive "0"s in the column direction from the row difference in the second character map, the control unit 100 identifies the arrangement information as a cell boundary in the column direction (S308; Yes → S310). In other words, the control unit 100 may refer to the arrangement information of the second character map and recognize the area of ​​consecutive "0"s as a pseudo-table border. By identifying the cell boundaries in the row and column directions, the control unit 100 can identify cells from the table. Here, the control unit 100 cancels the table area if no cells exist in the table (S312; No → S316). Also, if cells exist in the table, the control unit 100 identifies each cell in the table from the cell boundary (S314). [1.5 Operation Example] This will be explained based on a specific operation example. Figure 8 is a diagram showing an example of a document. The control unit 100 outputs the document in Figure 8 as image data. Figure 9 is a diagram schematically showing the state based on the first character map. For example, the first character map divides the area of ​​the image data in Figure 8 into areas with a resolution of the size of the first character map (16 rows × 16 columns), and indicates whether characters are placed in each area. Figure 10 is a diagram schematically showing the state in which the first character map of Figure 9 is superimposed on the image data showing the document in Figure 8. Here, the first character map is "1" when characters are placed in each area, and "0" when no characters are placed. Figure 11 is a diagram showing only the first character map. Here, the difference in the layout information between the second and third rows is "5", so the control unit 100 does not determine it to be a table area. However, the control unit 100 determines that the difference between the third and fourth rows of the layout information is "0", so it may be a table area.When the control unit 100 performs the table space estimation process, it determines that there is almost no difference between rows 3 and 10. That is, it identifies H100 as a table space. H110 in Figure 12 corresponds to this H100 in the image data (document). In other words, the control unit 100 estimates that the image data (document) has a table space at H110. Next, an example of the operation of the cell identification process will be explained. Figure 13 is a diagram showing an example of data in the second character map. The second character map is a map with a higher resolution than the first character map, and is shown, for example, in a two-dimensional arrangement of 32 rows and 32 columns. The second character map is applied to the area that the control unit 100 has estimated to be a table space in the table space estimation process. That is, the cell identification process is performed on the table space estimated in the table space estimation process. By applying the second character map to the table space of the image data, the areas where characters are placed are set to "1" and the areas where characters are not placed are set to "0". For example, in the row direction, the column indicated by H150 is "0", so it can be seen that it is a cell boundary. Similarly, in the column direction, the column indicated by H160 is "0", so it can be seen that it is a cell boundary. In this way, the control unit 100 defines a cell boundary as an area in the row direction or column direction where there are consecutive "0"s in the arrangement information of the second character map. For example, the control unit 100 can identify C150, which is surrounded by the cell boundaries H150, H152, H160, and H162, as a cell. Then, for example, as shown in Figure 14, the control unit 100 can identify the area of ​​multiple cells. [1.6 Effects] In this way, according to this embodiment, a table area can be appropriately identified regardless of whether or not the image data contains grid lines. Furthermore, according to this embodiment, when a table area is identified, cells can be further identified and table data can be output based on the cell information. [2. Second Embodiment] The second embodiment is an embodiment for estimating multiple table areas from a single document. The second embodiment has the same hardware and software configuration as the first embodiment. This embodiment will be described focusing on the differences from the first embodiment. Specifically, in Figure 6, the control unit 100 estimates the table area estimated in S218 as the first table area.Here, the control unit 100 estimates a row as a second table area if, for example, a predetermined number of rows are separated from the first table area, and the difference between adjacent rows is small. For example, Figure 15 shows an example of the first character map in this embodiment. The control unit 100 recognizes H200 as the first table area because the difference in layout information from the 3rd to the 6th row is 0. The control unit 100 compares the layout information of the 6th and 7th rows and does not recognize them as the same table area because the difference is large. Similarly, the control unit 100 does not recognize the 8th and 9th rows as table areas. The control unit 100 estimates the 10th to 12th rows as table areas because the difference in layout information is 0. That is, the control unit 100 estimates H202 as the second table area. Then, the control unit 100 applies S110 to S114 in Figure 5 to each table area. For example, the control unit 100 determines a second character map corresponding to the size of the first table area and performs cell identification processing. After outputting the first table data corresponding to the first table area, the control unit 100 determines a second character map for the second table area and performs cell identification processing. In this way, the control unit 100 can output not only the first table data corresponding to the first table area but also the second table data corresponding to the second table area. [3. Third Embodiment] The third embodiment is an embodiment in which the output processing of table data of the first embodiment is incorporated into other applications or systems. In the third embodiment, the hardware and software configuration of the core processing part is the same as in the first embodiment. This embodiment will be explained mainly in terms of the differences from the first embodiment. In this embodiment, for example, a system that scans and electronically stores receipts, delivery slips, invoices, etc. (hereinafter referred to as receipts, etc.) will be explained as an example. Such a system, for example, performs character recognition on scanned receipts to recognize the registration number of the qualified issuer, the name of the trading partner, the transaction date, and the transaction amount, and saves them as electronic data. In addition, such a system can recognize the details of a transaction from the itemized list contained in receipts, delivery slips, etc. This embodiment applies the table data output process described in the first embodiment when recognizing the contents contained in the table area of ​​such receipts, etc.Figure 16 shows an example of the display screen W300 when a receipt or similar document is recognized. The display screen W300 displays the scanned image of the receipt or similar document on R300, along with the recognition results. When the details list button B300 is selected, the system recognizes the list of details contained in the receipt and acquires it as transaction details data. For example, the control unit 100 recognizes the area of ​​R302 as a table area and outputs it as table data. Figure 17 shows an example of the display screen W310 when the details list button B300 is selected. The display screen W310 has the details list screen W315 superimposed on it. The details list screen W315 includes the area of ​​R310 showing the recognized image and the area of ​​R320 showing the recognition results. The control unit 100 executes table data output processing to recognize the table written on the receipt and outputs it as table data. Furthermore, since the scanned document is a receipt, the control unit 100 can recognize the identified cells as the product name and price. The control unit 100 may modify the recognition results or change the area to be recognized by the user on the details list screen W315. In this embodiment, the control unit 100 may perform the process of recognizing characters in the image data at the beginning, or after recognizing the table area (for example, the area of ​​area R302). For example, the control unit 100 identifies characters in the image data (for example, by identifying characters from their shape or outline) and identifies the placement of characters in the image data from the position of the characters. Then, after recognizing the table area (area R302) from the placement of characters in the image data, the control unit 100 may perform character recognition processing on that table area. Also, when the control unit 100 performs character recognition in a table area, it may perform character recognition only on a part of the table area, or it may perform character recognition on the entire table area. Furthermore, if character recognition has already been performed on the image data, the control unit 100 may identify a table area or recognize the details based on the results of the recognized characters. [4. Modifications] This disclosure is not limited to the embodiments described above, and various modifications are possible. That is, embodiments obtained by combining technical means that are appropriately modified without departing from the gist of this disclosure are also included in the technical scope.The embodiments described above used an image forming apparatus as an example of a processing apparatus. However, the processing apparatus can be applied to other devices. For example, an example of a processing apparatus may be an information processing apparatus such as a smartphone or tablet. It may also be a home appliance equipped with IoT functionality (e.g., an air conditioner, refrigerator, television, etc.). Furthermore, it is not limited to stationary devices, but may also be portable devices or in-vehicle devices. For example, an in-vehicle device may be a car navigation system. In addition, although the embodiments described above are explained separately for the sake of explanation, they can be combined and implemented to the extent possible. Furthermore, we intend to obtain rights to any of the technologies described in this specification through amendments or divisional applications, etc. In each embodiment, the program that operates in each device is a program that controls the CPU, etc. (a program that makes the computer function) in order to realize the functions of the embodiments described above. The information handled by these devices is temporarily stored in a temporary storage device (e.g., RAM) during processing, and then stored in various ROM or HDD storage devices, and read, modified, and written by the CPU as needed. Here, the recording medium for storing the program may be any of the following: semiconductor media (e.g., ROM, non-volatile memory cards, etc.), optical recording media / magneto-optical recording media (e.g., DVD (Digital Versatile Disc), CD (Compact Disc), BD (Blu-ray® Disc), etc.), magnetic recording media (e.g., magnetic tape, flexible disk, etc.). Furthermore, when distributing the program to the market, the program may be stored on a portable recording medium and distributed, or transferred to a server computer connected via a network such as the Internet. In this case, the storage device of the server device is of course also included in this disclosure. In addition, the above-mentioned data may not be stored within the device, but may be stored on an external device and retrieved as needed. For example, the data may be stored on a NAS (Network Attached Storage) or on the cloud.The scope of this disclosure is not limited to the configurations explicitly described in the specification, but also includes combinations of the technologies disclosed herein. While the configurations for which patent protection is sought are described in the attached claims, there is no intention to exclude from the technical scope simply because they are not described in the claims. Furthermore, the phrases "in the case of..." and "when..." in the specification are explained as examples and do not imply that the configuration is limited to those described. Configurations that are not in these cases or when are also disclosed to the extent that they would be obvious to a person skilled in the art, and the author intends to obtain patent rights for them. Also, the descriptions of processes and data flows described in the specification are not limited to the order in which they are described. For example, configurations with some parts of the process deleted or the order rearranged are also disclosed, and the author intends to obtain patent rights for them. Furthermore, while the functions described in the embodiments are described as being performed by each device, they may be implemented by a single device or by utilizing an external server. Additionally, each functional block or feature of the device used in the embodiments described above may be implemented or executed by an electrical circuit, such as an integrated circuit or multiple integrated circuits. Electrical circuits designed to perform the functions described herein may include general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or combinations thereof. A general-purpose processor may be a microprocessor, a conventional processor, controller, microcontroller, or state machine. The aforementioned electrical circuits may consist of digital or analog circuits. Furthermore, if advances in semiconductor technology lead to the emergence of integrated circuit technologies that replace current integrated circuits, one or more aspects of this disclosure may utilize such new integrated circuits.[Explanation of Symbols] 1 System 10 Image forming apparatus 100 Control unit 110 Memory unit 112 Storage 114 ROM 116 RAM 130 Display unit 140 Operation unit 150 Image reading unit 160 Image forming unit 170 Communication unit.

Claims

1. It comprises an acquisition unit for acquiring image data and a control unit, The control unit, A first character map is generated from the aforementioned image data at a first resolution, which shows the presence or absence of characters in two dimensions. The difference in positional information between the row of interest and the rows adjacent to that row of interest is calculated. The region where the calculated difference is below a threshold is identified as a table region. Image processing device.

2. The image processing apparatus according to claim 1, wherein the control unit outputs the characters contained in the table area as table data.

3. The image processing apparatus according to claim 1, wherein the arrangement information is the number of characters that are consecutive in the row direction.

4. The image processing apparatus according to claim 1, wherein the control unit identifies the range of the number of characters that exist consecutively in the row direction as the range of columns included in the table area.

5. The control unit, From the aforementioned image data, a second character map is generated at a second resolution higher than the first resolution, which shows in two dimensions whether or not characters are placed in the table area. A cell boundary is determined to be a continuous area where no characters are placed in the row or column direction. The area enclosed by the aforementioned cell boundary is identified as a cell. The image processing apparatus according to claim 1.

6. Steps to acquire image data, The steps include generating a first character map from the aforementioned image data, which shows the presence or absence of character placement in two dimensions, at a first resolution, The steps include: calculating the difference in placement information between the row of interest and the rows adjacent to that row, and identifying the region where the calculated difference is less than or equal to a threshold as a tablespace; Image processing methods including [specific details omitted].

7. In a computer that stores image data, A function to generate a first character map at a first resolution that shows the presence or absence of character placement in two dimensions from the aforementioned image data, This function calculates the difference in placement information between a row of interest and rows adjacent to that row, and identifies areas where the calculated difference is below a threshold as tablespaces. A program that achieves this.

Images