Table recognition method and device
Through text projection and state machine recognition of the starting position of table rows, the problems of table recognition consistency and real-time in the prior art are solved, the recognition accuracy and robustness are improved, and it is suitable for a variety of table structures, saving the training of neural networks Time and annotation cost.
Patent Information
- Application Number
- PCT/CN2024/072760
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-04
- Filing Date
- 2024-01-17
- Publication Date
- 2024-11-07
AI Technical Summary
The prior art has the problem of identification consistency in table recognition, especially in complex scenarios, and the identification is inaccurate, and the table structure cannot be recognized and returned in real time. The training time and labeling cost of neural networks are high, so it cannot be applied to multiple styles. Wireless table structure.
Automatically identify the starting position of the table row and column through text projection and state machine, generate binarized images, accumulate pixel values to obtain row projection and column projection, determine the starting and end positions of rows and columns through state machine scanning, eliminate interference factors, and apply Identify scenarios in various table structures.
It improves the accuracy and robustness of table recognition, saves training time and labeling costs of neural networks, realizes real-time recognition and high processing speed, and is suitable for various styles of wireless table structures.
Smart Images

Figure CN2024072760_07112024_PF_FP_ABST
Abstract
Description
Table recognition method and device
[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on May 4, 2023, with application number 202310493421.1 and application name “A Form Recognition Method and Device”, the entire contents of which are incorporated by reference into this application. Technical Field
[0002] The present application relates to the field of image recognition technology, and in particular to a table recognition method and device. Background Art
[0003] With the development of computer technology, the use of electronic documents has become increasingly popular. Users often convert paper documents into electronic documents by scanning, photographing, or other methods to use and save them. Later, when other users use electronic documents, they can use technologies such as image processing and optical character recognition (OCR) to convert the original non-editable electronic documents into editable documents to meet their needs.
[0004] Summary of the Invention
[0005] However, when electronic documents include tables, the diversity of table structures leads to numerous issues with consistent table recognition. For example, coverage is incomplete. For example, recognition of some real-world scenarios, especially complex ones, is inaccurate. For example, for some scenarios, pre-building datasets suitable for these scenarios to train neural networks can save both training time and the cost of labeling the training sets. For example, it's impossible to recognize and return table structures in real time.
[0006] To solve the above technical problems, the present application provides a table recognition method and electronic device. The technical solution provided by the present application automatically identifies the starting and ending positions of table rows and columns through text projection and a state machine, thereby improving the accuracy of table recognition.
[0007] In order to achieve the above technical objectives, this application provides the following technical solutions:
[0008] In a first aspect, a table recognition method is provided, which is applied to a first device. The method includes: obtaining a text row area corresponding to a table area in an image. Based on the text row area, a binary image is generated. The pixel values of the pixels in a single row of the binary image are accumulated to obtain a row projection corresponding to the binary image; and the pixel values of the pixels in a single column of the binary image are accumulated to obtain a column projection corresponding to the binary image. The row projection and the column projection are scanned by a state machine to determine the starting position and ending position of the row and the starting position and ending position of the column. The table position is determined based on the starting position and ending position of the row and the starting position and ending position of the column.
[0009] In this way, the electronic device recognizes the table structure based on text projection and the state machine, so that the table recognition method has high robustness and avoids interference from interference factors.
[0010] Moreover, compared with table structure recognition based on neural networks, the table recognition method provided in the embodiments of the present application can be applied to more table recognition scenarios, such as wireless table structure recognition scenarios of various styles, without having to pre-build data sets suitable for these scenarios to train neural networks, thereby saving neural network training time and training set annotation costs.
[0011] In addition, the table recognition method provided in the embodiment of the present application can recognize and return the table structure in real time, meeting the user's usage needs with a higher processing speed.
[0012] According to a first aspect, generating a binary image based on a text line region includes: scaling the text line region according to a preset condition, and generating a binary image based on the scaled text line region.
[0013] In this way, based on the standardized text height, the difficulty of setting the counting threshold in the above-mentioned state transition conditions and direction process is reduced.
[0014] For example, the preset condition indicates a preset interval for scaling text line areas in an electronic device. Generally, the preset interval is smaller than the text line height. Therefore, scaling the text line area is equivalent to downsampling the image, reducing the number of sampling points during image recognition. This effectively reduces algorithm processing steps and time during subsequent table recognition, improving table recognition speed.
[0015] According to the first aspect, or any implementation of the first aspect above, generating a binary image based on a text line region includes: correcting a tilt angle of the text line region, and generating a binary image based on the tilt-corrected text line region.
[0016] In some scenarios, tables in the original image may be tilted due to factors such as shooting angle. Therefore, after determining the text row area in the original image, the electronic device can first correct the tilt angle of the text row area. The electronic device then determines the corresponding table structure based on the tilt-corrected text row area. This enables the recognition of tilted tables and expands the applicable table recognition scenarios.
[0017] According to the first aspect, or any implementation of the first aspect above, the state machine includes a ready state and an activated state, the ready state is used to indicate scanning a new column or row, and the activated state is used to indicate that a certain column or row is being scanned; the ready state or the activated state includes two transfer directions, namely maintaining the current state or transferring to the next state.
[0018] According to the first aspect, or any implementation of the first aspect above, the ready state or the active state includes multiple sub-states.
[0019] Optionally, maintaining the current state in the transition direction of the ready state or the active state is, for example, transitioning from the current sub-state to the next sub-state.
[0020] According to the first aspect, or any implementation method of the first aspect above, the row projection and the column projection are scanned by a state machine to determine the starting position and the ending position of the row and the starting position and the ending position of the column, including: determining the transfer direction or transfer condition of the ready state or the activated state through one or more of the counting threshold, the counting situation, the signal value, the buffer value, and the fluctuation threshold, the counting situation is used to indicate the number of sub-states transferred within the ready state or the activated state, and the number is less than or equal to the counting threshold, and the buffer is used to convert the input signal.
[0021] The counting threshold may be determined according to the character width, for example, the counting threshold may be half a character width, a full character width, or the like.
[0022] Optionally, a single state machine scans the row and column projections to determine the starting and ending positions of the rows and columns. The input transition method for the ready state determines the transition condition and direction of the ready state by determining the relationship between the signal and the sum of the buffer minimum value and the fluctuation threshold, as well as the count. The input transition method for the active state determines the transition condition and direction of the active state by determining the difference between the signal and the buffer maximum value and the fluctuation threshold, as well as the count.
[0023] Optionally, the forward state machine scans the row projection and the column projection to determine the starting position of the row and the starting position of the column. The input conversion method of the ready state is to determine the transfer condition and direction of the ready state by judging the relationship between the signal and the sum of the minimum value of the buffer and the fluctuation threshold, as well as the counting situation. The forward state machine does not need to determine the end position of the column, so the activation state does not need to convert the input, and determines to switch to the end state or the next sub-state of the activation state based on whether it is in the last sub-state of the activation state. The input conversion method of the end state is to determine the transfer condition and direction of the end state by judging the size of the difference between the signal and the maximum value of the buffer and the fluctuation threshold, as well as the counting situation.
[0024] Optionally, the reverse state machine scans the row and column projections to determine the row and column end positions. The input transition method for the ready state determines the transition condition and direction of the ready state by determining the relationship between the signal and the sum of the buffer minimum value and the fluctuation threshold, as well as the count. Because the reverse state machine only detects one end position per scan, the algorithm stops once the reverse state machine enters the active state. Therefore, there is no transition to the active state.
[0025] According to the first aspect, or any implementation of the first aspect above, scanning row projections and column projections by a state machine to determine the starting position and ending position of a row and the starting position and ending position of a column includes: scanning the column projections horizontally by a state machine, and during the scanning process, determining the starting position and ending position of each column according to the state transition of the state machine, wherein the position corresponding to the transition of the state machine from the ready state to the active state is determined to be the starting position of the nth column, and the ending position of the nth column is the position corresponding to the transition of the state machine from the active state to the ready state closest to the starting position of the nth column, where n is a positive integer. Scanning the row projections vertically by a state machine, and during the scanning process, determining the starting position and ending position of each row according to the state transition of the state machine, wherein the position corresponding to the transition of the state machine from the ready state to the active state is determined to be the starting position of the mth row, and the ending position of the mth row is the position corresponding to the transition of the state machine from the active state to the ready state closest to the starting position of the mth row, where m is a positive integer.
[0026] For example, the device uses a state machine to scan column projections horizontally. The state machine transitions from the ready state to the active state, meaning it transitions from preparing to scan a new column to scanning a column. The corresponding position is the start position of column n. The state machine then continues scanning the current column n, remaining in the active state. The state machine then transitions from the active state to the ready state (this transition occurs at a position closest to the start position of column n, determined by continuous scanning). This transition occurs from scanning a column to scanning the next new column. The corresponding position is the end position of column n. The state machine then transitions from the current ready state to the active state, meaning it transitions from preparing to scan the next new column to scanning the next new column. The corresponding position is the start position of column n+1. In other words, the state machine continuously scans column projections horizontally in one direction (e.g., from left to right or right to left), looping through the ready and active state transitions to determine the start and end positions of multiple columns in a table.
[0027] For example, the device uses a state machine to scan row projections vertically. The state machine transitions from the ready state to the active state, meaning it transitions from the ready-to-scan state to the currently scanning state. The corresponding position is the start position of row m. The state machine then continues scanning the current row m, remaining in the active state. The state machine then transitions from the active state to the ready state (this transition occurs at a position closest to the start position of row m, determined by continuous scanning). This transition occurs from the currently scanning state to the state of scanning the next row. The corresponding position is the end position of row m. The state machine then transitions from the current ready state to the active state, meaning it transitions from the ready-to-scan state to the currently scanning state. The corresponding position is the start position of row m+1. In other words, the state machine continuously scans row projections vertically in one direction (e.g., from top to bottom or bottom to top), looping through the ready-to-active and active state transitions to determine the start and end positions of multiple rows in a table.
[0028] According to the first aspect, or any implementation of the first aspect, the state machine includes a forward state machine and a reverse state machine, the forward state machine includes a ready state, an active state, and an end state, and the reverse state machine includes a ready state and an active state. The end state is used to indicate that scanning of the previous row or column has ended prematurely and that scanning of the next row or column is not yet ready.
[0029] According to the first aspect, or any implementation of the first aspect above, a state machine is used to scan row projections and column projections to determine the starting position and ending position of rows and the starting position and ending position of columns, including: scanning the column projections horizontally by a forward state machine, and during the scanning process, determining the starting position of each column according to the state transition of the forward state machine, wherein the position corresponding to the transition of the forward state machine from the ready state to the active state is determined to be the starting position of the nth column, and then the forward state machine is transferred from the active state to the end state and then to the ready state, and it is determined to be ready to scan the n+1th column, where n is a positive integer. Scanning the column projections horizontally by a reverse state machine in a direction opposite to the scanning direction of the forward state machine, and during the scanning process, determining the ending position of the nth column corresponding to the starting position of the nth column according to the state transition of the reverse state machine, wherein scanning is started from the starting position of the n+1th column as the starting point and the position corresponding to the transition of the reverse state machine from the ready state to the active state is determined to be the ending position of the nth column.
[0030] For example, the device scans column projections horizontally using a forward state machine. The forward state machine transitions from a ready state to an active state, meaning it is ready to scan a new column and is currently scanning a column. The corresponding position is the starting position of column n. The forward state machine then continues scanning the current column n. The forward state machine transitions from an active state to an end state, meaning it has prematurely completed scanning column n but is not yet ready to scan column n+1. The forward state machine then continues scanning, transitioning from an end state to an active state, meaning it is not ready to scan column n+1 but is currently scanning column n+1. The corresponding position is the starting position of column n+1. In other words, the forward state machine continuously scans column projections horizontally in one direction (e.g., from left to right or right to left), looping through the ready, active, and end state transitions to determine the starting positions of multiple columns in a table.
[0031] After determining the starting positions of multiple columns, the device can use the reverse state machine to scan the column projections horizontally in a direction opposite to the scanning direction of the forward state machine to determine the end positions corresponding to the starting positions of these determined columns. For example, take the scanning direction of the forward state machine as from left to right and the scanning direction of the reverse state machine as from right to left. In the process of determining the end position of the nth column, the device first obtains the starting position of the nth column, determines the first starting position to the right of the starting position of the nth column, that is, determines the starting position of the n+1th column, and starts scanning to the left with the starting position of the n+1th column as the starting point. Determine that the reverse state machine has transitioned from the ready state to the active state, that is, the reverse state machine has transitioned from the state of preparing to scan a new column to the state of scanning a column, then the corresponding position is the end position of the nth column. After the reverse state machine switches to the active state, the device ends the scan of the reverse state machine. That is, the reverse state machine scans the column projection multiple times in the horizontal direction opposite to the scanning direction of the forward state machine, starting from the starting position of the column that has been determined by the forward state machine, and loops the above-mentioned transition process from the ready state to the active state, so as to determine the end positions of multiple columns in the table respectively.
[0032] According to the first aspect, or any implementation of the first aspect above, the row projection and the column projection are scanned by a state machine to determine the starting position and the ending position of the row and the starting position and the ending position of the column, including: scanning the row projection along the vertical direction by the forward state machine, and during the scanning process, determining the starting position of each row according to the state transition of the forward state machine, wherein the position corresponding to the transition of the forward state machine from the ready state to the active state is determined to be the starting position of the mth row, and then the forward state machine transitions from the active state to the end state and then to the ready state, and determines to be ready to scan the m+1th row, where m is a positive integer. Scanning the row projection along the vertical direction by the reverse state machine in a direction opposite to the scanning direction of the forward state machine, and during the scanning process, determining the ending position of the mth row corresponding to the starting position of the mth row according to the state transition of the reverse state machine, wherein scanning starts from the starting position of the m+1th row as the starting point and determines the position corresponding to the transition of the reverse state machine from the ready state to the active state is the ending position of the mth row.
[0033] For example, the device uses the forward state machine to scan row projections vertically. The forward state machine transitions from the ready state to the active state, meaning it transitions from preparing to scan a new row to scanning a row. The corresponding position is the start position of row m. The forward state machine then continues scanning the current row m. The forward state machine transitions from the active state to the end state, meaning it transitions from scanning row m to having prematurely completed scanning row m but not yet ready to scan row m+1. The forward state machine then continues scanning, transitioning from the end state to the active state, meaning it transitions from not yet ready to scan row m+1 to scanning row m+1. The corresponding position is the start position of row m+1. In other words, the forward state machine continuously scans row projections vertically in one direction (e.g., from top to bottom or bottom to top), looping through the ready state, active state, and end state transitions to determine the start positions of multiple rows in the table.
[0034] After determining the starting positions of multiple rows, the device can use the reverse state machine to scan the row projections in the vertical direction, in the opposite direction to the scanning direction of the forward state machine, to determine the end positions corresponding to the start positions of these determined rows. For example, the scanning direction of the forward state machine is from top to bottom, and the scanning direction of the reverse state machine is from bottom to top. In the process of determining the end position of the mth row, the device first obtains the starting position of the mth row, determines the first starting position below the starting position of the mth row, that is, determines the starting position of the m+1th row, and starts scanning upward with the starting position of the m+1th row as the starting point. Determine that the reverse state machine has transitioned from the ready state to the active state, that is, the reverse state machine has transitioned from the state of preparing to scan a new row to the state of scanning a row, then the corresponding position is the end position of the mth row. After the reverse state machine switches to the active state, the device ends the scan of the reverse state machine. That is, the reverse state machine scans the row projection multiple times in the vertical direction opposite to the scanning direction of the forward state machine, starting from the starting position of the row that has been determined by the forward state machine, and loops the above-mentioned transition process from the ready state to the active state, thereby determining the end positions of multiple rows in the table respectively.
[0035] Optionally, since the forward state machine does not need to detect the end position of the column or row, the activation state of the forward state machine ends early, and the end state of the forward state machine is actually a special state separated from the activation state of the single state machine.
[0036] In this way, when the forward state machine only detects the starting position, the counting threshold can be set more flexibly, so as to enter and exit each state more quickly, improve the sensitivity of detection, and facilitate the processing of narrow blank spaces or rows and columns with less text changes.
[0037] According to the first aspect, or any implementation of the first aspect above, determining the position of a table based on the starting and ending positions of a row and the starting and ending positions of a column includes: determining row and column separation positions based on the starting and ending positions of the row and the starting and ending positions of the column, respectively; and determining coordinates of cells included in the table based on the row and column separation positions.
[0038] According to the first aspect, or any implementation of the first aspect above, a binary image is generated based on the text line area, including: setting the pixel value within the text line area to 1, and setting the pixel value of the area outside the text line area to 0.
[0039] In this way, after the original image is converted into a binary image, the row projection and the column projection are determined based on the binary image, which can better process images with a colored background.
[0040] In addition, interference factors such as the background in the original image, text outside the table, shooting angle, and lighting will affect the recognition of table structure. Determining row and column projections based on binary images rather than determining projections based on original pixel values can effectively eliminate the above interference factors and improve the robustness of table structure recognition.
[0041] According to the first aspect, or any implementation of the first aspect above, scanning the row projections and column projections by a state machine to determine the starting and ending positions of the row and the starting and ending positions of the column includes normalizing the row projections and the column projections, respectively. Scanning the normalized row projections and column projections by the state machine to determine the starting and ending positions of the row and the starting and ending positions of the column.
[0042] In this way, the electronic device scans the row projection and column projection based on the state machine to determine the starting and ending positions of the rows and columns respectively, and can eliminate some noise interference caused by factors such as text detection errors and image shooting angles.
[0043] In addition, in the process of determining the starting and ending positions of rows and columns, the row projection and column projection are normalized so that the projection threshold is a fluctuating threshold rather than a fixed threshold. This can effectively eliminate the influence of cross-cell text (such as merged cells), making the scanning method of the state machine applicable to recognition scenarios of various table structures.
[0044] In a second aspect, an electronic device is provided. The electronic device includes: a processor and a memory, the memory is coupled to the processor, the memory is used to store computer program code, the computer program code includes computer instructions, and when the processor reads the computer instructions from the memory, the electronic device executes: obtaining a text row area corresponding to a table area in an image. Based on the text row area, a binary image is generated. The pixel values of the pixel points in a single row of the binary image are accumulated to obtain a row projection corresponding to the binary image; and the pixel values of the pixel points in a single column of the binary image are accumulated to obtain a column projection corresponding to the binary image. The row projection and the column projection are scanned by a state machine to determine the starting position and the ending position of the row and the starting position and the ending position of the column. The table position is determined based on the starting position and the ending position of the row and the starting position and the ending position of the column.
[0045] According to a second aspect, generating a binary image based on a text line region includes: scaling the text line region according to a preset condition, and generating a binary image based on the scaled text line region.
[0046] According to the second aspect, or any implementation of the second aspect above, generating a binary image based on the text line region includes: correcting the tilt angle of the text line region. Generating a binary image based on the tilt-corrected text line region.
[0047] According to the second aspect, or any implementation of the second aspect above, the state machine includes a ready state and an activated state, the ready state is used to indicate scanning a new column or row, and the activated state is used to indicate that a certain column or row is being scanned; the ready state or the activated state includes two transfer directions, namely maintaining the current state or transferring to the next state.
[0048] According to the second aspect, or any implementation of the second aspect, the ready state or the active state includes multiple sub-states.
[0049] According to the second aspect, or any implementation method of the second aspect above, the row projection and the column projection are scanned by a state machine to determine the starting position and the ending position of the row and the starting position and the ending position of the column, including: determining the transfer direction or transfer condition of the ready state or the activated state through one or more of the counting threshold, the counting situation, the signal value, the buffer value, and the fluctuation threshold, the counting situation is used to indicate the number of sub-states transferred within the ready state or the activated state, and the number is less than or equal to the counting threshold, and the buffer is used to convert the input signal.
[0050] According to the second aspect, or any implementation of the second aspect above, scanning row projections and column projections by a state machine to determine the starting position and ending position of rows and the starting position and ending position of columns includes: scanning the column projections horizontally by a state machine, and during the scanning process, determining the starting position and ending position of each column according to the state transition of the state machine, wherein the position corresponding to the transition of the state machine from the ready state to the active state is determined to be the starting position of the nth column, and the ending position of the nth column is the position corresponding to the transition of the state machine from the active state to the ready state closest to the starting position of the nth column, where n is a positive integer. Scanning the row projections vertically by a state machine, and during the scanning process, determining the starting position and ending position of each row according to the state transition of the state machine, wherein the position corresponding to the transition of the state machine from the ready state to the active state is determined to be the starting position of the mth row, and the ending position of the mth row is the position corresponding to the transition of the state machine from the active state to the ready state closest to the starting position of the mth row, where m is a positive integer.
[0051] According to the second aspect, or any implementation of the second aspect, the state machine includes a forward state machine and a reverse state machine, the forward state machine includes a ready state, an active state, and an end state, and the reverse state machine includes a ready state and an active state. The end state is used to indicate that scanning of the previous row or column has ended prematurely and that scanning of the next row or column is not yet ready.
[0052] According to the second aspect, or any implementation of the second aspect above, a state machine is used to scan row projections and column projections to determine the starting position and ending position of rows and the starting position and ending position of columns, including: scanning the column projections horizontally by a forward state machine, and during the scanning process, determining the starting position of each column according to the state transition of the forward state machine, wherein the position corresponding to the transition of the forward state machine from the ready state to the active state is determined to be the starting position of the nth column, and then the forward state machine is transferred from the active state to the end state and then to the ready state, and it is determined to be ready to scan the n+1th column, where n is a positive integer. Scanning the column projections horizontally by a reverse state machine in a direction opposite to the scanning direction of the forward state machine, and during the scanning process, determining the ending position of the nth column corresponding to the starting position of the nth column according to the state transition of the reverse state machine, wherein scanning is started from the starting position of the n+1th column as the starting point and the position corresponding to the transition of the reverse state machine from the ready state to the active state is determined to be the ending position of the nth column.
[0053] According to the second aspect, or any implementation of the second aspect above, the row projection and the column projection are scanned by a state machine to determine the starting position and the ending position of the row and the starting position and the ending position of the column, including: scanning the row projection along the vertical direction by a forward state machine, and during the scanning process, determining the starting position of each row according to the state transition of the forward state machine, wherein the position corresponding to the transition of the forward state machine from the ready state to the active state is determined to be the starting position of the mth row, and then the forward state machine transitions from the active state to the end state and then to the ready state, and determines to be ready to scan the m+1th row, where m is a positive integer. Scanning the row projection along the vertical direction in a direction opposite to the scanning direction of the forward state machine by a reverse state machine, and during the scanning process, determining the ending position of the mth row corresponding to the starting position of the mth row according to the state transition of the reverse state machine, wherein scanning starts from the starting position of the m+1th row as the starting point and determines the position corresponding to the transition of the reverse state machine from the ready state to the active state is the ending position of the mth row.
[0054] According to the second aspect, or any implementation of the second aspect above, determining the position of the table based on the starting and ending positions of the row and the starting and ending positions of the column includes: determining the row separator position and the column separator position based on the starting and ending positions of the row and the starting and ending positions of the column, respectively; and determining the coordinates of cells included in the table based on the row separator position and the column separator position.
[0055] According to the second aspect, or any implementation of the second aspect above, a binary image is generated based on the text line area, including: setting the pixel value within the text line area to 1, and setting the pixel value of the area outside the text line area to 0.
[0056] According to the second aspect, or any implementation of the second aspect above, scanning the row projections and column projections by a state machine to determine the starting and ending positions of the row and the starting and ending positions of the column includes normalizing the row projections and the column projections, respectively. Scanning the normalized row projections and column projections by the state machine to determine the starting and ending positions of the row and the starting and ending positions of the column.
[0057] The technical effects corresponding to the second aspect and any implementation method of the second aspect can be found in the technical effects corresponding to the above-mentioned first aspect and any implementation method of the first aspect, and will not be repeated here.
[0058] In a third aspect, a server is provided. The server includes a processor and a memory, the memory being coupled to the processor and configured to store computer program code. The computer program code includes computer instructions. When the processor reads the computer instructions from the memory, the server executes the table recognition method described in the first aspect and any possible implementation thereof.
[0059] The technical effects corresponding to the third aspect and any implementation method of the third aspect can be found in the technical effects corresponding to the above-mentioned first aspect and any implementation method of the first aspect, and will not be repeated here.
[0060] In a fourth aspect, an electronic device is provided. The electronic device has the function of implementing the table recognition method described in the first aspect and any possible implementation thereof. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions.
[0061] The technical effects corresponding to the fourth aspect and any implementation method of the fourth aspect can be found in the technical effects corresponding to the above-mentioned first aspect and any implementation method of the first aspect, and will not be repeated here.
[0062] In a fifth aspect, a server is provided. The server has the function of implementing the table recognition method described in the first aspect and any possible implementation thereof. The function can be implemented by hardware, or by hardware executing corresponding software implementation. The hardware or software includes one or more modules corresponding to the above-mentioned functions.
[0063] The technical effects corresponding to the fifth aspect and any implementation method of the fifth aspect can be found in the technical effects corresponding to the above-mentioned first aspect and any implementation method of the first aspect, and will not be repeated here.
[0064] In a sixth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program (also referred to as instructions or code). When the computer program is executed by a first device, the first device performs the method of the first aspect or any one of the embodiments of the first aspect. Optionally, the first device is an electronic device or a server.
[0065] The technical effects corresponding to the sixth aspect and any implementation method of the sixth aspect can be found in the technical effects corresponding to the above-mentioned first aspect and any implementation method of the first aspect, and will not be repeated here.
[0066] In a seventh aspect, a computer program product is provided. When the computer program product is executed on a first device, the first device executes the method of the first aspect or any one of the embodiments of the first aspect. Optionally, the first device is an electronic device or a server.
[0067] The technical effects corresponding to the seventh aspect and any implementation method of the seventh aspect can be found in the technical effects corresponding to the above-mentioned first aspect and any implementation method of the first aspect, and will not be repeated here.
[0068] In an eighth aspect, a circuit system is provided, wherein the circuit system includes a processing circuit configured to execute the method of the first aspect or any one of the embodiments of the first aspect.
[0069] The technical effects corresponding to the eighth aspect and any one of the implementation methods of the eighth aspect can be found in the technical effects corresponding to the above-mentioned first aspect and any one of the implementation methods of the first aspect, and will not be repeated here.
[0070] In a ninth aspect, a chip system is provided. The chip system includes at least one processor and at least one interface circuit, wherein the at least one interface circuit is configured to perform transceiver functions and send instructions to the at least one processor. When the at least one processor executes the instructions, the at least one processor performs the method of the first aspect or any one of the embodiments of the first aspect.
[0071] The technical effects corresponding to the ninth aspect and any implementation method of the ninth aspect can be found in the technical effects corresponding to the above-mentioned first aspect and any implementation method of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0072] Figure 1 is a schematic diagram of the provided table recognition scenario;
[0073] FIG2 is a schematic diagram of a neural network structure provided;
[0074] FIG3 is a schematic diagram of the positional relationship of the blank areas provided;
[0075] FIG4 is a schematic diagram of the hardware structure of the device provided in an embodiment of the present application;
[0076] FIG5 is a flowchart of a table recognition method according to an embodiment of the present application;
[0077] FIG6 is a schematic diagram of the starting and ending positions of columns or rows provided in an embodiment of the present application;
[0078] FIG7 is a schematic diagram of a state machine state transition according to an embodiment of the present application;
[0079] FIG8 is a schematic diagram of sub-state transitions within a state machine according to an embodiment of the present application;
[0080] FIG9 is a schematic diagram of a forward state machine state transition according to an embodiment of the present application;
[0081] FIG10 is a schematic diagram of a reverse state machine state transition according to an embodiment of the present application;
[0082] FIG11 is a diagram illustrating the recognition effect of the starting position and the ending position of a row of a table provided in an embodiment of the present application;
[0083] FIG12 is a second flow chart of the table recognition method provided in an embodiment of the present application;
[0084] FIG13 is a third flow chart of the table recognition method provided in an embodiment of the present application;
[0085] FIG14 is a schematic diagram of the first device structure provided in an embodiment of the present application. DETAILED DESCRIPTION
[0086] The technical solutions in the embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. Among them, in the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to include expressions such as "one or more", unless there is a clear indication to the contrary in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one or more (including two).
[0087] References to "one embodiment" or "some embodiments" etc. described in this specification mean that one or more embodiments of the present application include the specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in another way. The term "connected" includes direct and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.
[0088] In the embodiments of this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a concrete manner.
[0089] With the widespread use of electronic documents, users often convert paper documents into electronic documents by scanning, photographing, or other methods to use and save them. Later, when using electronic documents, users can use technologies such as image processing and OCR to convert the original non-editable electronic documents into editable documents to meet their needs.
[0090] Exemplarily, as shown in FIG1(a), an electronic device obtains an uneditable document (such as an image), wherein the image includes a table. The electronic device performs table detection on the original image through image recognition technology; as shown in FIG1(b), the area where the table is located in the original image is determined. Thereafter, the electronic device performs text detection on the area where the table is located, and as shown in FIG1(c), the text row area in the area where the table is located in the original image is determined. Thereafter, as shown in FIG1(d), the table structure is identified based on the text row area in the table. Afterwards, after determining the table structure, the electronic device can match the identified text position with the table position to generate an editable document.
[0091] Among them, different table structure recognition methods may be required for table images of different styles. For example, different table structure recognition methods are usually used for wired (framed) tables and wireless (framed) tables. For example, for wired (framed) tables, the table line detection method of image semantic segmentation can be used to complete the recognition of the table structure. However, for scenarios with wireless (framed) tables or tables with missing lines (frames), since the cell boundaries are not clear, there are still many problems in the consistency of table structure recognition.
[0092] In some examples, a table structure recognition method using a neural network is used to recognize the table structure. As shown in FIG2 , this type of neural network generally has an “encoder-decoder” structure, wherein the encoder can obtain the visual features of the image after receiving the input image and send the visual features to the decoder. Afterwards, the decoder can output the table structure one by one in an auto-regressive manner based on the received visual features, thereby realizing the recognition of the table structure included in the electronic device. Similarly, the text in the cell is recognized by the neural network, and then the text can be matched with the recognized table.
[0093] Among them, this type of neural network including the "encoder-decoder" structure usually requires a large amount of manually annotated table data for training before use, and the collection and annotation of table structures are very difficult. Depending on the complexity of the table, the annotation time for a single table may be between several minutes and tens of minutes, so the cost of building a table structure recognition dataset for training neural networks is very high. The currently open source datasets cannot well cover the table structures in all real scenarios. If the input image recognized by the neural network includes a table structure that has not been covered in the dataset, or the table structure is relatively complex, it will cause serious errors in the output table structure. In addition, the reasoning process of the neural network to recognize the table structure requires more computing resources, and its reasoning time is often proportional to the number of cells included in the image, resulting in reduced efficiency in table structure recognition.
[0094] In other examples, the table structure is determined by identifying blank areas within the area where the table is located. For example, Figure 3 (a) shows an example of vertical overlap (i.e., at least partial horizontal alignment) between blank areas A and B. Figure 3 (b) shows an example of horizontal overlap (i.e., at least partial vertical alignment) between blank areas A and B. After detecting blank areas in an image, the vertically overlapping blank areas are grouped. Two blank areas are connected if and only if they overlap vertically, and the connected blank areas form a group. The top border of the topmost blank area in each group is defined as the top border of the table candidate. Similarly, the bottom border of the bottommost blank area is defined as the bottom border of the table candidate. A dynamic threshold is then used to filter the height of the blank areas, and blank areas with insufficient height are discarded. For the remaining blank areas, the text distributed between the blank areas is collected, and column separators are added to the right edge of the text. Similarly, row separators are added to each column separator. The text between the separators is assigned to cells.
[0095] However, the above solution may result in inaccurate detection frames for blank areas and content areas. A small number of falsely detected blank areas or content areas may result in the appearance of extra rows or columns. Inaccurate detection frames may also lead to errors in the determination of vertically overlapping blank areas, resulting in incorrect recognition of table structures. Furthermore, when converting electronic documents by photographing them, the electronic document image may rotate or shift, causing the detection frames for blank areas or content areas to rotate or shift as well. This significantly increases the error rate in determining vertically overlapping blank areas.
[0096] In some other examples, the table structure is identified by projection thresholds. For example, the image is converted into a grayscale image, and the pixel values of the grayscale image are accumulated in the vertical and horizontal directions to obtain row projections and column projections. Afterwards, the row projections and column projections are scanned respectively, and according to the size of the projection values, the intervals where the projection values are less than the threshold are identified as troughs, otherwise they are identified as peaks. For the peak and trough areas, a second scan is performed, and the threshold at this time is set to the average value of the projection values in the interval to obtain new peaks and troughs. Row separations and column separations are added to the troughs obtained by the second scan, thereby completing the identification of the table structure.
[0097] However, the above solution directly projects grayscale pixels, which doesn't work well with images with color backgrounds. Furthermore, a fixed projection threshold can lead to misrecognition, such as misidentifying rows or columns with less text as valleys or misidentifying projected noise as non-existent rows or columns. Furthermore, it can't effectively address issues like rotation and displacement in the image, which can still lead to increased recognition error rates.
[0098] Based on this, an embodiment of the present application provides a table recognition method, which automatically identifies the row and column separation of a table through text projection and a state machine, thereby improving the accuracy of table structure recognition.
[0099] Optionally, the table recognition method provided in the embodiment of the present application can be applied to the device 100. For example, the device 100 can be a terminal device such as a computer, a mobile phone, a computer, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR) device, a mixed reality (MR) device, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a wearable device, an artificial intelligence (AI) device, etc. The operating system installed on the above device 100 includes but is not limited to This application does not limit the specific type of the device 100 or the installed operating system.
[0100] Optionally, the device 100 may also be a device or server with computing capabilities such as a cloud server or a network server. The server may be a single server, a server cluster consisting of multiple servers, or a cloud computing service center.
[0101] Optionally, the table recognition method provided in the embodiments of the present application is primarily used in scenarios where table structures are extracted and restored in document digitization programs, such as converting non-editable documents in the original format of images, PDFs, etc. into editable documents. Optionally, the table recognition method provided in the embodiments of the present application can also be applied to other programs that rely on table recognition or document recognition, such as automated financial report analysis programs.
[0102] Exemplarily, as shown in Figure 4, the device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195, etc.
[0103] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the device 100. In other embodiments of the present application, the device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
[0104] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.
[0105] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.
[0106] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly retrieve it from the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.
[0107] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.
[0108] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, processor 110 may include multiple I2C busses. Processor 110 may be coupled to a touch sensor, a charger, a flash, a camera 193, and the like via different I2C bus interfaces. For example, processor 110 may be coupled to a touch sensor via an I2C interface, enabling communication between processor 110 and the touch sensor via the I2C bus interface, thereby enabling touch functionality of device 100.
[0109] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display 194 and the camera 193. MIPI interfaces include the camera serial interface (CSI) and the display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the camera function of the device 100. The processor 110 and the display 194 communicate via the DSI interface to implement the display function of the device 100.
[0110] USB interface 130 is an interface that complies with USB standards and specifications, and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. USB interface 130 can be used to connect a charger to charge device 100, or to transfer data between device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect to other first electronic devices, such as AR devices.
[0111] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is merely illustrative and does not constitute a structural limitation on the device 100. In other embodiments of the present application, the device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.
[0112] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the device 100. While charging the battery 142, the charging management module 140 can also provide power to the first electronic device via the power management module 141.
[0113] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 and provides power to the processor 110, the internal memory 121, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be provided in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be provided in the same device.
[0114] The wireless communication function of the device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.
[0115] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.
[0116] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied on the device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.
[0117] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs the sound signal through the audio device or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be provided in the same device as the mobile communication module 150 or other functional modules.
[0118] The wireless communication module 160 can provide wireless communication solutions applied to the device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.
[0119] In some embodiments, antenna 1 of device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that device 100 can communicate with a network and other devices via wireless communication technologies. Wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. GNSS may include the global positioning system (GPS), the global navigation satellite system (GLONASS), the Beidou navigation satellite system (BDS), the quasi-zenith satellite system (QZSS) and / or the satellite based augmentation system (SBAS).
[0120] Device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.
[0121] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be manufactured using a liquid crystal display (LCD), such as an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini-LED, a Micro-LED, a Micro-OLED, or a quantum dot light-emitting diode (QLED). In some embodiments, device 100 may include one or N display screens 194, where N is a positive integer greater than one.
[0122] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.
[0123] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.
[0124] The internal memory 121 can be used to store computer executable program code, and the executable program code includes instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.
[0125] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110. The device 100 can use the audio module 170 for functions such as music playback and recording. The audio module 170 may include a speaker, a receiver, a microphone, a headphone jack, and an application processor to implement audio functions.
[0126] The sensor module 180 may include a pressure sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.
[0127] Keys 190 include a power button, a volume button, and the like. Keys 190 may be mechanical keys or touch-sensitive keys. Device 100 may receive key inputs and generate key signal inputs related to user settings and function control of device 100.
[0128] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts and can also be used for touch vibration feedback.
[0129] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.
[0130] The SIM card interface 195 is used to connect a SIM card. The device 100 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1.
[0131] The following embodiment takes the device 100 as an electronic device and the non-editable electronic document as an image format document as an example to introduce in detail the table recognition method provided in the embodiment of the present application.
[0132] In some embodiments, a user may obtain an image containing a table by scanning, taking a photo, or other means. These images may be in RGB format or any other image format. In some examples, the electronic device, in response to a user operation, determines to convert these non-editable images into editable documents. During the conversion process, the electronic device needs to recognize the table included in the image, such as determining the table structure and matching the text in the table with the corresponding cells.
[0133] For example, Figure 5 is a flow chart of a table recognition method provided in an embodiment of the present application. As shown in Figure 5, the method includes the following process.
[0134] S501: The electronic device obtains a text row area corresponding to a table area in an original image.
[0135] The original image is a non-editable electronic document, and the user needs to convert the original image into an editable electronic document, so the electronic device needs to recognize the table included therein.
[0136] In some embodiments, the electronic device processes the original image in a preset manner to determine the table area and text row area included in the original image. The electronic device then matches the table area with the text row area to determine the text row area within the table area, i.e., the text row area corresponding to the table. Alternatively, the electronic device first determines the table area included in the original image, and then determines the text row area included in the table area.
[0137] In the following steps, the electronic device scans the projection value of the binary image corresponding to the text row area in the table area through the state machine to determine the table structure. The text row area below is used to indicate the text row area in the table area, which will not be described in detail below.
[0138] Optionally, the preset method includes, for example, processing the original image using an image recognition algorithm, a neural network, or the like. For example, the electronic device determines the table area using an algorithm such as YOLO (You Only Look Once). The electronic device determines the text line area using an algorithm such as DBNet (Differentiable Binarization Networks).
[0139] For example, an electronic device captures an original image as shown in FIG1(a), then performs table detection on the original image; as shown in FIG1(b), the electronic device determines the location of the table in the original image. The electronic device then performs text detection on the original image, as shown in FIG1(c), to determine the location of text lines in the original image, that is, to determine the text line area within the area where the table is located in the original image.
[0140] Optionally, during the above process, the electronic device may also perform text recognition on the original image, as shown in FIG1(c), to determine the content of the text line in the original image. The electronic device may use optical character recognition (OCR) technology to recognize the content of the text line in the original image. For example, the electronic device may use a CRNN-CTC (Convolutional Recurrent Neural Networks with Connectionist Temporal Classification) algorithm to recognize the content of the text line.
[0141] S502: The electronic device generates a binary image corresponding to the original image based on the text line area.
[0142] In some embodiments, after determining the text line area, the electronic device can use a binarization algorithm to set the pixel value of the text line area to 1 and the pixel value of other areas outside the text line area to 0, thereby generating a binarized image corresponding to the original image.
[0143] S503: The electronic device determines a row projection and a column projection based on the binarized image.
[0144] In some embodiments, after converting the original image into a binary image, the electronic device accumulates the pixel values of the pixels in the i-th row of the binary image to the row projection of the i-th row to obtain a set of one-dimensional signals, and accumulates the pixel values of the pixels in the j-th column of the binary image to the column projection of the j-th column to obtain another set of one-dimensional signals, where i and j are positive integers.
[0145] Optionally, after acquiring the two sets of one-dimensional signals, the electronic device normalizes the two sets of one-dimensional signals to eliminate adverse effects caused by singular sample data, thereby determining row and column projections for subsequent identification of the table structure. For example, the electronic device normalizes the value ranges of the two sets of one-dimensional signals to the interval [0, 1] by dividing each of the two sets of one-dimensional signals by the corresponding maximum value.
[0146] In this way, after the original image is converted into a binary image, the row projection and the column projection are determined based on the binary image, which can better process images with a colored background.
[0147] In addition, interference factors such as the background in the original image, text outside the table, shooting angle, and lighting will affect the recognition of table structure. Determining row and column projections based on binary images rather than determining projections based on original pixel values can effectively eliminate the above interference factors and improve the robustness of table structure recognition.
[0148] S504: The electronic device scans the row projection and the column projection through the state machine to determine the starting position and the ending position of the row and the column respectively.
[0149] In some embodiments, after determining the row projection and the column projection, the electronic device scans the row projection and the column projection respectively through a state machine to determine the starting position and the ending position of the row and the column.
[0150] For example, as shown in FIG6 , a partial table area, each column or row includes a starting position and an ending position. After the electronic device determines the starting position and the ending position of the column or row through the state machine, it can determine the separation position (such as the median position) between the previous column (such as the nth column) or row (such as the mth row) and the starting position of the current column (such as the n+1th column) or row (such as the m+1th row) as the corresponding two cell boundaries, thereby realizing the recognition of the table structure. The following is a detailed description of the process by which the electronic device determines the starting position and the ending position of the row and column by scanning the row projection and the column projection through the state machine.
[0151] Optionally, the state machine is, for example, a finite-state automaton (FSA). For example, as shown in FIG7 , the state machine includes two states, such as a ready state (Pre) and an active state (Active). The ready state indicates that a new column (or row) is ready to be scanned, and the active state indicates that a column (or row) is currently being scanned. Each state has two possible transition directions: maintaining the current state or transitioning to the next state.
[0152] Optionally, when the state machine is looping internally in each state, it can be subdivided into a group of sub-states based on the count, and when the count is increased by one, it transfers to the next sub-state, and when the count is reset to zero, it returns to the first sub-state. For example, a schematic diagram of the sub-state cycle inside the ready state is shown in Figure 8. Among them, for the ready state and the activated state, the electronic device can be preset with corresponding counting thresholds respectively. After the count reaches the counting threshold, it can be determined that the state machine has transferred to the last sub-state in the current corresponding state, and it is necessary to switch to another state, the count of the other state is reset to zero, and the state machine is restarted in the first sub-state of the current state. For example, the electronic device determines that the state machine has transferred to the last sub-state of the ready state and needs to switch the state machine to the activated state.
[0153] Optionally, in addition to maintaining the state of the state machine, the electronic device also needs to maintain a buffer for converting input signals. If the state machine transitions from a ready state to an active state, the buffer needs to be updated.
[0154] In some examples, when the electronic device starts scanning, the state machine is in the ready state (count cnt=0) and the buffer buf={0}. Each time an input signal v is processed i Based on the output signal, the state machine state change is determined. The electronic device is pre-set with a signal fluctuation threshold t, a counting threshold n1 corresponding to the ready state, and a counting threshold n2 corresponding to the active state. The counting threshold can be determined based on the character width, for example, the counting threshold can be half a character width, a full character width, or the like.
[0155] Optionally, the electronic device may first scan the column projection through a state machine to determine the starting position and ending position of the column, and then scan the row projection through a state machine to determine the starting position and ending position of the row; or the electronic device may first scan the row projection through a state machine to determine the starting position and ending position of the row, and then scan the column projection through a state machine to determine the starting position and ending position of the column; or the electronic device may also scan the column projection and the column projection simultaneously through a state machine to determine the starting position and ending position of the row and column, respectively. This embodiment of the present application is not limited to this.
[0156] The following describes in detail the process of determining the starting position and the ending position by taking the example of an electronic device scanning a column projection through a state machine to determine the starting position and the ending position of the column.
[0157] Optionally, the input conversion method of the ready state is to determine the transition condition and direction of the ready state by judging the relationship between the signal and the sum of the buffer minimum value and the fluctuation threshold, as well as the counting situation. For example, the signal is greater than or equal to the buffer minimum value plus the fluctuation threshold (v i ≥min(buf)+t) i =1, otherwise c i =0, the corresponding feasible transfer condition and direction are as follows:
[0158] a) If the signal is greater than or equal to the buffer minimum plus the fluctuation threshold, and the current state is in the last sub-state of the ready state, then the count is cleared and the state machine can transfer to the active state to reset and update the buffer. For example, if c i =1, cnt=n1, then state=Active, cnt=0, buf={v i}.
[0159] b) If the signal is greater than or equal to the buffer minimum plus the fluctuation threshold, and is not in the last sub-state of the ready state, then the count is increased by one, the state machine transitions to the next sub-state of the ready state, and updates the buffer. For example, if c i =1, cnt<n1, then cnt=cnt+1, buf=buf∪{v i}.
[0160] c) If the signal is less than the minimum value of the buffer plus the fluctuation threshold, the count is reset to zero, the state machine transfers to the first sub-state of the ready state, and updates the buffer. For example, if c i =0, then cnt = 0, buf = buf∪{v i}.
[0161] Optionally, the input conversion method of the activation state is to determine the transfer condition and direction of the activation state by judging the difference between the signal and the buffer maximum value and the fluctuation threshold, as well as the counting situation. For example, the signal is less than or equal to the buffer maximum value minus the fluctuation threshold (v i ≤max(buf)-t) i =1, otherwise c i =0, the corresponding feasible transfer condition and direction are as follows:
[0162] a) If the signal is less than or equal to the maximum value of the buffer minus the fluctuation threshold, and the current state is the last sub-state of the active state, then the count is cleared and the state machine can transfer to the ready state to reset and update the buffer. For example, if c i =1, cnt=n2, then state=Pre, cnt=0, buf={v i}.
[0163] b) If the signal is less than or equal to the buffer maximum value minus the fluctuation threshold, and is not in the last substate of the active state, then the count is incremented by one, the state machine can transition to the next substate of the active state, and update the buffer. For example, if c i =1, cnt<n2, then cnt=cnt+1, buf=buf∪{v i}.
[0164] c) If the signal is greater than the maximum value of the buffer minus the fluctuation threshold, the count is cleared and the state machine can transition to the first substate of the active state and update the buffer. For example, if c i =0, then cnt = 0, buf = buf∪{v i}.
[0165] Based on the transition conditions and directions between the ready and active states in the above example, the electronic device scans a set of one-dimensional signals horizontally in the binary image from left to right, thereby detecting the starting and ending positions of the column based on the state machine. It should be understood that the electronic device can also scan a set of one-dimensional signals horizontally in the binary image from right to left, thereby detecting the starting and ending positions of the column based on the state machine.
[0166] For example, an electronic device scans column projections horizontally using a state machine and determines that the state machine transitions from a ready state to an active state, that is, the state machine transitions from a state of preparing to scan a new column to a state of scanning a column. As shown in FIG6 , the corresponding position is the starting position of the nth column. The state machine then continues to scan the current nth column, with the state machine remaining in the active state. The state machine then transitions from the active state to a ready state, that is, the state machine transitions from a state of scanning a column to a state of scanning the next new column. As shown in FIG6 , the corresponding position is the ending position of the nth column. The state machine then transitions from the current ready state to an active state, that is, the state machine transitions from a state of preparing to scan the next new column to a state of scanning a new column. As shown in FIG6 , the corresponding position is the starting position of the n+1th column. That is, the state machine continuously scans column projections horizontally in one direction (e.g., from left to right or from right to left), looping through the above-described ready and active state transitions to determine the starting and ending positions of multiple columns in a table. Where n is a positive integer.
[0167] Similarly, the electronic device scans a set of one-dimensional signals in the vertical direction of the binary image from top to bottom (or from bottom to top), thereby realizing the starting position and ending position of the row detection based on the state machine. The specific method can refer to the transfer conditions and directions in the scanning column projection process of the above example, which will not be repeated here.
[0168] For example, an electronic device uses a state machine to scan a row projection in a vertical direction, and determines that the state machine transitions from a ready state to an active state. That is, the state machine transitions from a state of preparing to scan a new row to a state of scanning a row. In this case, as shown in FIG6 , the corresponding position is the starting position of the mth row. The state machine then continues to scan the current mth row, while the state machine remains in the active state. Thereafter, the state machine transitions from the active state to a ready state. That is, the state machine transitions from a state of scanning a row to a state of scanning the next new row. In this case, as shown in FIG6 , the corresponding position is the ending position of the mth row. Thereafter, the state machine transitions from the current ready state to an active state. That is, the state machine transitions from a state of preparing to scan the next new row to a state of scanning a new row. In this case, as shown in FIG6 , the corresponding position is the starting position of the m+1th row. That is, the state machine continuously scans the row projection in a vertical direction (e.g., from top to bottom or from bottom to top), looping through the above-mentioned transition process between the ready state and the active state, and can respectively determine the starting and ending positions of multiple rows in the table. Where m is a positive integer.
[0169] In this way, the electronic device scans the row projection and column projection based on the state machine to determine the starting and ending positions of the rows and columns respectively, and can eliminate some noise interference caused by factors such as text detection errors and image shooting angles.
[0170] In addition, in the process of determining the starting and ending positions of rows and columns, the row projection and column projection are normalized so that the projection threshold is a fluctuating threshold rather than a fixed threshold. This can effectively eliminate the influence of cross-cell text (such as merged cells), making the scanning method of the state machine applicable to recognition scenarios of various table structures.
[0171] S505: The electronic device determines the table position according to the starting position and ending position of the row and column.
[0172] In some embodiments, after determining the starting and ending positions of rows and columns, the electronic device may determine the row and column separation positions. Subsequently, the electronic device may determine the coordinates of the cells based on the row and column separation positions. Once the coordinates of all cells in the table are determined, the electronic device may determine the position of the table. Optionally, the electronic device may set the upper left corner of the original image as the coordinate origin.
[0173] For example, the electronic device determines the end position of the previous line and the start position of the next line, and then uses the median of these two positions as the separation position of the two lines.
[0174] For another example, the electronic device determines the end position of the previous column and the start position of the next column, and then uses the median of these two positions as the separation position between the two columns.
[0175] In some embodiments, after determining the table position, the electronic device may further match the table position with the recognized text position to generate a final original image recognition result. Optionally, as shown in FIG5 , after step S505 , the following step S506 may be further included.
[0176] S506: The electronic device generates an original image recognition result according to the text position and the table position.
[0177] In some embodiments, in step S501 above, the electronic device may use OCR technology to recognize text in the original image, thereby determining the text position (e.g., corresponding to text row coordinates). Then, after determining the table position (e.g., corresponding to cell coordinates), the electronic device may match the text position with the table position, such as filling the text row content into the corresponding cell. The electronic device may then generate a recognition result of the original image, such as outputting an editable document.
[0178] Optionally, in the process of matching the text position and the table position, the electronic device may determine whether the text position and the table position match according to the center point or area overlap ratio corresponding to the text position and the table position.
[0179] Exemplarily, as shown in (d) in FIG1 , the electronic device recognizes the table structure, confirms the text position and the table position, matches the text position and the table position, and then fills the text row content into the corresponding cell.
[0180] Optionally, after matching the table position with the text position within the table, the electronic device may further match the table position with the text position outside the table to generate a final original image recognition result.
[0181] In this way, the electronic device recognizes the table structure based on text projection and the state machine, so that the table recognition method has high robustness and avoids interference from interference factors.
[0182] Moreover, compared with table structure recognition based on neural networks, the table recognition method provided in the embodiments of the present application can be applied to more table recognition scenarios, such as wireless table structure recognition scenarios of various styles, without having to pre-build data sets suitable for these scenarios to train neural networks, which can save the training time of the neural network and the annotation cost of the training set.
[0183] In addition, the table recognition method provided in the embodiment of the present application can recognize and return the table structure in real time, meeting the user's usage needs with a higher processing speed.
[0184] In the above embodiment, in step S504, the electronic device scans the row projection or column projection based on a single state machine to determine the corresponding starting position and ending position. In some examples, the electronic device may also decouple the detection of the starting position and the ending position, and determine the starting position or the ending position separately using a forward state machine and a reverse state machine, thereby reducing the complexity of state transitions.
[0185] In some embodiments, the electronic device detects the starting position through a forward state machine and detects the ending position through a reverse state machine.
[0186] For example, still taking the example of an electronic device scanning a set of one-dimensional signals in the horizontal direction of a binary image and realizing the start position and end position of the column based on the state machine detection, the process of the electronic device determining the start position and end position through the forward-reverse state machine is introduced in detail.
[0187] In some examples, after acquiring the column projection, the electronic device scans a set of one-dimensional signals in the horizontal direction of the binary image from left to right, and determines the starting position of the column based on the forward state machine detection, but does not detect the ending position of the column. It should be understood that the embodiments of the present application use the forward direction from left to right and the reverse direction from right to left as an example to illustrate the projection scanning process of the forward-reverse state machine. The electronic device may also use the reverse direction from left to right and the forward direction from right to left, and the embodiments of the present application are not limited to this.
[0188] Optionally, as shown in FIG9 , the forward state machine includes three states, such as a ready state (Pre), an active state (Active), and a post state (Post). Among them, the ready state indicates that a new column is ready to be scanned, and the active state indicates that a certain column is being scanned. Optionally, since the forward state machine does not need to detect the end position of the column, the active state of the forward state machine ends early, and the post state of the forward state machine is actually a special state separated from the active state of the single state machine in the above embodiment. The post state can then indicate that the scan of the previous column has ended early and is not yet ready to scan the next column. Each state has two possible transfer directions, namely, maintaining the current state or transferring to the next state.
[0189] Optionally, the input conversion method of the ready state is to determine the transition condition and direction of the ready state by judging the relationship between the signal and the sum of the buffer minimum value and the fluctuation threshold, as well as the counting situation (the counting threshold corresponding to the ready state is n1). For example, the signal is greater than or equal to the buffer minimum value plus the fluctuation threshold (v i ≥min(buf)+t) i =1, otherwise c i =0, the corresponding feasible transfer condition and direction are as follows:
[0190] a) If the signal is greater than or equal to the buffer minimum plus the fluctuation threshold, and the current state is in the last sub-state of the ready state, then the count is cleared and the forward state machine can transition to the active state to reset and update the buffer. For example, if c i =1, cnt=n1, then state=Active, cnt=0, buf={v i}.
[0191] b) If the signal is greater than or equal to the buffer minimum plus the fluctuation threshold, and is not in the last substate of the ready state, then the count is increased by one, the forward state machine transitions to the next substate of the ready state, and updates the buffer. For example, if c i =1, cnt<n1, then cnt=cnt+1, buf=buf∪{v i}.
[0192] c) If the signal is less than the minimum value of the buffer plus the fluctuation threshold, the count is reset to zero, the forward state machine transfers to the first sub-state of the ready state, and updates the buffer. For example, if c i =0, then cnt = 0, buf = buf∪{v i}.
[0193] Alternatively, the forward state machine does not need to determine the end position of the column, so the activation state does not need to convert the input. Then a corresponding feasible transition condition and method are as follows. Among them, the counting threshold corresponding to the activation state of the forward state machine preset in the current electronic device is n3.
[0194] a) If the current state is the last sub-state of the active state, then the count is cleared, the forward state machine transfers to the end state, and updates the buffer. For example, if cnt = n3, then state = Post, cnt = 0, buf = buf ∪ {v i}.
[0195] b) If there is no last sub-state in the active state, then the count is increased by one, the forward state machine transfers to the next sub-state of the active state, and updates the buffer. For example, if cnt<n3, then cnt=cnt+1, buf=buf∪{v i}.
[0196] Optionally, since the activation state of the forward state machine ends early, the end state of the forward state machine is actually a special state separated from the activation state of the single state machine in the above embodiment, then the transfer conditions and direction of the end state are the same as the activation state of the above single state machine.
[0197] For example, the input conversion method of the end state is to determine the transfer condition and direction of the end state by judging the difference between the signal and the maximum value of the buffer and the fluctuation threshold, as well as the counting situation (the counting threshold corresponding to the end state is n2). For example, if the signal is less than or equal to the maximum value of the buffer minus the fluctuation threshold (v i ≤max(buf)-t) i =1, otherwise c i =0, the corresponding feasible transfer condition and direction are as follows:
[0198] a) If the signal is less than or equal to the maximum value of the buffer minus the fluctuation threshold, and the current state is in the last substate of the end state, then the count is cleared and the forward state machine can transfer to the ready state to reset and update the buffer. For example, if c i =1, cnt = n2, then state = Pre, cut = 0, buf = {v i}.
[0199] b) If the signal is less than or equal to the maximum value of the buffer minus the fluctuation threshold, and is not in the last substate of the end state, then the count is increased by one, the forward state machine can transfer to the next substate of the end state, and update the buffer. For example, if c i =1, cnt<n2, then cnt=cnb+1, buf=buf∪{v i}.
[0200] c) If the signal is greater than the maximum value of the buffer minus the fluctuation threshold, then the count is reset to zero and the forward state machine can transition to the first substate of the end state and update the buffer. For example, if c i =0, then cnt = 0, buf = buf∪{v i}.
[0201] In this way, through the above method, the electronic device determines the starting position of the column based on the scanning column projection from left to right of the forward state machine.
[0202] In this way, when the forward state machine only detects the starting position, the counting threshold can be set more flexibly, so as to enter and exit each state more quickly, improve the sensitivity of detection, and facilitate the processing of narrow blank spaces or rows and columns with less text changes.
[0203] In some examples, the electronic device scans a set of one-dimensional signals in the horizontal direction of the binarized image from right to left, determines the end position of the column based on the reverse state machine detection, but does not detect the starting position of the column.
[0204] Optionally, during the process of scanning the column projection based on the reverse state machine, the electronic device selects the first starting position to the right of the starting position 1 determined by the forward state machine (or the rightmost position if it does not exist) as the starting point of the scan, scans the one-dimensional signal in the horizontal direction to the left, and determines an ending position between the two starting positions, where the ending position matches the starting position 1. It can be seen that the reverse state machine needs to perform multiple right-to-left (reverse) scans of the one-dimensional signal in the horizontal direction to determine the ending position of the column.
[0205] Optionally, if the reverse state machine has not yet determined the end position between the two starting positions when scanning from the scanning starting point to the starting position to be matched (such as starting position 1 determined based on the forward state machine), then the reverse scan can be terminated (i.e., not terminated prematurely) and the ready state can be forcibly switched to the active state. Optionally, in this case, the electronic device can determine the end position corresponding to the starting position to be matched to coincide with the starting position.
[0206] Optionally, as shown in Figure 10, the reverse state machine includes two states, such as a ready state (Pre) and an active state (Active), wherein the ready state indicates that a new column is ready to be scanned, and the active state indicates that a certain column is being scanned.
[0207] Optionally, since the reverse state machine only detects one termination position per scan, the algorithm stops once the reverse state machine enters the active state, and therefore there is no transition in the active state.
[0208] Optionally, the input conversion method of the ready state is to determine the transition condition and direction of the ready state by judging the relationship between the signal and the sum of the buffer minimum value and the fluctuation threshold, as well as the counting situation (the counting threshold corresponding to the ready state is n1). For example, the signal is greater than or equal to the buffer minimum value plus the fluctuation threshold (v i ≥min(buf)+t) i =1, the corresponding feasible transfer condition and direction are as follows:
[0209] a) If the signal is greater than or equal to the buffer minimum plus the fluctuation threshold, and is currently in the last sub-state of the ready state, the reverse state machine can be transferred to the active state and the scan ends. For example, if c i =1, cnt=n1, then state=Active.
[0210] b) If the signal is greater than or equal to the buffer minimum plus the fluctuation threshold, and is not in the last substate of the ready state, then the count is increased by one, and the reverse state machine can transfer to the next substate of the ready state and update the buffer. For example, if c i=1, cnt<n1, then cnt=cnt+1, buf=buf∪{v i}.
[0211] c) If the signal is less than the minimum value of the buffer plus the fluctuation threshold, then the count is reset to zero and the reverse state machine can transfer to the first sub-state of the ready state and update the buffer. For example, if c i =0, then cnt = 0, buf = buf∪{v i}.
[0212] In this way, through the above method, the electronic device determines the end position of the column based on the scanning column projection from right to left of the reverse state machine.
[0213] Alternatively, the electronic device may first scan the column projection from right to left using the forward state machine to determine the column's end position. Then, based on the determined column's end position, the reverse state machine performs multiple scans to determine the corresponding start position for each end position. The present embodiment does not limit the scanning order of the forward and reverse state machine's column projections.
[0214] Exemplarily, the electronic device scans the column projection in the horizontal direction through the forward state machine, and determines that the forward state machine is transferred from the ready state to the active state, that is, the forward state machine is transferred from the state of preparing to scan a new column to the state of scanning a column, then as shown in Figure 6, the corresponding position is the starting position of the nth column. Thereafter, the forward state machine continues to scan the current nth column, and the forward state machine is transferred from the active state to the end state, that is, the forward state machine is transferred from the state of scanning the nth column to the state of having ended the scan of the nth column in advance and not yet ready to scan the n+1th column. Thereafter, the forward state machine continues to scan, and the forward state machine is transferred from the end state to the active state, that is, the forward state machine is transferred from the state of not yet ready to scan the n+1th column to the state of scanning the n+1th column, then as shown in Figure 6, the corresponding position is the starting position of the n+1th column. That is, the forward state machine continuously scans the column projection in one direction horizontally (such as from left to right or from right to left), loops the transition process of the above-mentioned ready state, activation state, and end state, and can determine the starting positions of multiple columns in the table respectively.
[0215] After determining the starting positions of multiple columns, the electronic device can use the reverse state machine to scan the column projections in the horizontal direction in the opposite direction to the scanning direction of the forward state machine to determine the end positions corresponding to the starting positions of these determined columns. For example, take the scanning direction of the forward state machine as from left to right and the scanning direction of the reverse state machine as from right to left. As shown in Figure 6, in the process of determining the end position of the nth column, the electronic device first obtains the starting position of the nth column, determines the first starting position to the right of the starting position of the nth column, that is, determines the starting position of the n+1th column, and starts scanning to the left with the starting position of the n+1th column as the starting point. Determine that the reverse state machine has transitioned from the ready state to the active state, that is, the reverse state machine has transitioned from the state of preparing to scan a new column to the state of scanning a column, then as shown in Figure 6, the corresponding position is the end position of the nth column. After the reverse state machine switches to the active state, the device ends the scan of the reverse state machine. That is, the reverse state machine scans the column projection multiple times in the horizontal direction opposite to the scanning direction of the forward state machine, starting from the starting position of the column that has been determined by the forward state machine, and loops the above-mentioned transition process from the ready state to the active state, so as to determine the end positions of multiple columns in the table respectively.
[0216] Similarly, the electronic device uses a forward-reverse state machine to scan a set of one-dimensional signals in the vertical direction of the binarized image, thereby detecting the starting and ending positions of a row based on the state machine. The specific method can be referred to the specific implementation method of the scanning column projection process in the above example, and will not be repeated here. Similarly, the embodiments of the present application do not limit the scanning order of the forward state machine and the reverse state for the row projection.
[0217] Optionally, the electronic device determines the starting position of the line by projecting the scan line from top to bottom using the forward state machine, and determines the ending position of the line by projecting the scan line from bottom to top using the reverse state machine. It should be understood that the embodiment of the present application uses the forward direction from top to bottom and the reverse direction from bottom to top as an example to illustrate the projection scanning process of the forward-reverse state machine. The electronic device may also use the reverse direction from top to bottom and the forward direction from bottom to top, and the embodiment of the present application is not limited to this.
[0218] Exemplarily, the electronic device scans the row projection in the vertical direction through the forward state machine, and determines that the forward state machine is transferred from the ready state to the active state, that is, the forward state machine is transferred from the state of preparing to scan a new row to the state of scanning a row, then as shown in FIG6 , the corresponding position is the starting position of the m-th row. Thereafter, the forward state machine continues to scan the current m-th row, and the forward state machine is transferred from the active state to the end state, that is, the forward state machine is transferred from the state of scanning the m-th row to the state of having ended the scan of the m-th row in advance and not yet ready to scan the m+1-th row. Thereafter, the forward state machine continues to scan, and the forward state machine is transferred from the end state to the active state, that is, the forward state machine is transferred from the state of not yet ready to scan the m+1-th row to the state of scanning the m+1-th row, then as shown in FIG6 , the corresponding position is the starting position of the m+1-th row. That is, the forward state machine continuously scans the row projection in one direction in the vertical direction (such as from top to bottom or from bottom to top), loops the above-mentioned transition process of the preparation state, activation state, and end state, and can determine the starting positions of multiple rows in the table respectively.
[0219] After determining the starting positions of multiple rows, the electronic device can use the reverse state machine to scan the row projections in a vertical direction, in a direction opposite to the scanning direction of the forward state machine, to determine the end positions corresponding to the starting positions of these determined rows. For example, take the forward state machine's scanning direction as top-to-bottom and the reverse state machine's scanning direction as bottom-to-top. As shown in Figure 6, in the process of determining the end position of the mth row, the electronic device first obtains the starting position of the mth row, determines the first starting position below the starting position of the mth row, that is, determines the starting position of the m+1th row, and begins scanning upward with the starting position of the m+1th row as the starting point. Determine that the reverse state machine has transitioned from the ready state to the active state, that is, the reverse state machine has transitioned from the state of preparing to scan a new row to the state of scanning a row. Then, as shown in Figure 6, the corresponding position is the end position of the mth row. After the reverse state machine switches to the active state, the device ends the scan of the reverse state machine. That is, the reverse state machine scans the row projection multiple times in the vertical direction opposite to the scanning direction of the forward state machine, starting from the starting position of the row that has been determined by the forward state machine, and loops the above-mentioned transition process from the ready state to the active state, thereby determining the end positions of multiple rows in the table respectively.
[0220] For example, Figure 11 is a diagram illustrating the recognition effect of the starting and ending positions of a row of a table provided in an embodiment of the present application. As shown in Figure 11, the pixel height of the image is 600 (corresponding to the horizontal axis), and the row projection value is normalized to between [0, 1] (corresponding to the vertical axis).
[0221] In some examples, the electronic device may detect the starting position and the ending position of a row through a single state machine.
[0222] In other examples, the electronic device may detect the starting position of a line using the forward state machine, such as determining starting position 101 and starting position 102. Subsequently, the electronic device may detect the ending position of the line using the reverse state machine. For example, during the process of scanning the line projection using the reverse state machine, the electronic device may select the first starting position 102 to the right of the starting position 101 determined by the forward state machine as the starting point of the scan, and then scan the projection values of the line projection vertically to the left to determine the ending position 103 between the two starting positions.
[0223] Optionally, if the electronic device fails to determine the end position 103, it can stop the reverse scan and force the reverse state machine to switch from the ready state to the active state. Optionally, in this case, the electronic device can also determine that the reference numeral 101 is used to indicate the starting position and the end position of the corresponding row.
[0224] In some scenarios, after determining the text line area in the original image, the electronic device can first scale the text line area. After scaling the text height to a preset range, the electronic device then determines the corresponding table structure based on the scaled text line area. This simplifies the difficulty of setting the counting threshold in the aforementioned state transition conditions and directions based on the standardized text height.
[0225] For example, Figure 12 is a flowchart of another table recognition method provided by an embodiment of the present application. As shown in Figure 12, the above step S502 may include step S1201 and step S1202.
[0226] S1201: The electronic device scales the text line area according to preset conditions.
[0227] The preset condition is used to indicate a preset interval preset in the electronic device for scaling the text line area.
[0228] In some embodiments, when acquiring a text line region in an original image, the electronic device may determine the short side height of a text line detection frame and estimate the text line height based on the short side height. The electronic device may then scale the text line height to within a preset range. Accordingly, after the text line height is scaled, the text line region is scaled.
[0229] In some embodiments, the electronic device recognizes text in the original image using OCR technology and can determine the text position (such as corresponding to the text line coordinates). Then, when the electronic device scales the text line height, it can simultaneously adjust the text line coordinates to correct the corresponding text position.
[0230] S1202: The electronic device generates a binary image corresponding to the original image based on the scaled text line area.
[0231] In some embodiments, after obtaining the scaled text line area, the electronic device can use a binarization algorithm to set the pixel value of the scaled text line area to 1 and the pixel values of other areas outside the scaled text line area to 0, thereby generating a binary image corresponding to the original image.
[0232] Afterwards, the electronic device may execute the above steps S503 to S505 again, and determine the position of the table based on the binary image corresponding to the scaled text line area. Then, as shown in FIG12 , the above step S506 may be implemented as step S1203.
[0233] S1203: After scaling the text position and the table position, generate an original image recognition result.
[0234] In some embodiments, after determining the position of the table (such as corresponding to the cell coordinates), the electronic device may reversely scale the cell coordinates and the text row coordinates based on the scaling ratio determined in step S1201 above to match the size of the original image. For example, in steps S1201 and S1202, the electronic device reduces the text row area and the text row coordinates and generates a corresponding binary image to determine the position of the table. Then, in step S1203, the electronic device enlarges the determined cell coordinates and text row coordinates to match the size of the original image. In addition, the electronic device may match the cell coordinates and the text row coordinates to generate the original image recognition result.
[0235] For example, if the preset interval is generally smaller than the text row height, scaling the text row area is equivalent to downsampling the image, reducing the number of sampling points during image recognition. This can effectively reduce algorithm processing steps and time during subsequent table recognition, improving table recognition speed.
[0236] In some scenarios, tables in the original image may be tilted due to factors such as shooting angle. Therefore, after determining the text row area in the original image, the electronic device can first correct the tilt angle of the text row area. The electronic device then determines the corresponding table structure based on the tilt-corrected text row area. This enables the recognition of tilted tables and expands the applicable table recognition scenarios.
[0237] For example, Figure 13 is a flowchart of another table recognition method provided by an embodiment of the present application. As shown in Figure 13, the above step S502 may include step S1301 and step S1302.
[0238] S1301: The electronic device corrects the tilt angle of the text line area.
[0239] In some embodiments, the electronic device may determine whether the text line region is tilted during the process of acquiring the text line region in the original image. Alternatively, the electronic device may determine the border of the original image as horizontal and vertical, and then determine whether the text line region is tilted based on the determined horizontal and vertical directions.
[0240] If the electronic device determines that a text line region is tilted, the electronic device may determine a minimum rotated rectangle or centerline encompassing the text line region. The electronic device may then estimate the tilt angle of the text line region based on the minimum rotated rectangle or centerline. The electronic device may then correct the text line region by rotating it based on the tilt angle.
[0241] Optionally, the electronic device may determine the minimum rotated rectangle of the text line area based on algorithms such as cv2.minAreaRect, and determine the tilt angle of the text line area based on the minimum rotated rectangle and the border of the original image.
[0242] Optionally, after identifying a text line region, the electronic device may determine a centerline by connecting the center points of the two short sides of the text line region. Furthermore, the electronic device may determine a weighted average of all text lines based on the long sides of the text line region. Based on the centerline and the weighted average, the electronic device may determine the tilt angle of the text line region.
[0243] In some embodiments, the electronic device recognizes text in the original image using OCR technology and can determine the text position (e.g., corresponding to the text line coordinates). Based on the above method, the electronic device can also determine the tilt angle corresponding to the text position and correct the text position by rotating it.
[0244] S1302: The electronic device generates a binary image corresponding to the original image based on the text line area after the tilt angle is corrected.
[0245] In some embodiments, after obtaining the text line area after correcting the tilt angle, the electronic device can use a binarization algorithm to set the pixel value of the text line area after correcting the tilt angle to 1, and set the pixel values of other areas outside the text line area after correcting the tilt angle to 0, thereby generating a binary image corresponding to the original image.
[0246] Afterwards, the electronic device may execute the above steps S503 to S505 again, and determine the position of the table based on the binarized image corresponding to the text line area after the tilt angle is corrected. Then, as shown in FIG13 , the above step S506 may be implemented as step S1303.
[0247] S1303: After performing tilt processing on the text position and the table position, an original image recognition result is generated.
[0248] In some embodiments, after determining the table position (e.g., corresponding to the cell coordinates), the electronic device may reversely rotate the cell coordinates and the text row coordinates based on the tilt angle determined in step S1201 to match the angle of the original image. Furthermore, the electronic device may match the cell coordinates and the text row coordinates to generate an original image recognition result.
[0249] In this way, the tilted table can be processed by rotating the text row detection frame (ie, rotating the text row area).
[0250] Optionally, after executing the above step S501 to determine the text line area, the electronic device may first scale the text line area and then perform tilt angle correction on the text line area, or first perform tilt angle correction on the text line area and then scale the text line area. That is, the embodiment of the present application does not limit the execution order between step S1201-step S1202 and step S1301-step S1302. Correspondingly, the embodiment of the present application also does not limit the execution order of step S1203 and step S1303. In addition, in this process, if the electronic device determines that the text line area is already within the preset interval, it is not necessary to scale the text line area; if the electronic device determines that the text line area does not need to be corrected for the tilt angle, it is not necessary to correct the tilt angle of the text line area.
[0251] The table recognition method provided by the embodiment of the present application is described in detail above with reference to Figures 5 to 13. The device provided by the embodiment of the present application is described in detail below with reference to Figure 14.
[0252] In one possible design, Figure 14 is a schematic diagram of the structure of a first device provided in an embodiment of the present application. As shown in Figure 14, the first device 1400 may include: a processing unit 1401. The first device 1400 may be used to implement the functions of the electronic device involved in the above method embodiment.
[0253] Optionally, the processing unit 1401 is used to support the first device 1400 to execute S501-S506 in Figure 5; and / or, support the first device 1400 to execute S1201, S1202 and S1203 in Figure 12; and / or, support the first device 1400 to execute S1301, S1302 and S1303 in Figure 13.
[0254] Optionally, as shown in Figure 14, the first device 1400 may further include a transceiver unit 1402, configured to support the first device 1400 in performing a receiving operation or a sending operation. The transceiver unit may include a receiving unit and a sending unit, and may be implemented by a transceiver or a transceiver-related circuit component, and may be a transceiver or a transceiver module.
[0255] Optionally, as shown in FIG14 , the first device 1400 may further include a display unit for supporting the first device 1400 to display interface content, such as displaying an original image, displaying an original image recognition result after table recognition, and the like.
[0256] Optionally, the operations and / or functions of each unit in the first device 1400 are respectively for implementing the corresponding processes of the table recognition method described in the above method embodiment. All relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional unit. For the sake of brevity, they will not be repeated here.
[0257] Optionally, the first device 1400 shown in FIG14 may further include a storage unit (not shown in FIG14 ) storing a program or instruction. When the processing unit 1401 executes the program or instruction, the first device 1400 shown in FIG14 may execute the table recognition method described in the above method embodiment.
[0258] The technical effects of the first device 1400 shown in FIG14 may refer to the technical effects of the table recognition method described in the above method embodiment, and will not be repeated here.
[0259] In addition to being in the form of the first device 1400, the technical solution provided in this application may also be a functional unit or chip in the first device, or an apparatus used in conjunction with the first device.
[0260] An embodiment of the present application also provides a chip system, including: a processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the chip system implements the method in any of the above method embodiments.
[0261] Optionally, there may be one or more processors in the chip system. The processor may be implemented in hardware or software. When implemented in hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented in software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.
[0262] Optionally, the memory in the chip system may be one or more. The memory may be integrated with the processor or may be provided separately from the processor, which is not limited in the embodiments of the present application. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or provided on different chips. The embodiments of the present application do not specifically limit the type of memory or the configuration of the memory and the processor.
[0263] Exemplarily, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD) or other integrated chips.
[0264] It should be understood that each step in the above method embodiment can be completed by hardware integrated logic circuits in a processor or by software instructions. The method steps disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.
[0265] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer, the computer executes the above-mentioned related steps to implement the table recognition method in the above-mentioned embodiment.
[0266] The embodiment of the present application further provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the table recognition method in the above-mentioned embodiment.
[0267] In addition, embodiments of the present application further provide a device. Specifically, the device may be a component or module, and may include one or more processors and a memory connected to each other. The memory is configured to store a computer program. When the computer program is executed by one or more processors, the device performs the table recognition method described in each of the above method embodiments.
[0268] The apparatus, computer-readable storage medium, computer program product, or chip provided in the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding methods provided above, and will not be repeated here.
[0269] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner or can be implemented by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disc (CD-ROM) or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC).
[0270] Through the description of the above embodiments, those skilled in the art will clearly understand that for the sake of convenience and brevity, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed; that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working processes of the above-mentioned systems, devices, and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0271] In the several embodiments provided in this application, it should be understood that the disclosed methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of modules or units, which can be electrical, mechanical or other forms.
[0272] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.
[0273] Computer-readable storage media include, but are not limited to, any of the following: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and other media that can store program code.
[0274] It should be noted that any part or all of any features of any embodiment in this application can be freely combined, and the combined technical solutions are also within the scope of this application.
[0275] The above is only a specific embodiment of the present application, but the scope of protection of this application is not limited to this. Any changes or substitutions within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.
Claims
1. A table recognition method, characterized in that: Applied to a first device, the method includes: Get the text row area corresponding to the table area in the image; Based on the text line area, generating a binary image; Accumulating the pixel values of the pixel points in a single row in the binary image to obtain a row projection corresponding to the binary image; and accumulating the pixel values of the pixel points in a single column in the binary image to obtain a column projection corresponding to the binary image; Scanning the row projection and the column projection by a state machine to determine the starting position and the ending position of the row and the starting position and the ending position of the column; The table position is determined according to the starting position and the ending position of the row and the starting position and the ending position of the column.
2. The method according to claim 1, characterized in that The step of generating a binary image based on the text line region includes: According to preset conditions, scaling the text line area; The binary image is generated based on the scaled text line area.
3. The method according to claim 1 or 2, characterized in that: The step of generating a binary image based on the text line region includes: Correcting the tilt angle of the text line area; The binary image is generated based on the text line region after the tilt angle is corrected.
4. The method according to any one of claims 1 to 3, characterized in that The state machine includes a ready state and an activated state, wherein the ready state is used to indicate scanning a new column or row, and the activated state is used to indicate scanning a certain column or row; the ready state or the activated state includes two transfer directions, namely maintaining the current state or transferring to the next state.
5. The method according to claim 4, characterized in that The ready state or the active state includes a plurality of sub-states.
6. The method according to claim 5, characterized in that Scanning the row projection and the column projection by a state machine to determine the starting position and the ending position of the row and the starting position and the ending position of the column includes: The transfer direction or transfer condition of the ready state or the activated state is determined by one or more of the counting threshold, counting situation, signal value, buffer value, and fluctuation threshold. The counting situation is used to indicate the number of sub-states transferred within the ready state or the activated state, and the number is less than or equal to the counting threshold. The buffer is used to convert the input signal.
7. The method according to any one of claims 4 to 6, characterized in that: Scanning the row projection and the column projection by a state machine to determine the starting position and the ending position of the row and the starting position and the ending position of the column includes: Scanning the column projections in the horizontal direction by the state machine, during the scanning process, determining the starting position and the ending position of each column according to the state transition of the state machine, wherein the position corresponding to the transition of the state machine from the ready state to the active state is determined to be the starting position of the nth column, and the ending position of the nth column is the position corresponding to the transition of the state machine from the active state to the ready state that is closest to the starting position of the nth column, and n is a positive integer; The row projections are scanned in a vertical direction by the state machine. During the scanning process, the starting position and the ending position of each row are determined according to the state transition of the state machine, wherein the position corresponding to the transition of the state machine from the ready state to the activated state is determined to be the starting position of the mth row, and the ending position of the mth row is the position corresponding to the transition of the state machine from the activated state to the ready state that is closest to the starting position of the mth row, and m is a positive integer.
8. The method according to any one of claims 4 to 6, characterized in that: The state machine includes a forward state machine and a reverse state machine, the forward state machine includes the ready state, the activated state and the end state, and the reverse state machine includes the ready state and the activated state; wherein the end state is used to indicate that the scanning of the previous row or column has been ended in advance and it is not yet ready to scan the next row or column.
9. The method according to claim 8, characterized in that Scanning the row projection and the column projection by a state machine to determine the starting position and the ending position of the row and the starting position and the ending position of the column includes: Scanning the column projections in the horizontal direction by the forward state machine, during the scanning process, determining the starting position of each column according to the state transition of the forward state machine, wherein determining the position corresponding to the transition of the forward state machine from the ready state to the active state is the starting position of the nth column, and then the forward state machine is transferred from the active state to the end state and then to the ready state, and determining to be ready to scan the n+1th column, where n is a positive integer; The reverse state machine scans the column projection along the horizontal direction in a direction opposite to the scanning direction of the forward state machine. During the scanning process, the end position of the nth column corresponding to the starting position of the nth column is determined according to the state transition of the reverse state machine, wherein the scanning is started with the starting position of the n+1th column as the starting point to determine that the position corresponding to the transition of the reverse state machine from the ready state to the activated state is the end position of the nth column.
10. The method according to claim 8, characterized in that Scanning the row projection and the column projection by a state machine to determine the starting position and the ending position of the row and the starting position and the ending position of the column includes: Scanning the row projections along the vertical direction by the forward state machine, during the scanning process, determining the starting position of each row according to the state transition of the forward state machine, wherein determining that the position corresponding to the transition of the forward state machine from the ready state to the active state is the starting position of the mth row, and then the forward state machine transitions from the active state to the end state and then to the ready state, determining to be ready to scan the m+1th row, where m is a positive integer; The reverse state machine scans the row projection along a vertical direction in a direction opposite to the scanning direction of the forward state machine. During the scanning process, the end position of the mth row corresponding to the starting position of the mth row is determined according to the state transition of the reverse state machine, wherein the scanning is started with the starting position of the m+1th row as the starting point to determine that the position corresponding to the transition of the reverse state machine from the ready state to the activated state is the end position of the mth row.
11. The method according to any one of claims 1 to 10, characterized in that The determining the table position according to the starting position and the ending position of the row and the starting position and the ending position of the column comprises: Determine the row separation position and the column separation position according to the starting position and the ending position of the row and the starting position and the ending position of the column respectively; The coordinates of cells included in the table are determined according to the separation positions of the rows and the separation positions of the columns.
12. The method according to any one of claims 1 to 11, characterized in that The step of generating a binary image based on the text line region includes: The pixel values within the text line area are set to 1, and the pixel values outside the text line area are set to 0.
13. The method according to any one of claims 1 to 12, characterized in that Scanning the row projection and the column projection by a state machine to determine the starting position and the ending position of the row and the starting position and the ending position of the column includes: Normalizing the row projection and the column projection respectively; The normalized row projection and column projection are scanned by the state machine to determine the starting position and the ending position of the row and the starting position and the ending position of the column.
14. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is coupled to the processor, the memory is used to store computer program code, the computer program code includes computer instructions, and when the processor reads the computer instructions from the memory, the electronic device executes the method as described in any one of claims 1-13.
15. A server, characterized in that: include: A processor and a memory, wherein the memory is coupled to the processor, the memory is used to store computer program code, the computer program code includes computer instructions, and when the processor reads the computer instructions from the memory, the server executes the method according to any one of claims 1 to 13.
16. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises a computer program, and when the computer program is executed on an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 13.
17. A computer program product, characterized in that When the computer program product is executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 13.
Citation Information
Patent Citations
Form recognition method and device in image, electronic equipment and storage medium
CN111814722A
Table structure identification method for wireless frame table image
CN114529922A
Table merging cell detection method
CN114529923A
Excel table analysis method, system and equipment and storage medium
CN114970475A
Method and device for parsing table in document image
US20190266394A1
Cited By
Wireless table detection method and system
CN119274197A
