A table recognition method, device, equipment and storage medium

Abstract

The application provides a table recognition method, device and equipment and a storage medium. The method comprises: acquiring a track point sequence collected when a user writes on a screen, and preprocessing the track point sequence; determining table track points and text track points from the preprocessed track point sequence; rendering the table track points into an image to obtain a pure table image, and rendering the text track points into an image to obtain a text image; performing table recognition on the pure table image to obtain a table recognition result, and performing text recognition on the text image to obtain a text recognition result; matching the table recognition result and the text recognition result to obtain a matching result; and generating a table containing text content based on recognition information containing the table recognition result, the text recognition result and the matching result. The table recognition method provided by the application can restore a table containing text content written by a user.

Description

Technical Field

[0001] This invention relates to the field of table recognition technology, and in particular to a table recognition method, apparatus, device, and storage medium. Background Technology

[0002] In some scenarios, it is necessary to recognize tables written by users on the screen. Current table recognition solutions are mostly based on table images, that is, to obtain table images by taking pictures of the table written by users on the screen, and to obtain the table by analyzing the table images.

[0003] While table image recognition schemes can achieve table recognition, they have significant limitations. Specifically, they perform well on high-quality table images but poorly on low-quality ones. Summary of the Invention

[0004] In view of this, the present invention provides a table recognition method, apparatus, device, and storage medium to solve the problem that existing table image-based recognition schemes have significant limitations. The technical solution is as follows:

[0005] A table recognition method, comprising:

[0006] Acquire the trajectory point sequence collected when the user writes on the screen, and preprocess the trajectory point sequence;

[0007] The table trajectory points and text trajectory points are determined from the preprocessed trajectory point sequence;

[0008] The table trajectory points are rendered into an image to obtain a pure table image, and the text trajectory points are rendered into an image to obtain a text image;

[0009] Perform table recognition on the pure table image to obtain table recognition results, and perform text recognition on the text image to obtain text recognition results;

[0010] The table recognition result is matched with the text recognition result to obtain the matching result;

[0011] Based on the recognition information including the table recognition result, the text recognition result, and the matching result, a table containing text content is generated.

[0012] Optionally, the trajectory point sequence is a sequence of trajectory points collected up to the first identification time, and the identification information is the identification information corresponding to the first identification time;

[0013] The table recognition method also includes:

[0014] Every preset time interval, it checks whether the trajectory point sequence collected up to the current time has changed compared to the trajectory point sequence collected up to the previous identification time;

[0015] If so, then based on the trajectory points that have changed from the trajectory point sequence collected up to the second identification time to the trajectory point sequence collected up to the previous identification time, and the identification information corresponding to the previous identification time, the identification information corresponding to the second identification time is determined, wherein the second identification time is the current time or a time after the current time;

[0016] Based on the recognition information corresponding to the second recognition time, a table containing text content is generated.

[0017] Optionally, determining the table trajectory points and text trajectory points from the preprocessed trajectory point sequence includes:

[0018] The preprocessed trajectory point sequence is rendered into an image, which is used as the image to be detected.

[0019] Detect a table region and several text regions within the table region from the image to be detected;

[0020] From the preprocessed sequence of trajectory points, determine the trajectory points belonging to the table area and the trajectory points belonging to each text area respectively;

[0021] The table trajectory points are determined from the trajectory points belonging to the table area, and the trajectory points belonging to each text area are determined as text trajectory points.

[0022] Optionally, determining the trajectory points belonging to the table region and the trajectory points belonging to each text region from the preprocessed trajectory point sequence includes:

[0023] From the preprocessed trajectory point sequence, determine the trajectory point sequence belonging to the table area and the trajectory point sequences belonging to each text area, wherein each trajectory point sequence is a trajectory point sequence that makes up a stroke.

[0024] The trajectory points contained in the sequence of trajectory points belonging to the table area are determined as the trajectory points belonging to the table area;

[0025] For each text region, the trajectory points contained in the trajectory point subsequence belonging to that text region are determined as the trajectory points belonging to that text region.

[0026] Optionally, determining the trajectory point subsequences belonging to the table region and the trajectory point subsequences belonging to each text region from the preprocessed trajectory point sequence includes:

[0027] For each of the table area and the plurality of text areas:

[0028] From the preprocessed trajectory point sequence, a trajectory point subsequence that meets a preset condition is determined, wherein the preset condition is that the proportion of trajectory points located in the region in the trajectory point sequence is greater than a preset proportion threshold.

[0029] The sequence of trajectory points that meets the preset conditions is determined as the sequence of trajectory points belonging to this region.

[0030] Optionally, determining the table trajectory points from the trajectory points belonging to the table region includes:

[0031] Traverse the sequence of trajectory points belonging to the table region:

[0032] Determine the number of trajectory points in each text region of the currently traversed trajectory point subsequence, and obtain the corresponding number of trajectory points for each text region;

[0033] If the maximum number of trajectory points among the obtained number of trajectory points is greater than the target number, the current traversed trajectory point subsequence will be filtered out, wherein the target number is the product of a set threshold and the number of trajectory points in the current traversed trajectory point subsequence.

[0034] After the traversal is complete, the trajectory points contained in the remaining trajectory point subsequence are determined as the table trajectory points.

[0035] Optionally, rendering the text trajectory points into an image to obtain a text image includes:

[0036] Render the text trajectory points of each text region into images to obtain the text images corresponding to each text region;

[0037] Perform table recognition on the pure table image to obtain table recognition results, and perform text recognition on the text image to obtain text recognition results, including:

[0038] The pure table image is input into a pre-trained table recognition model to obtain a table recognition result containing table structure information and the position information of each table cell. The table recognition model is trained using a training table image labeled with table structure information and the position information of each table cell.

[0039] For each text region, the text image corresponding to that text region is input into a pre-trained text recognition model to obtain a text recognition result containing the text content corresponding to that text region. The text recognition model is trained using training text images labeled with text content.

[0040] Optionally, determining the identification information corresponding to the second identification time based on the trajectory points that have changed compared to the trajectory point sequence collected up to the second identification time and the identification information corresponding to the previous identification time includes:

[0041] Based on the positional information of trajectory points that have changed from the trajectory point sequence collected up to the second identification time to the trajectory point sequence collected up to the previous identification time, several areas of change are identified.

[0042] If the plurality of variable regions include text variable regions, then the variable text content is determined based on the text trajectory points belonging to the text variable regions.

[0043] If the plurality of changing regions include a table changing region, then the table recognition result is re-acquired based on the table trajectory points belonging to the table changing region and other table trajectory points;

[0044] Based on the changed text content and / or the re-acquired table recognition results, the recognition information corresponding to the previous recognition time is updated, and the updated recognition information is used as the recognition information corresponding to the second recognition time.

[0045] Optionally, based on the positional information of the trajectory points whose changes occurred during the collection of trajectory point sequences up to the second identification time compared to the collection of trajectory point sequences up to the previous identification time, several change regions are determined, including:

[0046] Obtain a pre-constructed modification table, wherein the position in the modification table corresponding to each pixel position of the screen is a first value;

[0047] Modify the first value at the position corresponding to the changed trajectory point in the modification table to the second value;

[0048] Search for connected components in the modified table that have the value of the second value, and identify each connected component as a variable region.

[0049] Optionally, determining text-varying regions and / or table-varying regions from the plurality of changing regions includes:

[0050] For each affected area:

[0051] The changed area is matched with each text area and each table cell area in the table identified at the previous identification time.

[0052] If the changed area matches the text area, then the changed area is determined to be a text changed area;

[0053] If the changed area matches a table cell range, then the changed area is determined to be a changed area of ​​the table.

[0054] If the changed region does not match any of the text regions and table cell regions identified in the previous identification time, then the table region and text region are detected based on the pre-trained detection model.

[0055] A table recognition device includes: a trajectory point data acquisition module, a trajectory point determination module, an image acquisition module, a recognition module, a matching module, and a table generation module;

[0056] The trajectory point data acquisition module is used to acquire the trajectory point sequence collected when the user writes on the screen, and to preprocess the trajectory point sequence.

[0057] The trajectory point determination module is used to determine table trajectory points and text trajectory points from the preprocessed trajectory point sequence;

[0058] The image acquisition module is used to render the table trajectory points into an image to obtain a pure table image, and to render the text trajectory points into an image to obtain a text image;

[0059] The recognition module is used to perform table recognition on the pure table image to obtain table recognition results, and to perform text recognition on the text image to obtain text recognition results;

[0060] The matching module is used to match the table recognition result with the text recognition result to obtain a matching result;

[0061] The table generation module is used to generate a table containing text content based on recognition information including the table recognition result, the text recognition result, and the matching result.

[0062] A processing device, comprising: a memory and a processor;

[0063] The memory is used to store programs;

[0064] The processor is used to execute the program to implement each step of the table recognition method described above.

[0065] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the table recognition method described in any of the preceding claims.

[0066] The table recognition method, apparatus, device, and storage medium provided by this invention first acquire a sequence of trajectory points collected when a user writes on the screen, and preprocesses the trajectory point sequence. Then, table trajectory points and text trajectory points are determined from the preprocessed trajectory point sequence. Next, the table trajectory points and text trajectory points are rendered into images. Then, table recognition is performed on the rendered pure table image, and text recognition is performed on the rendered text image. Next, the table recognition result and text recognition result are matched to obtain a matching result. Finally, based on the recognition information including the table recognition result, text recognition result, and matching result, a table containing text content is generated. The table recognition method provided by this invention can reconstruct a table containing text content written by a user. Because the table recognition method provided by this invention reconstructs a table containing text content written by a user by analyzing the writing trajectory point sequence, rather than by analyzing a table image (obtained by photographing a table written by a user on the screen), it does not have the problems of recognition methods based on table images. That is, the table recognition method provided by this invention has no limitations. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0068] Figure 1 This is a schematic diagram of a hardware architecture according to an embodiment of the present invention;

[0069] Figure 2 A flowchart illustrating the table recognition method provided in an embodiment of the present invention;

[0070] Figure 3 This invention provides a schematic diagram of the process for determining table trajectory points and text trajectory points from a preprocessed sequence of trajectory points.

[0071] Figure 4 A flowchart illustrating a table recognition method suitable for real-time handwriting scenarios provided in an embodiment of the present invention;

[0072] Figure 5 This is a schematic diagram of the structure of the table recognition device provided in an embodiment of the present invention;

[0073] Figure 6 This is a schematic diagram of the processing device provided in an embodiment of the present invention. Detailed Implementation

[0074] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0075] Given the shortcomings of current table image-based recognition schemes, this invention has conducted research and, through continuous research, has finally proposed a table recognition method with better performance. This table recognition method does not obtain the table by analyzing the table image, but by analyzing the trajectory point data of the user's writing.

[0076] Before introducing the table recognition method provided by this invention, the hardware architecture involved in this invention will be described first.

[0077] In one possible implementation, such as Figure 1 As shown, the hardware architecture involved in this invention may include: electronic device 101 and server 102.

[0078] For example, electronic device 101 can be any electronic product that can interact with a user through one or more means such as a keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device, such as personal computer, laptop, tablet, mobile phone, smart TV, etc.

[0079] It should be noted that, Figure 1 This is just one example; there can be many types of electronic devices, not limited to... Figure 1 The laptop in the middle.

[0080] For example, server 102 can be a single server, a server cluster consisting of multiple servers, or a cloud computing server center. Server 102 may include processors, memory, and network interfaces, etc.

[0081] For example, electronic device 101 can establish a connection and communicate with server 102 through a wireless communication network; for example, electronic device 101 can establish a connection and communicate with server 102 through a wired network.

[0082] Electronic device 101 can acquire trajectory point data when the user writes and send the trajectory point data to server 102. Server 102 obtains a table by analyzing the trajectory point data when the user writes.

[0083] In another possible implementation, the hardware architecture involved in this invention may include an electronic device. The electronic device is one with strong data processing capabilities.

[0084] Electronic devices can acquire trajectory point data when a user writes, and obtain tables by analyzing the trajectory point data.

[0085] Those skilled in the art should understand that the above-described electronic devices and servers are merely examples, and other existing or future electronic devices or servers that are applicable to this invention should also be included within the scope of protection of this invention, and are hereby incorporated by reference.

[0086] The table recognition method provided by the present invention will be described in the following embodiments.

[0087] Please see Figure 2 The diagram illustrates a flowchart of a table recognition method provided in an embodiment of the present invention. The method may include:

[0088] Step S201: Obtain the trajectory point sequence collected when the user writes on the screen, and preprocess the trajectory point sequence.

[0089] When a user writes on the screen, a sequence of trajectory points can be collected. The information for each trajectory point in the sequence can include, but is not limited to, coordinates, time, thickness, pressure value, etc.

[0090] After obtaining the trajectory point sequence, it can be preprocessed, such as coordinate processing and resampling, to facilitate subsequent processing.

[0091] Step S202: Determine the table trajectory points and text trajectory points from the preprocessed trajectory point sequence.

[0092] The user-written content consists of two parts: a table and the text within the table. Correspondingly, the preprocessed trajectory point sequence includes table trajectory points and text trajectory points. The purpose of this step is to determine the table trajectory points and text trajectory points from the preprocessed trajectory point sequence.

[0093] Specifically, the preprocessed trajectory point sequence can be rendered into an image as the image to be detected. Then, the table region and several text regions in the table region can be detected from the image to be detected. Finally, the table trajectory points belonging to the table region and the text trajectory points belonging to each text region can be determined from the preprocessed trajectory point sequence.

[0094] Step S203: Render the table trajectory points into an image to obtain a pure table image, and render the text trajectory points into an image to obtain a text image.

[0095] Among them, a plain table image is an image containing a table (plain table) in the user's handwritten content, and a text image is an image containing text in the user's handwritten content.

[0096] Step S204: Perform table recognition on the pure table image to obtain the table recognition result, and perform text recognition on the text image to obtain the text recognition result.

[0097] Optionally, a plain table image can be input into a pre-trained table recognition model to obtain a table recognition result containing table structure information and cell location information. The table recognition model is trained using training table images labeled with table structure information and cell location information. It should be noted that the table structure information indicates the table's structure, such as the number of rows and columns.

[0098] Optionally, the text image can be input into a pre-trained text recognition model to obtain a text recognition result containing the text content. The text recognition model is trained using training text images labeled with text content.

[0099] Step S205: Match the table recognition result with the text recognition result to obtain the matching result.

[0100] The matching results indicate which part of the text matches which table cell.

[0101] Step S206: Based on the recognition information including table recognition results, text recognition results, and matching results, generate a table containing text content.

[0102] Specifically, the table can be reconstructed based on the table recognition results, and then text content can be filled into the reconstructed table based on the text recognition results and matching results, thus obtaining a table containing text content.

[0103] Optionally, after generating a table containing text content, the generated table containing text content can be displayed on the screen.

[0104] The table recognition method provided in this embodiment of the invention first acquires a sequence of trajectory points collected when the user writes on the screen, and preprocesses the trajectory point sequence. Then, it determines the table trajectory points and text trajectory points from the preprocessed trajectory point sequence. Next, it renders the table trajectory points and text trajectory points into images. Then, it performs table recognition on the rendered pure table image and text recognition on the rendered text image. Next, it matches the table recognition result with the text recognition result to obtain a matching result. Finally, based on the recognition information including the table recognition result, text recognition result, and matching result, it generates a table containing text content. The table recognition method provided in this embodiment of the invention can reconstruct a table containing text content written by the user. Since the table recognition method provided in this embodiment of the invention reconstructs a table containing text content written by the user by analyzing the writing trajectory point sequence, rather than by analyzing a table image (obtained by photographing a table written by the user on the screen), it does not have the problems existing in recognition methods based on table images. That is, the table recognition method provided in this embodiment of the invention does not have any limitations.

[0105] In another embodiment of the present invention, the steps of “preprocessing the trajectory point sequence” in step S201 and “step S202: determining the table trajectory points and text trajectory points from the preprocessed trajectory point sequence” are described.

[0106] After obtaining the trajectory point sequence, it is first preprocessed. Considering that the trajectory points need to be rendered into an image later, if the original coordinates of the trajectory points are directly rendered, the rendered image will have a lot of invalid blank space when the user writing area occupies a small area of ​​the screen. In order not to affect the subsequent processing effect, the trajectory point sequence can be traversed to obtain the trajectory point at the top left corner of the user writing area. Then, the coordinates of the top left corner trajectory point (x_l, y_l) are subtracted from the coordinates of all trajectory points in the trajectory point sequence to obtain the trajectory point sequence with (x_l, y_l) as the origin.

[0107] After obtaining the trajectory point sequence with (x_l, y_l) as the origin, a resampling algorithm can be used to resample the trajectory point sequence to make the trajectory point sequence smoother.

[0108] After obtaining the preprocessed trajectory point sequence, the table trajectory points and text trajectory points can be determined from the preprocessed trajectory point sequence. Please refer to [link / reference]. Figure 3 This illustrates a flowchart of the process for determining table trajectory points and text trajectory points from a preprocessed sequence of trajectory points, which may include:

[0109] Step S301: Render the preprocessed trajectory point sequence into an image, which is used as the image to be detected.

[0110] Step S302: Detect the table region and several text regions within the table region from the image to be detected.

[0111] It should be noted that the table area here refers to the entire area containing the table.

[0112] Optionally, a pre-trained detection model can be used to detect table regions and several text regions within the table regions from the image to be detected. The detection model can be trained using training images labeled with the table regions and several text regions within the table regions.

[0113] Specifically, the image to be detected can be input into the detection model. The detection model first detects several regions, then determines the category of each region (the category of a region is either text or table), and finally outputs the location information of the region with the category of table, i.e., the table region, and the location information of the region with the category of text, i.e., the text region. The location information of each region can be the diagonal coordinates (x1, y1, x2, y2) of the bounding rectangle of the region, where (x1, y1) is the coordinate of the upper left corner of the bounding rectangle, and (x2, y2) is the coordinate of the lower right corner of the bounding rectangle.

[0114] Step S303: From the preprocessed trajectory point sequence, determine the trajectory points belonging to the table area and the trajectory points belonging to each text area.

[0115] Specifically, the process of determining the trajectory points belonging to the table area and the trajectory points belonging to each text area from the preprocessed trajectory point sequence can include:

[0116] Step S3031: From the preprocessed trajectory point sequence, determine the trajectory point subsequences belonging to the table area and the trajectory point subsequences belonging to each text area.

[0117] The preprocessed trajectory point sequence contains several trajectory point sub-sequences, and each trajectory point sub-sequence is a trajectory point sequence that makes up a stroke.

[0118] Specifically, the process of determining the trajectory point subsequences belonging to the table region and the trajectory point subsequences belonging to each text region from the preprocessed trajectory point sequence can include:

[0119] For each region within the table region and several text regions, from the preprocessed trajectory point sequence, a trajectory point subsequence that meets a preset condition is identified. This subsequence is then designated as belonging to that region. The preset condition is that the proportion of trajectory points located in that region within the trajectory point sequence is greater than a preset threshold.

[0120] For example, the proportion threshold is 80%. If 85% of the trajectory points in a trajectory point sequence are located in a region, then the trajectory point sequence is determined to belong to that region.

[0121] It should be noted that whether a trajectory point is located in a region can be determined based on the coordinates of the trajectory point and the location information of the region.

[0122] Step S3032a: Determine the trajectory points contained in the trajectory point subsequence belonging to the table area as trajectory points belonging to the table area.

[0123] Since the table area includes both tables and text, the trajectory points belonging to the table area will include not only table trajectory points but also text trajectory points.

[0124] Step S3032b: For each text region, determine the trajectory points contained in the trajectory point subsequence belonging to that text region as the trajectory points belonging to that text region.

[0125] Step S304: Determine the trajectory points belonging to each text region as text trajectory points, and determine the table trajectory points from the trajectory points belonging to the table region.

[0126] Since the trajectory points belonging to the table area include not only table trajectory points but also text trajectory points, it is necessary to determine the table trajectory points from the trajectory points belonging to the table area.

[0127] There are several ways to determine table trajectory points from trajectory points belonging to the table area. In one possible implementation, trajectory points other than the determined text trajectory points in the preprocessed trajectory point sequence can be determined as table trajectory points.

[0128] To obtain more accurate table trajectory points, this embodiment provides another implementation method:

[0129] Traverse the trajectory point sequence belonging to the table area: Determine the number of trajectory points in each text area of ​​the currently traversed trajectory point sequence, obtaining the corresponding number of trajectory points for each text area; if the maximum number of trajectory points among the obtained number of trajectory points is greater than the target number, then the currently traversed trajectory point sequence can be determined as the trajectory point sequence that makes up the text strokes, and the currently traversed trajectory point sequence is filtered out; after the traversal is completed, the trajectory points contained in the remaining trajectory point sequence are determined as table trajectory points. The target number is the product of a set threshold (a value greater than 0 and less than 1) and the number of trajectory points in the currently traversed trajectory point sequence.

[0130] For example, the current traversed trajectory point subsequence includes 50 trajectory points. Assuming there are 3 text boxes (text box 1, text box 2, and text box 3), the number of trajectory points in the current traversed trajectory point subsequence located in text box 1 is 45, the number of trajectory points in text box 2 is 5, and the number of trajectory points in text box 3 is 0. If the maximum number 45 is greater than the target number (set threshold * 50), then it can be determined that the current traversed trajectory point subsequence is a trajectory point subsequence that makes up the text strokes, and therefore, it is filtered out.

[0131] In another embodiment of the present invention, the specific implementation process of "step S203: render the table trajectory points into an image to obtain a pure table image, and render the text trajectory points into an image to obtain a text image", "step S204: perform table recognition on the pure table image to obtain a table recognition result, and perform text recognition on the text image to obtain a text recognition result" and "step S205: match the table recognition result with the text recognition result to obtain a matching result" will be described.

[0132] After determining the table trajectory points and text trajectory points, image rendering can be performed. Specifically, the table trajectory points are rendered into images to obtain images that include the pure table, i.e., pure table images. For text trajectory points, since several text regions are involved, the text trajectory points of each text region are rendered into images during rendering to obtain the text images corresponding to each text region.

[0133] After obtaining a plain table image and several text images, the content in the images is further identified.

[0134] For a pure table image, the pure table image is input into a pre-trained table recognition model to obtain the table recognition result output by the table recognition model. The table recognition result includes table structure information and the position information of each table cell.

[0135] Optionally, the table structure information in the table recognition result can be the structured text `table_str`, where "" is an example of the structured text `table_str`, and "" appears in pairs, indicating the beginning of a table row and the end of a table row, with "" corresponding to a table cell. The position information of the table cell can be the diagonal coordinates (top-left and bottom-right corner coordinates) of the bounding rectangle of the table cell.

[0136] After obtaining the table recognition result through the table recognition model, a table structure array `array_table` can be constructed based on the result. This array `array_table` contains a series of data structure cells, each corresponding to a table cell and storing its information, including row and column information and position information. It's important to note that when constructing the `array_table`, the position information of the table cells in the table recognition result needs to be restored to the screen coordinate system; that is, the position information in each cell is the position information in the screen coordinate system.

[0137] For each text region, the text image corresponding to each text region is input into a pre-trained text recognition model to obtain the text recognition result containing the text content output by the text recognition model.

[0138] After obtaining the text content corresponding to several text regions, a text structure array (array) can be constructed based on the text content corresponding to the several text regions and the position information of the several text regions. _ text, array of text structures _ The text structure contains several data structures, each named text_struct. Each text_struct corresponds to a text region and stores information about that text region, including the text content, the text identifier, and the position information of the text region.

[0139] After obtaining the table recognition results and the text content corresponding to several text regions, it is necessary to match the text content corresponding to the several text regions with each table cell. During the matching process, each table cell can be traversed, and the intersection-union ratio (IUR) of the rectangle corresponding to the position information of the currently traversed table cell and the rectangle corresponding to the position information of each text region can be calculated to obtain the IUR corresponding to each text region. From the obtained IUR, the text content corresponding to the text region with an IUR greater than the set IUR threshold is determined as the text content that matches the currently traversed table cell.

[0140] More specifically, it iterates through each data structure cell in the table structure array `array_table`, and compares the bounding rectangle (the outer rectangle of the table cell) corresponding to the position information of the currently traversed cell with the text structure array `array_table`. _The intersection-over-union (IoU) ratio is calculated for the bounding rectangles (outer rectangles of the text content) corresponding to the position information in each `text_struct` data structure within the `text` dataset. The text content within the `text_struct` whose IoU ratio exceeds a set threshold is then identified as... The text content that matches the table cell corresponding to the currently traversed cell will have its text identifier in the text_struct corresponding to the intersection-union ratio (IU) that is greater than the set IU threshold stored in the currently traversed cell. In this way, each cell in the final table structure array array_table contains the row and column information of the table cell, the position information of the table cell, and the information of the text that matches the table cell.

[0141] Next, a table containing text content can be generated based on the table structure array array_table and the text structure array array_text. Optionally, the generated table containing text content can also be displayed on the screen.

[0142] It should be noted that the trajectory point sequence obtained in step S201 can be the trajectory point sequence collected up to the time when the user finishes writing. In this case, the table generated by steps S201 to S206 is the final table. In addition, the trajectory point sequence obtained in step S201 can also be the trajectory point sequence at the first recognition time in the real-time handwriting scenario. After obtaining the table corresponding to the first recognition time through steps S201 to S206, recognition can be performed every once in a while, thus realizing table recognition in the real-time handwriting scenario.

[0143] Next, based on the above embodiments, an example of a table recognition method suitable for real-time handwriting scenarios is provided. Please refer to [link / reference]. Figure 4 The diagram illustrates the flowchart of this table recognition method suitable for real-time handwriting scenarios, which may include:

[0144] Step S401: Obtain the trajectory point sequence collected up to the first identification time, and preprocess the obtained trajectory point sequence.

[0145] When a user writes on the screen, a sequence of trajectory points can be collected. In order to achieve online recognition, the trajectory point sequence is acquired and processed at regular intervals. In this step, "the sequence of trajectory points collected up to the first recognition time" refers to the first sequence of trajectory points acquired.

[0146] Step S402: Determine the table trajectory points and text trajectory points from the preprocessed trajectory point sequence.

[0147] Step S403: Render the table trajectory points into an image to obtain a pure table image, and render the text trajectory points into an image to obtain a text image.

[0148] Step S404: Perform table recognition on the pure table image to obtain the table recognition result, and perform text recognition on the text image to obtain the text recognition result.

[0149] Step S405: Match the table recognition result with the text recognition result to obtain the matching result.

[0150] Step S406: Based on the recognition information including table recognition results, text recognition results, and matching results, generate a table containing text content.

[0151] In step S406, "the recognition information that includes the table recognition result, the text recognition result and the matching result" is the recognition information corresponding to the first recognition time, and the table generated by step S406 is the table corresponding to the first recognition time.

[0152] The specific implementation process and related explanations of steps S401 to S406 can be found in the specific implementation process and related explanations of steps S201 to S206 above. This embodiment will not repeat them here.

[0153] Step S407: Every preset time interval, check whether the trajectory point sequence collected up to the current time has changed compared to the trajectory point sequence collected up to the previous identification time. If yes, execute steps S408 and S409; otherwise, execute step S407 until the user finishes writing.

[0154] Step S408: Based on the trajectory points that have changed from the trajectory point sequence collected up to the second identification time to the trajectory point sequence collected up to the previous identification time, and the identification information corresponding to the previous identification time, determine the identification information corresponding to the second identification time.

[0155] The second identification time is either the "current time" or a time after the "current time".

[0156] Assuming the previous identification time is t1, at time t2, which is separated from t1 by a preset time interval Δt, it is detected whether the trajectory point sequence collected up to time t2 has changed compared to the trajectory point sequence collected up to time t1. If there is a change, in one possible implementation, the identification information corresponding to time t2 can be determined based on the trajectory points that have changed compared to the trajectory point sequence collected up to time t1, and the identification information corresponding to time t1. To prevent identification from being performed before the data is fully written, in another possible implementation, it is possible to wait N*Δt (N is an integer greater than or equal to 1, such as 2). Assuming the time after waiting N*Δt is t3, the identification information corresponding to time t3 is determined based on the trajectory points that have changed compared to the trajectory point sequence collected up to time t1, and the identification information corresponding to time t1. The "current time" mentioned above is t2, and the second identification time is either t2 or t3, preferably t3.

[0157] Specifically, the process of determining the identification information corresponding to the second identification time based on the trajectory point sequence collected up to the second identification time compared to the trajectory point sequence collected up to the previous identification time, and the identification information corresponding to the previous identification time, may include:

[0158] Step S4081: Based on the position information of the trajectory points that have changed from the trajectory point sequence collected up to the second identification time to the trajectory point sequence collected up to the previous identification time, determine several change areas.

[0159] Specifically, the process of determining several changed regions based on the positional information of trajectory points that have changed from the trajectory point sequence collected up to the second identification time to the trajectory point sequence collected up to the previous identification time may include:

[0160] Step a1: Obtain the pre-built modification table.

[0161] In this embodiment, the size of the modification table can be the same as the screen size. However, this embodiment is not limited to this. For example, the size of the modification table can also be proportional to the screen size, and the position in the modification table corresponding to each pixel position on the screen is a first value, such as 0. For example, if the screen size is 200*100, then the size of the modification table is also 200*100, and the position in the modification table corresponding to each pixel position on the screen is 0.

[0162] Step a2: Modify the first value at the position corresponding to the changed trajectory point in the modification table to the second value to obtain the modified table.

[0163] For example, if there are 20 trajectory points in the trajectory point sequence collected up to the second identification time that have changed compared to the trajectory point sequence collected up to the previous identification time, then the 0 in the position corresponding to the position of these 20 trajectory points in the modification table will be changed to 1.

[0164] Step a3: Search for connected components with the second value from the modified modification table, and identify each connected component as a changed region.

[0165] Specifically, the modified table is traversed from left to right and from top to bottom. The first position `pl` where the value is the second value (e.g., 1) is found. `pl` is used as the top-left corner point. Starting from this point, the search is recursively performed in eight directions: up, down, left, right, top-left, bottom-left, top-right, and bottom-right, to find connected components with the second value (e.g., 1). The maximum x and y coordinates within each connected component are obtained to determine the bottom-right corner point `pr`. By traversing the modified table, several connected components and their top-left and bottom-right corners are obtained. Each connected component represents a changed region, and the coordinates of its top-left and bottom-right corners serve as the location information of the changed region. It's worth noting that during the traversal, to prevent duplicate traversals, values ​​in already traversed parts can be modified to the third value (e.g., 2).

[0166] Step S4082: If several changing areas include text changing areas, then determine the changing text content based on the text trajectory points belonging to the text changing areas. If several changing areas include table changing areas, then re-acquire the table recognition result based on the table trajectory points belonging to the table changing areas and other table trajectory points.

[0167] It should be noted that, in this embodiment, the text change area refers to the area where the text changes, and the table change area refers to the area where the table changes.

[0168] The process of identifying text and / or table variation areas from several variation areas may include: for each variation area, performing:

[0169] Step b1: Match the changed area with each text area and each table cell area in the table identified at the previous identification time.

[0170] Specifically, the intersection-union ratio (IUGR) of the changed region with each text region in the table identified at the previous recognition time is calculated, and the IUGR of the changed region with each table cell region in the table identified at the previous recognition time is also calculated. This yields the IUGR corresponding to each text region and the IUGR corresponding to each table cell region. If any of the obtained IUGRs is greater than a set threshold, and the region corresponding to the IUGR is a text region, then the changed region is determined to match the text region. If any of the obtained IUGRs is greater than a set threshold, and the region corresponding to the IUGR is a table cell region, then the changed region is determined to match the table cell region. If no IUGR is greater than a set threshold, then the changed region does not match any of the text regions or table cell regions in the table identified at the previous recognition time.

[0171] Specifically, when calculating the intersection-union ratio (IUU) of the changed region with each text region in the table identified at the previous identification time, the IUU of the bounding rectangle of the changed region's location information (the bounding rectangle of the changed region) and the bounding rectangle of the text content in each text_struct in the text structure array array_text corresponding to the previous identification time can be calculated. Similarly, when calculating the IUU of the changed region with each table cell region in the table identified at the previous identification time, the IUU of the bounding rectangle of the changed region's location information (the bounding rectangle of the changed region) and the bounding rectangle of the table cell in each cell in the table structure array array_table can be calculated.

[0172] Step b2-a: If the changed area matches the text area in the table identified at the previous identification time, then the changed area is determined to be a text changed area.

[0173] Step b2-b: If the changed area matches the table cell area identified at the previous identification time, then the changed area is determined to be the table changed area.

[0174] Step b2-c: If the changed area does not match any of the text areas and table cell areas in the table identified at the previous identification time, then the table area and text area are detected in the changed area based on the pre-trained detection model.

[0175] If several changing regions contain text changing regions, the changed text content is determined based on the text trajectory points belonging to the text changing regions. Specifically, the trajectory point sequence collected up to the second recognition time is preprocessed, and the text trajectory points belonging to the text changing regions are determined from the preprocessed trajectory point sequence. The text trajectory points belonging to the text changing regions are rendered into an image, and the image is recognized based on the text recognition model to obtain the changed text content.

[0176] If several changing regions include table changing regions, the table recognition result is re-acquired based on the table trajectory points belonging to the table changing regions and other table trajectory points. Specifically, table trajectory points belonging to the table changing regions are determined from the trajectory point sequence collected up to the second recognition time after preprocessing. Then, the table trajectory points belonging to the table changing regions and other table trajectory points are rendered into an image, and the image is recognized based on the table recognition model to obtain the table recognition result.

[0177] Step S4083: Based on the changed text content and / or the re-acquired table recognition results, update the recognition information corresponding to the previous recognition time, and use the updated recognition information as the recognition information corresponding to the second recognition time.

[0178] If all the changed regions are text changed regions, the recognition information corresponding to the previous recognition time is updated based on the changed text content (updating the text_struct corresponding to the text region that matches the changed text region in the text structure array_text). If all the changed regions are table changed regions, the recognition information corresponding to the previous recognition time is updated based on the re-acquired table recognition result (updating the table structure array_table). If the changed regions contain both text changed regions and table changed regions, the recognition information corresponding to the previous recognition time is updated based on the changed text content and the re-acquired table recognition result (updating the text_struct corresponding to the text region that matches the changed text region in the text structure array_text, and updating the table structure array_table).

[0179] Step S409: Based on the recognition information corresponding to the second recognition time, generate a table containing text content.

[0180] Based on the updated table structure array array_table and text structure array array_text, a table containing text content is generated.

[0181] The table recognition method provided in this embodiment of the invention can reconstruct a table containing text content written by a user. Since the table recognition method provided in this embodiment of the invention reconstructs a table containing text content written by a user by analyzing the sequence of writing trajectory points, rather than by analyzing a table image (a table image obtained by taking a picture of a table written by a user on a screen), it does not have the problems of recognition methods based on table images. At the same time, the table recognition method provided in this embodiment of the invention can realize table recognition in real-time handwriting scenarios.

[0182] This invention also provides a table recognition device. The table recognition device provided in this invention will be described below. The table recognition device described below can be referred to in correspondence with the table recognition method described above.

[0183] Please see Figure 5 The diagram shows a schematic of the structure of a table recognition device provided in an embodiment of the present invention. The table recognition device may include: a trajectory point data acquisition module 501, a trajectory point determination module 502, an image acquisition module 503, a recognition module 504, a matching module 505, and a table generation module 506.

[0184] The trajectory point data acquisition module 501 is used to acquire the trajectory point sequence collected when the user writes on the screen, and to preprocess the trajectory point sequence.

[0185] The trajectory point determination module 502 is used to determine the table trajectory points and text trajectory points from the preprocessed trajectory point sequence.

[0186] The image acquisition module 503 is used to render the table trajectory points into an image to obtain a pure table image, and to render the text trajectory points into an image to obtain a text image.

[0187] The recognition module 504 is used to perform table recognition on the pure table image to obtain table recognition results, and to perform text recognition on the text image to obtain text recognition results.

[0188] The matching module 505 is used to match the table recognition result with the text recognition result to obtain a matching result.

[0189] The table generation module 506 is used to generate a table containing text content based on the recognition information including the table recognition result, the text recognition result, and the matching result.

[0190] Optionally, the trajectory point data acquisition module 501 acquires a trajectory point sequence that is a sequence of trajectory points collected up to the first identification time, and the identification information is the identification information corresponding to the first identification time.

[0191] The table recognition device may also include: a trajectory point change detection module and a recognition information update module.

[0192] The trajectory point change detection module is used to detect, at preset time intervals, whether the trajectory point sequence collected up to the current time has changed compared to the trajectory point sequence collected up to the previous identification time.

[0193] The identification information update module is used to determine the identification information corresponding to the second identification time based on the changed trajectory points and the identification information corresponding to the previous identification time when the trajectory point sequence collected up to the current time is different from the trajectory point sequence collected up to the previous identification time. The second identification time is the current time or a time after the current time.

[0194] The table generation module is also used to generate a table containing text content based on the recognition information corresponding to the second recognition time.

[0195] Optionally, the trajectory point determination module 502 may include: a trajectory point rendering submodule, a detection submodule, a first trajectory point determination submodule, and a second trajectory point determination submodule.

[0196] The trajectory point rendering submodule is used to render the preprocessed trajectory point sequence into an image, which is used as the image to be detected.

[0197] The detection submodule is used to detect a table region and several text regions within the table region from the image to be detected.

[0198] The first trajectory point determination submodule is used to determine the trajectory points belonging to the table area and the trajectory points belonging to each text area from the preprocessed trajectory point sequence.

[0199] The second trajectory point determination submodule is used to determine the table trajectory points from the trajectory points belonging to the table area, and to determine the trajectory points belonging to each text area as text trajectory points.

[0200] Optionally, when the first trajectory point determination submodule determines the trajectory points belonging to the table region and the trajectory points belonging to each text region from the preprocessed trajectory point sequence, it is specifically used for:

[0201] From the preprocessed trajectory point sequence, determine the trajectory point sequence belonging to the table area and the trajectory point sequences belonging to each text area, wherein each trajectory point sequence is a trajectory point sequence that makes up a stroke.

[0202] The trajectory points contained in the sequence of trajectory points belonging to the table area are determined as the trajectory points belonging to the table area;

[0203] For each text region, the trajectory points contained in the trajectory point subsequence belonging to that text region are determined as the trajectory points belonging to that text region.

[0204] Optionally, when the first trajectory point determination submodule determines the trajectory point subsequence belonging to the table region and the trajectory point subsequences belonging to each text region from the preprocessed trajectory point sequence, it is specifically used for:

[0205] For each of the table area and the plurality of text areas:

[0206] From the preprocessed trajectory point sequence, a trajectory point subsequence that meets a preset condition is determined, wherein the preset condition is that the proportion of trajectory points located in the region in the trajectory point sequence is greater than a preset proportion threshold.

[0207] The sequence of trajectory points that meets the preset conditions is determined as the sequence of trajectory points belonging to this region.

[0208] Optionally, when the second trajectory point determination submodule determines the table trajectory point from the trajectory points belonging to the table area, it is specifically used for:

[0209] Traverse the sequence of trajectory points belonging to the table region:

[0210] Determine the number of trajectory points in each text region of the currently traversed trajectory point subsequence, and obtain the corresponding number of trajectory points for each text region;

[0211] If the maximum number of trajectory points among the obtained number of trajectory points is greater than the target number, the current traversed trajectory point subsequence will be filtered out, wherein the target number is the product of a set threshold and the number of trajectory points in the current traversed trajectory point subsequence.

[0212] After the traversal is complete, the trajectory points contained in the remaining trajectory point subsequence are determined as the table trajectory points.

[0213] When rendering the text trajectory points into images to obtain text images, the image acquisition module 503 is specifically used to render the text trajectory points of each text region into images respectively, so as to obtain the text images corresponding to each text region.

[0214] The recognition module 504 includes: a pure table recognition submodule and a text content recognition submodule.

[0215] The pure table recognition submodule is used to input the pure table image into a pre-trained table recognition model to obtain a table recognition result containing table structure information and the position information of each table cell. The table recognition model is trained using a training table image labeled with table structure information and the position information of each table cell.

[0216] The text content recognition submodule is used to input the text image corresponding to each text region into a pre-trained text recognition model to obtain a text recognition result containing the text content corresponding to the text region. The text recognition model is trained using training text images labeled with text content.

[0217] Optionally, the identification information update module includes: a change area determination submodule, a change area processing submodule, and an identification information update submodule.

[0218] The variable region determination submodule is used to determine several variable regions based on the position information of trajectory points that have changed from the trajectory point sequence collected up to the second identification time to the trajectory point sequence collected up to the previous identification time.

[0219] The variable region processing submodule is used to determine the changed text content based on the text trajectory points belonging to the text variable region if the plurality of variable regions include a text variable region, and to re-acquire the table recognition result based on the table trajectory points belonging to the table variable region and other table trajectory points if the plurality of variable regions include a table variable region.

[0220] The identification information update submodule is used to update the identification information corresponding to the previous identification time based on the changed text content and / or the re-acquired table identification results, and the updated identification information is used as the identification information corresponding to the second identification time.

[0221] Optionally, when the variable region determination submodule determines several variable regions based on the position information of trajectory points that have changed from the trajectory point sequence collected up to the second identification time to the trajectory point sequence collected up to the previous identification time, it is specifically used for:

[0222] Obtain a pre-constructed modification table, wherein the position in the modification table corresponding to each pixel position of the screen is a first value;

[0223] Modify the first value at the position corresponding to the changed trajectory point in the modification table to the second value;

[0224] Search for connected components in the modified table that have the value of the second value, and identify each connected component as a variable region.

[0225] Optionally, the variable area processing submodule is further configured to determine text variable areas and / or table variable areas from the plurality of variable areas.

[0226] When determining text change regions and / or table change regions from the plurality of change regions, the change region processing submodule is specifically used for:

[0227] For each affected area:

[0228] The changed area is matched with each text area and each table cell area in the table identified at the previous identification time.

[0229] If the changed area matches the text area, then the changed area is determined to be a text changed area;

[0230] If the changed area matches a table cell range, then the changed area is determined to be a changed area of ​​the table.

[0231] If the changed region does not match any of the text regions and table cell regions identified in the previous identification time, then the table region and text region are detected based on the pre-trained detection model.

[0232] The table recognition device provided in this embodiment of the invention can reconstruct a table containing text content written by a user by analyzing the sequence of writing trajectory points. Since it is not reconstructed by analyzing a table image (a table image obtained by taking a picture of a table written by a user on a screen), it does not have the problems of recognition methods based on table images. At the same time, the table recognition method provided in this embodiment of the invention can realize table recognition in real-time handwriting scenarios.

[0233] This invention also provides a processing device; please refer to [link / reference]. Figure 6 The diagram shows the structure of the processing device, which may include: a processor 601, a communication interface 602, a memory 603, and a communication bus 604.

[0234] In this embodiment of the invention, the number of processor 601, communication interface 602, memory 603, and communication bus 604 is at least one, and processor 601, communication interface 602, and memory 603 communicate with each other through communication bus 604.

[0235] The processor 601 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0236] The memory 603 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0237] The memory stores a program, which the processor can call. The program is used for:

[0238] Acquire the trajectory point sequence collected when the user writes on the screen, and preprocess the trajectory point sequence;

[0239] The table trajectory points and text trajectory points are determined from the preprocessed trajectory point sequence;

[0240] The table trajectory points are rendered into an image to obtain a pure table image, and the text trajectory points are rendered into an image to obtain a text image;

[0241] Perform table recognition on the pure table image to obtain table recognition results, and perform text recognition on the text image to obtain text recognition results;

[0242] The table recognition result is matched with the text recognition result to obtain the matching result;

[0243] Based on the recognition information including the table recognition result, the text recognition result, and the matching result, a table containing text content is generated.

[0244] Optionally, the refined and extended functions of the program can be found in the description above.

[0245] This invention also provides a readable storage medium that stores a program suitable for execution by a processor, the program being used for:

[0246] Acquire the trajectory point sequence collected when the user writes on the screen, and preprocess the trajectory point sequence;

[0247] The table trajectory points and text trajectory points are determined from the preprocessed trajectory point sequence;

[0248] The table trajectory points are rendered into an image to obtain a pure table image, and the text trajectory points are rendered into an image to obtain a text image;

[0249] Perform table recognition on the pure table image to obtain table recognition results, and perform text recognition on the text image to obtain text recognition results;

[0250] The table recognition result is matched with the text recognition result to obtain the matching result;

[0251] Based on the recognition information including the table recognition result, the text recognition result, and the matching result, a table containing text content is generated.

[0252] Optionally, the refined and extended functions of the program can be found in the description above.

[0253] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0254] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0255] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A table recognition method, characterized in that, include: Obtain the trajectory point sequence collected when the user writes on the screen, and preprocess the trajectory point sequence; wherein, the preprocessing of the trajectory point sequence includes: traversing the trajectory point sequence to obtain the trajectory point at the upper left corner of the user's writing area, subtracting the coordinates of the upper left corner trajectory point from the coordinates of all trajectory points in the trajectory point sequence to obtain the trajectory point sequence with the upper left corner trajectory point as the origin of the coordinates; The table trajectory points and text trajectory points are determined from the preprocessed trajectory point sequence; The table trajectory points are rendered into an image to obtain a pure table image, and the text trajectory points are rendered into an image to obtain a text image; Perform table recognition on the pure table image to obtain table recognition results, and perform text recognition on the text image to obtain text recognition results; The table recognition result is matched with the text recognition result to obtain the matching result; Based on the recognition information including the table recognition result, the text recognition result, and the matching result, a table containing text content is generated.

2. The table recognition method according to claim 1, characterized in that, The trajectory point sequence is the trajectory point sequence collected up to the first identification time, and the identification information is the identification information corresponding to the first identification time; The table recognition method also includes: Every preset time interval, it checks whether the trajectory point sequence collected up to the current time has changed compared to the trajectory point sequence collected up to the previous identification time; If so, then based on the trajectory points that have changed from the trajectory point sequence collected up to the second identification time to the trajectory point sequence collected up to the previous identification time, and the identification information corresponding to the previous identification time, the identification information corresponding to the second identification time is determined, wherein the second identification time is the current time or a time after the current time; Based on the recognition information corresponding to the second recognition time, a table containing text content is generated.

3. The table recognition method according to claim 1 or 2, characterized in that, The step of determining the table trajectory points and text trajectory points from the preprocessed trajectory point sequence includes: The preprocessed trajectory point sequence is rendered into an image, which is used as the image to be detected. Detect a table region and several text regions within the table region from the image to be detected; From the preprocessed sequence of trajectory points, determine the trajectory points belonging to the table area and the trajectory points belonging to each text area respectively; The table trajectory points are determined from the trajectory points belonging to the table area, and the trajectory points belonging to each text area are determined as text trajectory points.

4. The table recognition method according to claim 3, characterized in that, The step of determining the trajectory points belonging to the table region and the trajectory points belonging to each text region from the preprocessed trajectory point sequence includes: From the preprocessed trajectory point sequence, determine the trajectory point sequence belonging to the table area and the trajectory point sequences belonging to each text area, wherein each trajectory point sequence is a trajectory point sequence that makes up a stroke. The trajectory points contained in the sequence of trajectory points belonging to the table area are determined as the trajectory points belonging to the table area; For each text region, the trajectory points contained in the trajectory point subsequence belonging to that text region are determined as the trajectory points belonging to that text region.

5. The table recognition method according to claim 4, characterized in that, The step of determining the trajectory point subsequences belonging to the table region and the trajectory point subsequences belonging to each text region from the preprocessed trajectory point sequence includes: For each of the table area and the plurality of text areas: From the preprocessed trajectory point sequence, a trajectory point subsequence that meets a preset condition is determined, wherein the preset condition is that the proportion of trajectory points located in the region in the trajectory point sequence is greater than a preset proportion threshold. The sequence of trajectory points that meets the preset conditions is determined as the sequence of trajectory points belonging to this region.

6. The table recognition method according to claim 3, characterized in that, The step of rendering the text trajectory points into an image to obtain a text image includes: Render the text trajectory points of each text region into images to obtain the text images corresponding to each text region; Perform table recognition on the pure table image to obtain table recognition results, and perform text recognition on the text image to obtain text recognition results, including: The pure table image is input into a pre-trained table recognition model to obtain a table recognition result containing table structure information and the position information of each table cell. The table recognition model is trained using a training table image labeled with table structure information and the position information of each table cell. For each text region, the text image corresponding to that text region is input into a pre-trained text recognition model to obtain a text recognition result containing the text content corresponding to that text region. The text recognition model is trained using training text images labeled with text content.

7. The table recognition method according to claim 2, characterized in that, The determination of the identification information corresponding to the second identification time based on the trajectory point sequence collected up to the second identification time compared to the trajectory point sequence collected up to the previous identification time, and the identification information corresponding to the previous identification time, includes: Based on the positional information of trajectory points that have changed from the trajectory point sequence collected up to the second identification time to the trajectory point sequence collected up to the previous identification time, several areas of change are identified. If the plurality of variable regions include text variable regions, then the variable text content is determined based on the text trajectory points belonging to the text variable regions. If the plurality of changing regions include a table changing region, then the table recognition result is re-acquired based on the table trajectory points belonging to the table changing region and other table trajectory points; Based on the changed text content and / or the re-acquired table recognition results, the recognition information corresponding to the previous recognition time is updated, and the updated recognition information is used as the recognition information corresponding to the second recognition time.

8. The table recognition method according to claim 7, characterized in that, Based on the positional information of trajectory points that have changed from the trajectory point sequence collected up to the second identification time to the trajectory point sequence collected up to the previous identification time, several change regions are determined, including: Obtain a pre-constructed modification table, wherein the position in the modification table corresponding to each pixel position of the screen is a first value; Modify the first value at the position corresponding to the changed trajectory point in the modification table to the second value; Search for connected components in the modified table that have the value of the second value, and identify each connected component as a variable region.

9. The table recognition method according to claim 7, characterized in that, Determining text-modified regions and / or table-modified regions from the aforementioned modified regions includes: For each affected area: The changed area is matched with each text area and each table cell area in the table identified at the previous identification time. If the changed area matches the text area, then the changed area is determined to be a text changed area; If the changed area matches a table cell range, then the changed area is determined to be a changed area of ​​the table. If the changed region does not match any of the text regions and table cell regions identified in the previous identification time, then the table region and text region are detected based on the pre-trained detection model.

10. A form recognition device, characterized in that, include: The system includes a trajectory point data acquisition module, a trajectory point determination module, an image acquisition module, a recognition module, a matching module, and a table generation module. The trajectory point data acquisition module is used to acquire the trajectory point sequence collected when the user writes on the screen, and to preprocess the trajectory point sequence; specifically, when preprocessing the trajectory point sequence, the trajectory point data acquisition module is used to traverse the trajectory point sequence to obtain the trajectory point at the upper left corner of the user's writing area, and to subtract the coordinates of the upper left corner trajectory point from the coordinates of all trajectory points in the trajectory point sequence to obtain a trajectory point sequence with the upper left corner trajectory point as the origin of the coordinates; The trajectory point determination module is used to determine table trajectory points and text trajectory points from the preprocessed trajectory point sequence; The image acquisition module is used to render the table trajectory points into an image to obtain a pure table image, and to render the text trajectory points into an image to obtain a text image; The recognition module is used to perform table recognition on the pure table image to obtain table recognition results, and to perform text recognition on the text image to obtain text recognition results; The matching module is used to match the table recognition result with the text recognition result to obtain a matching result; The table generation module is used to generate a table containing text content based on recognition information including the table recognition result, the text recognition result, and the matching result.

11. A processing apparatus, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement each step of the table recognition method as described in any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements each step of the table recognition method as described in any one of claims 1 to 9.

Images