Display method and device of hand-drawn table, electronic equipment and storage medium

Abstract

The invention provides a hand-drawn table display method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a hand-drawn table drawn by a user, and converting the hand-drawn table into a table image; performing table element segmentation on the table image to obtain table elements in the table image; determining coordinate information of text elements and coordinate information of frame line elements in the table image; determining an inclination angle of each line of text in the table image based on the coordinate information of the text element, and adjusting the inclination angle of the text in the table image to a target inclination angle based on the inclination angle; fitting the coordinate points of each frame line element in the table image based on the coordinate information of the frame line element to obtain a beautified frame line, and replacing the frame line in the table image with the beautified frame line; and displaying the adjusted and beautified freehand form on the electronic equipment. According to the method, the aesthetic property of the hand-drawn table can be effectively improved.

Description

A display method and device of a hand-drawn table, an electronic device, and a storage medium TECHNICAL FIELD

[0001] The present application relates to the computer field, and more particularly, to a display method and device of a hand-drawn table, an electronic device, and a storage medium. BACKGROUND

[0002] Currently, in teaching or meeting scenarios, drawing a hand-drawn table in a touch panel is a common content display method. However, when a user draws a hand-drawn table in a touch panel, problems such as distorted table lines and insufficient cell writing area in the table may occur, which affect the aesthetics of the table.

[0003] SUMMARY

[0004] The present application provides a display method and device of a hand-drawn table, an electronic device, and a storage medium, which can beautify a hand-drawn table drawn by a user and effectively improve the aesthetics of the hand-drawn table.

[0005] In a first aspect, a display method of a hand-drawn table is provided. The method includes obtaining a hand-drawn table drawn by a user and converting the hand-drawn table into a table image; performing table element segmentation on the table image to obtain table elements in the table image; wherein the table elements in the table image include text elements and frame line elements; determining coordinate information of the text elements and coordinate information of the frame line elements in the table image; determining an inclination angle of each row of text in the table image based on the coordinate information of the text elements, and adjusting the inclination angle of the text in the table image to a target inclination angle based on the inclination angle; wherein the inclination angle is an inclination angle of the each row of text relative to a reference line; fitting coordinate points of each frame line element in the table image based on the coordinate information of the frame line elements to obtain a beautified frame line, and replacing the frame line in the table image with the beautified frame line; and displaying the hand-drawn table after the adjustment and beautification on an electronic device.

[0006] The technical solution described above, by obtaining a hand-drawn table drawn by a user and converting the hand-drawn table into a table image, can retain the personalized elements of the user's hand-drawn table in the converted table image. By performing table element segmentation on the table image, various table elements in the table image can be obtained. The table elements include text elements and frame line elements, and then the coordinate information of the text elements and the coordinate information of the frame line elements in the table image are determined, which can provide accurate basis for subsequent beautification of the table image. Based on the coordinate information of the text elements, the inclination angle of each row of text in the table image is determined, and based on the inclination angle, the inclination angle of the text in the table image is adjusted to a target inclination angle, which can ensure that all text rows in the table image maintain a consistent target inclination angle, so that the final displayed table is more regular and professional, facilitating reading and quick information capture. In addition, by fitting the coordinate points of the frame line elements, a beautified frame line is generated, which can eliminate the jaggedness, burrs or discontinuity of the original frame line, making the table boundary clearer and smoother, and significantly improving the overall aesthetic of the table image. Finally, the hand-drawn table after adjustment and beautification is displayed on the electronic device, improving the user experience.

[0007] In combination with the first aspect, in some implementations of the first aspect, the converting the hand-drawn table into a table image comprises: obtaining trajectory data when the user draws the hand-drawn table; based on the trajectory data, obtaining a plurality of drawing trajectories in the hand-drawn table; obtaining time sequence information of each of the plurality of drawing trajectories; and rendering the plurality of drawing trajectories according to the time sequence information of each of the plurality of drawing trajectories to obtain the table image.

[0008] The technical solution described above obtains the trajectory when the user draws the hand-drawn table, obtains a plurality of drawing trajectories in the hand-drawn table based on the trajectory data, and renders each drawing trajectory in the order of drawing according to the time sequence information, which can reflect the time sequence information of the drawing trajectory in the table image, providing accurate reference for subsequent table element segmentation and effectively improving the accuracy of table element segmentation.

[0009] In combination with the first aspect and the above implementations, in some implementations of the first aspect, after performing table element segmentation on the table image to obtain table elements in the table image, the method further comprises: determining whether the table elements include a critical text element; wherein the critical text element is a text element having a position overlap area with at least two cell elements; in a case where it is determined that the critical text element exists, determining time sequence information of a drawing trajectory of the critical text element based on the table image; based on the time sequence information, determining a target cell to which the critical text element belongs among the at least two cell elements; and based on the target cell, determining coordinate information of the critical text element in the table image.

[0010] The technical solution determines the cell to which the critical text element belongs based on the timing information of the drawing trajectory of the critical text element, effectively improving the accuracy and stability of determining the attribution of the critical text.

[0011] In combination with the first aspect and the implementation manners described above, in some implementation manners of the first aspect, the rendering of the plurality of drawing trajectories according to the timing information of each of the plurality of drawing trajectories to obtain the table image comprises: determining the gray value of the plurality of drawing trajectories in the table image according to the timing information of each of the plurality of drawing trajectories; and rendering the plurality of drawing trajectories based on the gray value to obtain the table image.

[0012] The technical solution determines the gray value of the plurality of drawing trajectories in the table image according to the timing information of each of the drawing trajectories, and renders the plurality of drawing trajectories based on the gray value, so that the order of drawing of each drawing trajectory can be visually observed in the table image, which is beneficial to providing a basis for subsequent table element segmentation.

[0013] In combination with the first aspect and the implementation manners described above, in some implementation manners of the first aspect, the table element segmentation of the table image to obtain the table element in the table image comprises: inputting the table image into a table element segmentation model to obtain the table element in the table image; and the table element segmentation model is used to segment the input table image into a plurality of table elements, and the table element segmentation model is obtained based on a table database.

[0014] In combination with the first aspect and the implementation manners described above, in some implementation manners of the first aspect, the table database is generated by: determining a table matrix of a target table to be generated according to a preset number of rows and a preset number of columns; each element in the table matrix corresponds to a cell in the target table; traversing each element in the table matrix to set the characteristic attribute of the cell corresponding to the element; determining a target frame line matching the characteristic attribute of the cell according to a preset frame line library; determining target text used to fill in the cell in a preset text library according to the characteristic attribute of the cell; generating the target table according to the target frame line and the target text; taking the position information of the target frame line and the filling area of the target text as annotation information of the target table; and generating the table database based on the target table and the annotation information.

[0015] The technical solution can automatically synthesize table data and annotations based on the constituent elements of the table, and the collection and annotation of the table data need to be performed manually, thereby effectively ensuring the richness and effectiveness of the table database and reducing the data cost.

[0016] In some implementations of the first aspect, in combination with the first aspect and the above implementations, the characteristic attribute of the cell includes a size attribute and a coordinate attribute; determining, according to the preset frame line library, a target frame line matching the characteristic attribute of the cell includes: selecting, in the preset frame line library, the target frame line matching the size attribute of the cell and matching the coordinate attribute of the cell; or selecting a random frame line in the preset frame line library, and adjusting the random frame line based on the size attribute and the coordinate attribute to obtain the target frame line; wherein the target frame line matches the size attribute of the cell and matches the coordinate attribute of the cell; determining, according to the characteristic attribute of the cell, a target text in a preset text library for filling into the cell includes: selecting, in the preset text library, the target text matching the coordinate attribute of the cell.

[0017] In some implementations of the first aspect, in combination with the first aspect and the above implementations, the table element segmentation model includes a feature extraction module and an element segmentation module, the element segmentation module includes multiple branches for segmenting different types of table elements, table elements of the same type correspond to the same branch, and inputting the table image into the table element segmentation model to obtain table elements in the table image includes: performing multi-level feature extraction on the table image through the feature extraction module, and fusing the extracted multi-level features to obtain a high-dimensional feature map; performing feature analysis on the high-dimensional feature map through the element segmentation module to determine the type of each feature point in the high-dimensional feature map; inputting each feature point into a branch corresponding to the type of the feature point to segment the high-dimensional feature map and obtain table elements of the table image.

[0018] The above technical solution can capture complex structural information of a table, including row and column division, cell boundaries, merged cells, etc., by performing multi-level feature extraction and fusion on a table image to generate a high-dimensional feature map, ensuring accurate identification and restoration of the table structure. The element segmentation module performs feature analysis on the high-dimensional feature map to determine the type information of each feature point (which may correspond to a pixel or region in the table image), achieving automatic classification of table elements and greatly improving processing efficiency and accuracy. Inputting the high-dimensional feature map into a multi-branch structure corresponding to the type information enables targeted segmentation of different types of table elements, which can enable the model to learn multiple table element segmentation tasks simultaneously during training, improving learning efficiency and model performance.

[0019] In some implementations of the first aspect, the plurality of branches includes a first branch for segmenting a target element, the target element including a text element or a cell element, and the inputting the each feature point into the branch corresponding to the type of the feature point to segment the high-dimensional feature map to obtain the table element of the table image includes: inputting the high-dimensional feature map into the first branch to predict an overall region and a center region of the target element; determining a center point and a target number of the target element based on the center region of the target element; and taking the center point of the target element as an aggregation center to aggregate pixels in the overall region of the target element to obtain a plurality of sub-regions, wherein the number of the sub-regions is the target number, and each of the sub-regions corresponds to a target element; and obtaining the target element in the table image based on the plurality of sub-regions.

[0020] The above technical solution effectively identifies and extracts the cell element and the text element in the table image from the high-dimensional feature map through accurate prediction, adaptive clustering and segmentation, realizes accurate reconstruction and structured output of the table structure, and has high recognition accuracy and strong adaptability.

[0021] In some implementations of the first aspect, the plurality of branches includes a second branch for segmenting the frame line element, the second branch including a spatial attention network, and the spatial attention network includes a frame line feature extraction module, a horizontal fusion module and a vertical fusion module; and the inputting the each feature point into the branch corresponding to the type of the feature point to segment the high-dimensional feature map to obtain the table element of the table image includes: inputting the high-dimensional feature map into the frame line feature extraction module to obtain a frame line fusion feature map; inputting feature values corresponding to each row of feature points in the frame line fusion feature map into the horizontal fusion module as a sequence to perform feature fusion to obtain a horizontal frame line element in the frame line element; and inputting feature values corresponding to each column of feature points in the frame line fusion feature map into the vertical fusion module as a sequence to perform feature fusion to obtain a vertical frame line element in the frame line element.

[0022] The above technical solution can accurately identify the frame line in the table image through multi-scale feature fusion by the spatial attention network and sequence feature capture by the fusion module, adapt to complex table layouts, and effectively improve the accuracy of table structure analysis.

[0023] In some implementations of the first aspect, the adjusting the inclination angle of the text in the table image to a target inclination angle based on the inclination angle includes: performing affine transformation on the text to adjust a height of the text to a target height and adjust the inclination angle of the text in the table image to the target inclination angle based on the inclination angle.

[0024] The technical solution can significantly improve the neatness and consistency of the table text, and does not process the handwriting and other personalized features of the user-drawn table, so that the personalized features of the user-drawn table can be retained while beautifying the table image.

[0025] With reference to the first aspect and the implementation forms above, in some implementation forms of the first aspect, based on the coordinate information of the frame line elements in the table element, the coordinate points of each frame line element in the table image are fitted to obtain a beautified frame line, including that the fitting includes at least one of frame line alignment, completion and smoothing operations in the table image.

[0026] With reference to the first aspect and the implementation forms above, in some implementation forms of the first aspect, before the adjusted and beautified user-drawn table is displayed on the electronic device, the method further includes: converting the font format corresponding to the text element into a preset font format.

[0027] The technical solution can make all the texts in the table image look more standardized, coordinated and consistent, and improve the overall visual effect of the table image.

[0028] The second aspect provides a display device of a user-drawn table, including: an acquisition module configured to acquire a user-drawn table and convert the user-drawn table into a table image; a segmentation module configured to perform table element segmentation on the table image to obtain table elements in the table image; wherein the table elements in the table image include text elements and frame line elements; a determination module configured to determine coordinate information of the text elements and coordinate information of the frame line elements in the table image; a beautification module configured to determine an inclination angle of each row of text in the table image based on the coordinate information of the text elements, and adjust the inclination angle of the text in the table image to a target inclination angle based on the inclination angle; based on the coordinate information of the frame line elements, fit coordinate points of each frame line element in the table image to obtain a beautified frame line, and replace the frame line in the table image with the beautified frame line; wherein the inclination angle is an inclination angle of each row of text relative to a reference line; and a display module configured to display the adjusted and beautified user-drawn table on an electronic device.

[0029] With reference to the second aspect, in some implementations of the second aspect, the obtaining module is specifically configured to: obtain trajectory data of the user when drawing the hand-drawn table; obtain a plurality of drawing trajectories in the hand-drawn table based on the trajectory data; obtain time sequence information of each of the plurality of drawing trajectories; and render the plurality of drawing trajectories according to the time sequence information of each of the plurality of drawing trajectories to obtain the table image.

[0030] With reference to the second aspect and the above implementations, in some implementations of the second aspect, the device further includes a judging module, which is specifically configured to: judge whether the table element contains a critical text element; the critical text element is a text element having a position overlap area with at least two cell elements; in a case where it is determined that the critical text element exists, determine time sequence information of a drawing trajectory of the critical text element according to the table image; determine a target cell to which the critical text element belongs among the at least two cell elements based on the time sequence information; and determine coordinate information of the critical text element in the table image based on the target cell.

[0031] With reference to the second aspect and the above implementations, in some implementations of the second aspect, the obtaining module includes a rendering unit, which is specifically configured to: determine a gray value of a plurality of drawing trajectories in the table image according to the time sequence information of each of the plurality of drawing trajectories; and render the plurality of drawing trajectories based on the gray value to obtain the table image.

[0032] With reference to the second aspect and the above implementations, in some implementations of the second aspect, the splitting module is specifically configured to: input the table image into a table element splitting model to obtain table elements in the table image; the table element splitting model is used to split the input table image into a plurality of table elements, and the table element splitting model is obtained based on a table database.

[0033] With reference to the second aspect and the above implementations, in some implementations of the second aspect, the splitting module further includes a table database generation module, which is specifically configured to: determine a table matrix of a target table to be generated according to a preset number of rows and a preset number of columns; each element in the table matrix corresponds to a cell in the target table; traverse each element in the table matrix to set a feature attribute of the cell corresponding to the element; determine a target frame line matching the feature attribute of the cell according to a preset frame line library; determine a target text used to fill in the cell in a preset text library according to the feature attribute of the cell; generate the target table according to the target frame line and the target text; take position information of the target frame line and a filling area of the target text as annotation information of the target table; and generate the table database based on the target table and the annotation information.

[0034] With reference to the second aspect and the foregoing implementation manners, in some embodiments of the second aspect, the characteristic attribute of the cell includes a size attribute and a coordinate attribute; the table database generation module includes a selection unit, which is specifically configured to: select, from the preset frame line library, the target frame line that matches the size attribute of the cell and matches the coordinate attribute of the cell; or select a random frame line from the preset frame line library, and adjust the random frame line based on the size attribute and the coordinate attribute to obtain the target frame line; wherein the target frame line matches the size attribute of the cell and matches the coordinate attribute of the cell; and the target text used for filling into the cell is determined from the preset text library based on the characteristic attribute of the cell, including: selecting, from the preset text library, the target text that matches the coordinate attribute of the cell.

[0035] With reference to the second aspect and the foregoing implementation manners, in some embodiments of the second aspect, the table element segmentation model includes a feature extraction module and an element segmentation module, the element segmentation module includes a plurality of branches for segmenting different types of table elements, the same type of table element corresponds to the same branch, and the segmentation module is specifically configured to: perform multi-level feature extraction on the table image through the feature extraction module, fuse the extracted multi-level features, and obtain a high-dimensional feature map; perform feature analysis on the high-dimensional feature map through the element segmentation module to determine the type of each feature point in the high-dimensional feature map; input each feature point into a branch corresponding to the type of the feature point to segment the high-dimensional feature map and obtain the table elements of the table image.

[0036] With reference to the second aspect and the foregoing implementation manners, in some embodiments of the second aspect, the plurality of branches include a first branch for segmenting a target element, the target element including a text element or a cell element, and the segmentation module includes a first segmentation unit, which is specifically configured to: input the high-dimensional feature map into the first branch to predict an overall region and a center region of the target element; determine a center point and a target number of the target element based on the center region of the target element; aggregate pixels in the overall region of the target element with the center point of the target element as an aggregation center to obtain a plurality of sub-regions; wherein the number of the sub-regions is the target number, and each of the sub-regions corresponds to a target element; and obtain the target element in the table image based on the plurality of sub-regions.

[0037] With reference to the second aspect and the foregoing implementation manners, in some implementations of the second aspect, the plurality of branches include a second branch for segmenting the frame line element, the second branch including a spatial attention network including: a frame line feature extraction module, a horizontal fusion module, and a vertical fusion module; and the segmentation module includes a second segmentation unit specifically configured to: input the high-dimensional feature map into the frame line feature extraction module to obtain a frame line fusion feature map; input feature values corresponding to each row of feature points in the frame line fusion feature map as a sequence into the horizontal fusion module for feature fusion to obtain a horizontal frame line element in the frame line element; and input feature values corresponding to each column of feature points in the frame line fusion feature map as a sequence into the vertical fusion module for feature fusion to obtain a vertical frame line element in the frame line element.

[0038] With reference to the second aspect and the foregoing implementation manners, in some implementations of the second aspect, the beautifying module is specifically configured to: based on the inclination angle, perform affine transformation on the text to adjust a height of the text to a target height and adjust an inclination angle of the text in the table image to a target inclination angle.

[0039] With reference to the second aspect and the foregoing implementation manners, in some implementations of the second aspect, the beautifying module is further configured to: the fitting includes at least one of frame line alignment, completion, and smoothing operation in the table image.

[0040] With reference to the second aspect and the foregoing implementation manners, in some implementations of the second aspect, the apparatus further includes a conversion module specifically configured to: convert a font format corresponding to the text element into a preset font format.

[0041] In a third aspect, an electronic device is provided, including a memory and a processor. The memory is configured to store executable program code, and the processor is configured to call and run the executable program code from the memory, so that the electronic device executes the display method of the hand-drawn table in the first aspect and any possible implementation of the first aspect.

[0042] In a fourth aspect, a computer program product is provided, which includes computer program code. When the computer program code is run on a computer, the computer executes the display method of the hand-drawn table in the first aspect and any possible implementation of the first aspect.

[0043] In a fifth aspect, a computer-readable storage medium is provided, which stores computer program code. When the computer program code is run on a computer, the computer executes the display method of the hand-drawn table in the first aspect and any possible implementation of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0044] FIG. 1 is a schematic flowchart of a display method of a hand-drawn table according to an embodiment of the present application;

[0045] FIG. 2 is a schematic diagram of a table image of a rendered hand-drawn table according to an embodiment of the present application;

[0046] FIG. 3 is a schematic diagram of a table image of another rendered hand-drawn table according to an embodiment of the present application;

[0047] FIG. 4 is a schematic diagram of a feature fusion pyramid structure according to an embodiment of the present application;

[0048] FIG. 5 is a schematic diagram of a structure of a three-branch element segmentation module according to an embodiment of the present application;

[0049] FIG. 6 is a schematic diagram of a structure of a spatial attention network according to an embodiment of the present application;

[0050] FIG. 7(a) is a schematic diagram of a table image with critical text elements according to an embodiment of the present application;

[0051] FIG. 7(b) is a schematic diagram of time sequence information of drawing trajectories of critical text elements according to an embodiment of the present application;

[0052] FIG. 8(a) is a schematic diagram of a table image after beautifying text elements according to an embodiment of the present application;

[0053] FIG. 8(b) is a schematic diagram of a table image after beautifying frame line elements according to an embodiment of the present application;

[0054] FIG. 8(c) is a schematic diagram of a table image after converting a font format of text elements into a preset font format according to an embodiment of the present application;

[0055] FIG. 9 is a schematic diagram of a structure of a display device of a hand-drawn table according to an embodiment of the present application;

[0056] FIG. 10 is a schematic diagram of a structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0057] The technical solutions in the present application will be described in detail below with reference to the drawings. In the description of the embodiments of the present application, unless otherwise specified, “ / ” represents the meaning of or, for example, A / B can represent A or B: “and / or” in the text only represents a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases of A alone, A and B together, and B alone. In addition, in the description of the embodiments of the present application, “multiple” means two or more than two.

[0058] Hereinafter, the terms "first", "second", etc. are used only for the purpose of description, and cannot be understood as implying or suggesting relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second", etc. can explicitly or implicitly include one or more of the features.

[0059] Currently, in both teaching scenarios and meeting scenarios, drawing a handwritten table in a touch panel is a common content display method. However, when a user draws a handwritten table in a touch panel, problems such as distorted table lines and insufficient cell writing area in the table may occur, which affect the aesthetics of the table.

[0060] Specifically, when a user directly draws table lines on a touch panel, due to reasons such as hand jitter, touch precision limitations, or lack of drawing skills, the drawn table lines may not be straight enough or well-aligned, making the table not aesthetically pleasing.

[0061] In addition, when a user draws a handwritten table, the user may not accurately plan the size of each cell in the table, resulting in some cells in the handwritten table appearing too crowded after filling in content, or even some content exceeding the cell range, affecting reading and understanding of the table, and also affecting the aesthetics of the table.

[0062] To solve the above technical problems, an embodiment of the present application provides a display method of a hand-drawn table. The execution subject of the method can be an electronic device, which can be a mobile phone, a tablet computer, a notebook computer, an ultra-mobile personal computer (UMPC), etc. The specific type of the electronic device is not limited in the present application.

[0063] FIG. 1 is a schematic flowchart of a display method of a hand-drawn table according to an embodiment of the present application.

[0064] For example, as shown in FIG. 1, the method 100 includes:

[0065] S101, obtaining a hand-drawn table drawn by a user, and converting the hand-drawn table into a table image.

[0066] S102, performing table element segmentation on the table image to obtain table elements in the table image.

[0067] The table elements in the table image include text elements and frame line elements.

[0068] S103, determining coordinate information of the text elements and coordinate information of the frame line elements in the table image.

[0069] S104, determine an inclination angle of each row of text in the table image based on the coordinate information of the text element, and adjust the inclination angle of the text in the table image to a target inclination angle based on the inclination angle.

[0070] wherein the inclination angle is an inclination angle of the each row of text relative to a reference line.

[0071] S105, fit the coordinate points of each frame line element in the table image based on the coordinate information of the frame line element to obtain a beautified frame line, and replace the frame line in the table image with the beautified frame line.

[0072] S106, display the hand-drawn table after the adjustment and beautification on the electronic device.

[0073] In the embodiments of the present application, by obtaining a hand-drawn table drawn by a user and converting the hand-drawn table into a table image, the personalized elements of the hand-drawn table drawn by the user can be retained in the converted table image. By performing table element segmentation on the table image, various table elements in the table image can be obtained. The table elements include text elements and frame line elements, and then the coordinate information of the text elements and the coordinate information of the frame line elements in the table image are determined, which can provide an accurate basis for subsequent beautification of the table image. Based on the coordinate information of the text elements, the inclination angle of each row of text in the table image is determined, and the inclination angle of the text in the table image is adjusted to a target inclination angle based on the inclination angle, which can ensure that all rows of text in the table image maintain a consistent target inclination angle, so that the final displayed table is more regular and professional, facilitating reading and quick information capture. In addition, by fitting the coordinate points of the frame line elements to generate a beautified frame line, the jaggedness, burrs or discontinuity of the original frame line can be eliminated, making the table boundary clearer and smoother, and significantly improving the overall aesthetic of the table image. Finally, the hand-drawn table after the adjustment and beautification is displayed on the electronic device, improving the user's experience.

[0074] The implementation of each step in the embodiment shown in FIG. 1 will be described in detail as follows:

[0075] For the above S101, it can be understood that the above-mentioned hand-drawn table drawn by the user refers to a hand-drawn table drawn by the user on an electronic device such as a touch panel, a mobile phone, etc. Specifically, the hand-drawn table usually contains several rows or columns, forming cells, and the cells can contain text, numbers, symbols, etc.

[0076] In some embodiments, a user draws a hand-drawn table on a tablet screen using a stylus equipped with a touchpad. The touchpad can capture the content displayed on the screen in real time during the user's hand-drawing process, identify the table structure and handwriting data of the hand-drawn table, and then synthesize a table image in a standard format.

[0077] Further, since most of the current recognition schemes for hand-drawn tables are implemented through deep learning models, such as TableNet model, the input of the deep learning model is usually an image matrix. Specifically, the image matrix of the hand-drawn table drawn by the user can be input into the deep learning model to identify the table structure and extract the content. Based on this, the trajectory data of the hand-drawn table drawn by the user can be obtained, and the trajectory data can be rendered into an image to obtain the input of the deep learning model, i.e., the image matrix.

[0078] It can be understood that the deep learning model can usually learn to identify the boundaries, cells, and text of the table based on the pixel information in the image matrix, and can further extract the structured information of the table (such as cell position, row and column division, and text content).

[0079] Specifically, the coordinate sequence recorded in the trajectory data of the hand-drawn table drawn by the user can be used to draw lines in the order of coordinate points to form a table image consistent with the handwriting process. The trajectory data of the hand-drawn table is usually a continuous coordinate sequence recorded by a handwriting input device (such as a stylus, an electronic whiteboard, etc.) when the user draws the table. Each coordinate point (x, y) represents the position of the pen tip at a certain time, and the sequence of multiple coordinate points constitutes the complete path of the handwriting in the process of drawing the table by the user. For example, {{x 00 , y 00 ,... x 0N , y 0N}, {x 10 , y 10 ,... x 1N , y 1N},...} represents a plurality of coordinate sequences of trajectories that may belong to different lines or texts, and each sub-sequence {x 00 , y 00 ,... x 0N , y 0N} corresponds to a continuous handwriting path.

[0080] In some embodiments, after obtaining the trajectory data of the hand-drawn table drawn by the user, the plurality of drawing trajectories can be rendered based on the trajectory data of the hand-drawn table to obtain a table image of the rendered hand-drawn table.

[0081] It can be understood that the rendering of the plurality of drawing trajectories refers to the conversion of the trajectory data of the handwritten table into a visual image representation, and the rendered table image of the hand-drawn table is shown in FIG. 2.

[0082] In order to more accurately segment the table elements in the table image subsequently, the time sequence information of each trajectory in the hand-drawn table can be embedded in the generated table image, so that the order of each trajectory can be intuitively distinguished in the table image.

[0083] In a possible implementation, the conversion of the hand-drawn table into the table image includes: obtaining trajectory data when the user draws the hand-drawn table; obtaining a plurality of drawing trajectories in the hand-drawn table based on the trajectory data; obtaining time sequence information of each of the plurality of drawing trajectories; and rendering the plurality of drawing trajectories according to the time sequence information of each of the plurality of drawing trajectories to obtain the table image.

[0084] It can be understood that, as described above, the trajectory data is a continuous coordinate sequence recorded by a handwriting input device (such as a stylus, an electronic whiteboard, etc.) when the user draws the table. The rendering of the plurality of drawing trajectories refers to the conversion of the trajectory data of the handwritten table into a visual image representation.

[0085] Specifically, after obtaining the trajectory data when the user draws the hand-drawn table, the adjacent coordinate points in the same trajectory can be connected into a line segment based on the trajectory data to form a plurality of drawing trajectories in the hand-drawn table. While collecting the trajectory data, the time stamp of each coordinate point can be recorded. In the process of forming the drawing trajectory, the time stamp can be used to construct complete time sequence information for each drawing trajectory, that is, the drawing order of each coordinate point and the time interval between adjacent coordinate points, and the plurality of trajectories can be rendered based on the time sequence information to obtain the table image.

[0086] In some embodiments, the rendering of the plurality of drawing trajectories based on the time sequence information of each of the plurality of drawing trajectories can be specifically: drawing trajectories in the order of the time stamp of each trajectory data, for example, lines or elements drawn at an earlier time are drawn first, and elements drawn subsequently are overlaid or added to the existing image. In addition to the order of the drawing trajectories, the time interval between adjacent trajectory points is also included in the time sequence information, and the time interval between adjacent trajectory points can be used to adjust the line thickness, color transparency, brush stroke style, etc. of the drawing trajectory when rendering. In addition, the time sequence information of the trajectory can also be used to create an animation playback of the drawing process of the table.

[0087] For example, if the user first draws the outer frame of the hand-drawn table and then fills the cells row by row, the rendering process of the table image can be to first render the outer frame track, then render the fill content inside each cell, and finally render other details in the table image. In addition, if the user draws a track in the hand-drawn table at a fast pen speed, i.e., the time interval between adjacent track points is short, the line of the track presented in the table image can be relatively thin, and if the user draws a track in the hand-drawn table at a slow pen speed, i.e., the time interval between adjacent track points is long, the line of the track presented in the table image can be relatively thick.

[0088] In addition, in order to enable the user to intuitively see the process of gradually drawing the table from scratch, an animation playback of the drawing process of the table can also be created using the time sequence information of the track. For example, the track data is divided into a series of key frames with equal time intervals, and then each frame is rendered step by step in time sequence to generate a continuous animation sequence, so that the time sequence information of the track can be embodied in the generated table image.

[0089] The above technical solution obtains the track when the user draws the hand-drawn table, obtains a plurality of drawing tracks in the hand-drawn table based on the track data, and renders each drawing track in the order of drawing according to the time sequence information, so that the time sequence information of the drawing track can be embodied in the table image, which can provide accurate reference for subsequent segmentation of table elements and effectively improve the accuracy of table element segmentation.

[0090] In addition to embodying the time sequence information of the track in the generated table image, in order to more reasonably segment each table element in the table image in the future, the time sequence information can also be added to the gray scale information of each drawing track.

[0091] In a possible implementation, the rendering of the plurality of drawing tracks according to the time sequence information of each of the plurality of drawing tracks to obtain the table image includes: determining gray scale values of the plurality of drawing tracks in the table image according to the time sequence information of each of the plurality of drawing tracks; and rendering the plurality of drawing tracks based on the gray scale values to obtain the table image.

[0092] It can be understood that in the process of determining the gray scale values of the plurality of drawing tracks in the table image, a gray scale mapping relationship can be defined according to the time sequence information of each of the plurality of tracks to convert the time sequence information into the gray scale values.

[0093] For example, the gray scale mapping relationship can be that the gray scale value of the earliest drawing track is the smallest and the gray scale value of the latest drawing track is the largest. Specifically, the plurality of drawing tracks can be numbered, and since each drawing track has a unique serial number, the serial number can be directly mapped to the gray scale values of the drawing tracks in the table image.

[0094] Further, since the gray value of each pixel point in the gray image is usually in the range of 0 to 255, the greater the gray value, the brighter the color. If the gray value of the background pixel of the table image is set to 255, that is, the gray value of other regions in the table image not occupied by the drawn track is set to 255, it means that the background of the table image is white. In order to make the drawn track clearly displayed in the table image, the gray value of the drawn track can be set to be lower than 255.

[0095] In order to ensure that the gray value of the drawn track in the table image does not exceed 255 and ensure that the gray value of the drawn track can be recycled, 125 can be taken as a cycle, such as the first track corresponds to a gray value of 0, the second track corresponds to a gray value of 1, and so on, until the 125th track corresponds to a gray value of 124. Then, the 126th track starts counting from 0 again, the 127th track corresponds to a gray value of 1, and so on.

[0096] It can be understood that when the number of drawn tracks in the above table image is very large, such as more than 125, the above method of taking 125 as a cycle to determine the gray value of the drawn track can be used to ensure that the table image containing multiple drawn tracks is reasonably generated, and the time sequence information of the track can be embodied in the generated table image. As shown in FIG. 3, the time sequence information of the drawn track is embodied by the gray value of the track in the table image, that is, the smaller the gray value of the track drawn earlier, the darker the color, and the greater the gray value of the track drawn later, the lighter the color.

[0097] The above technical solution determines the gray value of multiple drawn tracks in the table image according to the time sequence information of each drawn track, and renders the multiple drawn tracks based on the gray value, so that the order of drawing of each drawn track can be directly observed in the table image, which is beneficial to providing a basis for subsequent table element segmentation.

[0098] Further, after obtaining the table image of the hand-drawn table, the table elements in the table image can be segmented based on the table image.

[0099] For S102, it can be understood that the above table elements can include frame elements, cell elements, and text elements in the cells in the table, etc.

[0100] In some embodiments, the process of segmenting the table elements in the table image can specifically be: performing edge detection on the table image, determining the edge features of the table frame and other structures in the table image, and extracting the complete contour information of the table image from the edge image, which can represent the boundaries of the frame lines, cells and internal lines and other elements in the table. Then, according to the extracted edge and contour information, the continuous lines in the vertical and horizontal directions of the table image can be determined, which represent the rows and columns in the table image. After determining the table structure, the content in the cells of the table can be recognized, such as recognizing the text area in the cell by a text detection method.

[0101] After segmenting the table elements in the table image, i.e., the frame lines, cells and text content in the cells in the table, if it is detected that there may be mis-segmentation or incorrect merging in the segmentation result, manual intervention or algorithm correction can be performed on the segmentation structure to ensure the integrity and accuracy of each table element.

[0102] In order to make the process of segmenting the table elements in the table image more convenient, the embodiments of the present application can provide a table element segmentation model. The table image can be input into the table element segmentation model to obtain the segmented table elements of the table image.

[0103] Since the embodiments of the present application need to restore the content of each table element in the table image in detail, and the common scheme img2seq algorithm (an algorithm for converting an image into a sequence representation) for image recognition can convert the table image into structured data, but if the img2seq scheme is directly used, it may not meet the requirement of restoring the content of the hand-drawn table in detail, because the img2seq scheme focuses more on extracting the overall structure and main content of the table, and when processing the tiny details, individual differences or non-standard formats in the hand-drawn table, important information may be lost, resulting in unsatisfactory restoration effect. Therefore, the img2seq scheme cannot be directly used to recognize and convert the table image.

[0104] Further, in the process of segmenting the elements in the table image, the traditional table recognition scheme usually separates the cell detection task and the text detection task, i.e., using separate detection models to obtain the cell area and the text area in the table image. Since multiple models are used, this approach can cause redundant consumption of computing resources, affecting the efficiency of the table element segmentation process.

[0105] Since cell, text and frame line are all basic table elements, they are closely related in space and have clear logical relationship with each other (for example, text is located in a cell, and frame line is used to define the boundary of a cell). Such correlation means that their information can assist each other in the identification process to determine the overall structure of the table. In addition, although cell, text and frame line are closely related, they differ in visual features, such as the cell area is usually a continuous blank or filled area, the text is represented as a character sequence, and the frame line is a continuous line. Based on this, the table element segmentation model provided by the embodiments of the present application can segment the table elements in the table image by using the correlation and difference of the table elements.

[0106] In a possible implementation manner, the table element segmentation of the table image is performed to obtain the table elements in the table image, including: inputting the table image into a table element segmentation model to obtain the table elements in the table image; wherein the table element segmentation model is used to segment the input table image into multiple table elements, and the table element segmentation model is obtained based on a table database.

[0107] It can be understood that the table element segmentation model can segment the input table image into multiple table elements, including cell elements, text elements and frame line elements. After the table image is input into the table element segmentation model, the table element segmentation model can perform a series of processes such as feature extraction, feature analysis and element positioning on the table image, and finally output the classification result of each pixel point in the table image, that is, the table element category to which the pixel point belongs, such as cell element, text element, frame line element and the like.

[0108] Based on the classification result of the pixel point output by the table element segmentation model, the frame line, cell and text content of the table element can be further constructed to obtain the specific position information of the multiple table elements in the table image.

[0109] Further, the table element segmentation model can be obtained based on a table database, and the table database is a data set containing a large number of labeled table images, and each image is labeled with the position, category and other information of various table elements by artificial or semi-automatic manner. Through supervised learning on such a database, the model learns to identify the features of table elements in the image and can apply this knowledge to new unseen table images.

[0110] It can be understood that large-scale training data is the basis for building a table element segmentation model, the table data often contains rich information, the collection and annotation process is relatively complex, and the cost of building a table database is expensive. Based on this, the embodiment of the application can provide a data synthesis method, which can automatically synthesize table data and annotation based on the constituent elements of the table without manual collection and annotation of table data, can effectively ensure the richness and effectiveness of the data, and at the same time reduce the cost of building a table database.

[0111] In a possible implementation manner, the table database manner generation can include S11 to S17 as follows:

[0112] S11: determining a table matrix of a target table to be generated according to a preset number of rows and a number of columns; each element in the table matrix corresponds to a cell in the target table.

[0113] It can be understood that the size structure of the target table to be generated can be determined according to the preset number of rows and the number of columns, the target table is abstracted into a two-dimensional matrix, that is, a table matrix, and each element in the table matrix represents a cell in the target table. In this way, the structure and layout of the table can be conveniently simulated by operating the table matrix.

[0114] For example, a table row number M and a column number N are randomly selected, and a two-dimensional table matrix Mc of size M*N is constructed, which is used to represent the structure of the target table. Each element of the matrix corresponds to a minimum unit in the target table, that is, a cell in the target table. Wherein, the element Mc[i][j] at each position (i, j) in the table matrix represents the cell in the i-th row and the j-th column of the target table.

[0115] Further, a two-dimensional horizontal line matrix Mh of size (M+1)*N can also be constructed to represent the state of each row of horizontal lines of the target table, and each element of the matrix corresponds to the presence or absence of a horizontal line in the target table. By creating a two-dimensional matrix Mh with one more row (M+1 rows) than the target table and the same number of columns as the target table, the state of each row of horizontal lines in the target table is recorded. The additional row in Mh compared with Mc is to represent the extra horizontal line above the target table or the table edge.

[0116] It can be understood that in the above horizontal line matrix Mh, each element Mh[i][j] describes whether there is a horizontal line in the j-th column position in the region between the i-th row and the i+1-th row of the target table. The value of the matrix element is usually 0 or 1, where 1 represents a horizontal line and 0 represents no horizontal line.

[0117] In addition, a two-dimensional column line matrix Mv of size M*(N+1) can also be constructed to represent the vertical line state of each column of the target table, and each element of the matrix corresponds to the presence or absence of a vertical line in the target table. Similarly, a two-dimensional matrix Mv of N+1 columns and the same number of rows as the target table is created to record the state of each column of the target table. The additional column in Mv compared to Mc is to represent the extra vertical line on the left side of the target table or the table edge.

[0118] It can be understood that in the above vertical line matrix Mv, each element Mv[i][j] describes whether there is a vertical line in the i-th row position between the j-th column and the j+1-th column of the target table. Similarly, the value of the matrix element is usually 0 or 1, where 1 represents a vertical line and 0 represents no vertical line.

[0119] In some embodiments, at the initial stage, all elements of the horizontal line matrix Mh and the vertical line matrix Mv can be assigned a value of 1, which means that by default all the horizontal lines and vertical lines in the table are present, i.e. the table is initialized as a complete "full-line table" with all lines drawn.

[0120] S12: Traverse each element in the table matrix and set the characteristic properties of the cell corresponding to each element.

[0121] It can be understood that each element (i.e. each cell) in the matrix is accessed in turn, and each cell is assigned a set of characteristic properties. Exemplarily, these properties can include but are not limited to: whether the cell spans rows and columns, whether it is merged, whether it has a specific style, whether it contains a specific type of data (such as numerical, date, text, etc.), etc.

[0122] Specifically, the characteristic properties of the above-mentioned cell can include the row and column spanning properties of the cell, i.e. the cell can be set to be merged with other adjacent cells, or can remain as a single cell.

[0123] The above setting of the characteristic properties of the cell corresponding to each element can be specifically: randomly assigning row and column spanning properties to each cell in the target table. If the cell is located in the i-th row and j-th column, it can be assigned an integer row_span (row spanning property) indicating that the cell spans row_span rows downward. For example, if row_span is 2, it means that the cell merges its own row and the row below it. Similarly, the cell can also be assigned an integer col_span (column spanning property) indicating that the cell spans col_span columns to the right. For example, if col_span is 3, it means that the cell merges its own column and the two columns to the right.

[0124] Further, when a certain cell is set to span across rows and columns, the original borders within the area covered by the cell need to be removed to reflect the effect of the cell merging. For horizontal borders, the horizontal border matrix Mh[i+1:i+row_span,j:j+col_span] can be set to 0, indicating that the horizontal borders between all cells in the range of row i to (i+row_span-1) and column j to (j+col_span-1) are removed. For vertical borders, the vertical border matrix Mv[i:i+row_span,j+1:j+col_span] can be set to 0, indicating that the vertical borders between all cells in the range of row i to (i+row_span-1) and column j to (j+col_span-1) are removed.

[0125] It can be understood that i+1:i+row_span means all rows from the (i+1)th row to the (i+row_span)th row. Here, i is the row index of the current cell being traversed, and row_span is the row-span attribute value of the cell. Similarly, j:j+col_span means all columns from the jth column to the (j+col_span)th column. Here, j is the column index of the current cell, and col_span is the column-span attribute value of the cell. This range indicates the number of columns spanned to the right by the current cell.

[0126] Mh[i+1:i+row_span,j:j+col_span] means extracting a sub-matrix from the matrix Mh (i.e., the horizontal line matrix), with the starting position being the (i+1)th row and the jth column, and the ending position being the (i+row_span)th row and the (j+col_span)th column. In other words, it selects the horizontal borders between all cells in the rectangular area below (i+1 to i+row_span rows) and to the right (j to j+col_span columns) of the current cell that spans across rows and columns.

[0127] Mv[i:i+row_span,j+1:j+col_span] means extracting a sub-matrix from the matrix Mv (i.e., the column line matrix), with the starting position being the ith row and the (j+1)th column, and the ending position being the (i+row_span)th row and the (j+col_span)th column. In other words, it selects the vertical borders between all cells in the rectangular area below (i to i+row_span rows) and to the right (j+1 to j+col_span columns) of the current cell that spans across rows and columns.

[0128] Further, the characteristic attribute of the cell can further include a height of each row of the cell and a width of each column of the cell. By setting a random height for each row of the cell and a random width for each column of the cell in the table, the generated table can have diversified cell sizes, simulate the situation that the sizes of the cells in the real table are not completely consistent, and increase the complexity and generalization ability of the data set.

[0129] In addition, the characteristic attribute of the cell can further include a coordinate attribute of the cell.

[0130] In some embodiments, the coordinate attributes of all lines of the table can be determined based on the horizontal line matrix Mh and the column line matrix Mv, and the attributes include a starting point, an ending point, a direction, and the like of the line. The line coordinate attributes define the structural framework of the table, that is, the positions of the separation lines between the cells.

[0131] After the setting of the characteristic attribute of the cell is completed, the target frame line of the target table and the target text filled into the cell can be determined based on the characteristic attribute of the cell.

[0132] S13: determining a target frame line matched with the characteristic attribute of the cell according to a preset frame line library.

[0133] It can be understood that the preset frame line library can contain frame line tracks of different lengths and different types. By using the preset frame line library (containing frame lines of different styles, thicknesses, and colors), a target frame line matched with the characteristic attribute of the current cell is selected.

[0134] As described above, since the coordinate attribute of the cell can include a starting point, an ending point, and a direction of the line (that is, the frame line of the table), a target frame line matched with the coordinate attribute of the cell can be selected from the preset frame line library.

[0135] In a possible implementation manner, the characteristic attribute of the cell includes a coordinate attribute; and the determining a target frame line matched with the characteristic attribute of the cell according to a preset frame line library includes: selecting the target frame line matched with the coordinate attribute of the cell from the preset frame line library; or selecting a random frame line from the preset frame line library and adjusting the random frame line based on the coordinate attribute to obtain the target frame line; and the target frame line is matched with the coordinate attribute of the cell.

[0136] It can be understood that the target frame line matched with the coordinate attribute of the cell can be directly selected from the preset frame line library.

[0137] Since the stored frame lines in the preset frame line library are also limited, a target frame line that completely matches the coordinate attribute of the cell may not be determined. Therefore, a frame line can be randomly selected from the frame line library, and a transformation operation such as rotation, interpolation or sampling can be performed on the randomly selected frame line to adjust the length and shape of the frame line to ensure that the starting position and ending position of the frame line completely match the coordinate attribute of the cell.

[0138] S14: determining, according to the characteristic attribute of the cell, a target text for filling into the cell from a preset text library.

[0139] It can be understood that the above-mentioned preset text library includes various types of text data, such as text data of a character type, text data of a number type, and text data of a symbol type, etc. From the preset text library (containing various types of example text data), appropriate text content (i.e., the target text) is selected according to the characteristic attribute of the cell to fill into the cell.

[0140] In a possible implementation, the characteristic attribute of the cell includes a coordinate attribute, and the determining, according to the characteristic attribute of the cell, of a target text for filling into the cell from a preset text library includes: selecting, in the preset text library, the target text that matches the coordinate attribute of the cell.

[0141] It can be understood that the characteristic attribute of the cell can include a coordinate attribute, i.e., the starting point and ending point position of the cell.

[0142] Further, in the above-mentioned preset text library, a target text that can be placed inside the cell is selected, and the position and size of the target text completely match the coordinate attribute of the cell.

[0143] S15: generating a target table according to the target frame line and the target text.

[0144] It can be understood that according to the above-mentioned determined target frame line and target text, a complete table image is synthesized to integrate all the cells, frame lines and text content into a visual table.

[0145] S16: taking the position information of the target frame line and the filling area of the target text as annotation information of the target table.

[0146] It can be understood that after the above-mentioned target frame line and target text are determined, the position information (such as coordinates, width, height, etc.) of the target frame line and the filling area (i.e., the specific position of the text in the cell) of the target text can be recorded as annotation information of the target table. These annotation information is crucial for subsequent model training and can guide the model to learn how to recognize and locate the frame line and text in the table.

[0147] S17: generating the table database based on the target table and the annotation information.

[0148] It can be understood that the generated target table image is paired with its corresponding annotation information to form a data sample.

[0149] Further, the above steps are repeated to generate a plurality of table samples with different characteristic attributes, frame line styles, and text contents to cover various possible table scenarios. All generated table samples and their annotation information are sorted into a database to complete the construction of the table database for training the table element segmentation model.

[0150] The above technical solution can automatically synthesize table data and annotations based on the constituent elements of the table, and the collection and annotation of table data need to be performed manually, thereby effectively ensuring the richness and effectiveness of the table database and reducing the data cost.

[0151] Further, the above table element segmentation model can generally include a feature extraction module and an element segmentation module. The feature extraction module is specifically configured to extract effective features from the input table image, and the element segmentation module is specifically configured to segment elements in the table image according to different features.

[0152] In a possible implementation manner, the table element segmentation model includes a feature extraction module and an element segmentation module, the element segmentation module includes a plurality of branches for segmenting different types of table elements, the same type of table element corresponds to the same branch, and the table image is input into the table element segmentation model to obtain the table elements in the table image, which can include the following S21 to S23:

[0153] S21: performing multi-level feature extraction on the table image through the feature extraction module, and fusing the extracted multi-level features to obtain a high-dimensional feature map.

[0154] It can be understood that, as described above, since each element in the table has relevance and difference, the elements can assist each other or may be affected by each other in the process of segmenting the table elements in the table image. In order to ensure the accuracy of the segmentation process, the feature extraction module and the element segmentation module in the table element segmentation model can be designed accordingly.

[0155] Specifically, the core task of the above feature extraction module is to identify and extract features from the input table image, which are valuable for subsequent analysis or segmentation of table elements.

[0156] Exemplarily, these features can include table structure (such as row and column division, cell content), numerical distribution, text features, logical relationships, and the like. Since the table image can have target features of multiple scales, such as local small-range numerical patterns, global table layout rules, and even complex correlations across rows and columns.

[0157] Therefore, the feature pyramid structure can be used for feature extraction of the table image. The feature pyramid is a multi-scale feature representation method widely used in the field of computer vision. In the feature extraction process of the table image, the feature pyramid is a multi-level structure, each level corresponds to a specific scale, and different scale features can be extracted and saved respectively. The features of each level can also be synthesized to form a comprehensive feature extraction result containing multi-scale information.

[0158] In order to avoid losing feature information of the table image due to a large number of levels in the process of extracting features of the table image using the feature pyramid structure, the feature fusion pyramid structure can be used to solve the problem of loss of feature information in the feature pyramid structure.

[0159] Specifically, the feature fusion pyramid structure introduces a skip connection in the traditional feature pyramid structure, that is, a bridge is directly established between different levels, so that the features of the low layer can be directly transmitted to the high layer without relying entirely on the layer-by-layer progressive mode. Through the cross-connection, not only the bottom layer features can be quickly transmitted to the high layer, but also the deep fusion between different levels of features is promoted.

[0160] Exemplarily, the table image is input into the table element segmentation model, and a plurality of level feature extraction networks (such as convolutional neural networks) are used to process the input table image to obtain features of different scales in the table image. And each level corresponds to a specific feature map, which contains image feature information at the scale corresponding to the level. For example, the lower level is usually used to capture local details and fine features of the image, and the higher level pays attention to larger range structures and global information.

[0161] In order to enhance the integrity of different scale features, the feature fusion pyramid structure can be introduced in the embodiment of the present application. The structure makes the low layer features directly transmitted to the high layer through the skip connection, without relying entirely on the layer-by-layer progressive mode, thereby supplementing the feature information that may be lost in the middle layer.

[0162] As shown in FIG. 4, the input table image is processed in multiple levels by the feature fusion pyramid structure in FIG. 4, and the first layer feature, the second layer feature, the third layer feature and the fourth layer feature are obtained in turn. Since the layers can be connected in a skip manner, the features extracted in the first layer can be directly fused with the third layer features or directly fused with the fourth layer features. The multi-level fused features are obtained by step-by-step upward fusion.

[0163] Specifically, first, in order to facilitate processing, one level can be selected as a target level, and the feature map size of the target level is selected as a target size. Then, the feature map sizes of other feature layers to be fused are unified to the target size. For example, for a feature map with a large size, a convolution with a step of 2 (equivalent to down sampling) is used to reduce the size; for a feature map with a small size, a convolution with a size of 1*1 is used to increase the size of the feature map by using bilinear interpolation for up sampling. Second, the feature maps of different levels with unified sizes after size adjustment are spliced to form a fused feature map containing multi-scale information.

[0164] Further, the multi-layer convolutional neural network can also be applied to the fused feature map after splicing to perform deep fusion of features of different scales, and finally output comprehensive features, i.e., the high-dimensional feature map described above.

[0165] In order to avoid too large feature difference between feature layers far apart, resulting in unstable fusion process, the multi-stage fusion manner can be used in the embodiments of the present application. As shown in FIG. 4, for four-layer feature fusion, the features are fused step by step in three stages to ensure that the difference between the feature layers is small each time, and the stability of the fusion process is ensured.

[0166] S22: performing feature analysis on the high-dimensional feature map by the element segmentation module to determine the type information of each feature point in the high-dimensional feature map.

[0167] It can be understood that, as described above, the element segmentation model provided by the embodiments of the present application includes not only a feature extraction module but also an element segmentation module. The main responsibility of the element segmentation module is to analyze the high-dimensional feature map generated by the feature extraction module in depth to determine the category to which each feature point in the feature map belongs. The above-mentioned feature point can correspond to a pixel or a pixel block in the original table image, and the above-mentioned determination of the category to which each feature point in the feature map belongs is equivalent to the determination of which category of elements (such as cells, texts, frame lines, etc.) the feature point belongs to in the table.

[0168] Currently, traditional element segmentation modules are usually composed of a series of Convolutional Neural Networks (CNN). CNN extracts features from the local of each feature point through convolution operation, and then predicts the category of the feature point using these local features. This approach is suitable for single or similar type of feature analysis, but for multiple types of features with significant differences such as table images, it may face confusion and interference problems.

[0169] Based on this, the embodiments of the present application can provide a multi-branch element segmentation structure. As shown in FIG. 5, the structure can separate the segmentation process of cells, texts and frame lines into three independent branches, including a cell segmentation branch, a text segmentation branch and a frame line segmentation branch, and each branch focuses on analyzing and predicting the corresponding type of features, avoiding interference between different types of features.

[0170] S23: inputting each feature point into the branch corresponding to the type of the feature point to segment the high-dimensional feature map, and obtaining the table elements of the table image.

[0171] It can be understood that in each branch of the above multi-branch element segmentation structure, the feature analysis process can be designed according to the unique characteristics and needs of the table elements (cells, texts or frame lines) it is responsible for. For example, for the cell branch, it may pay more attention to boundary clarity and internal filling; for the text branch, it may pay more attention to character continuity and line distance; for the frame line branch, it may pay more attention to line continuity and intersection point identification.

[0172] Specifically, the cell segmentation branch and the text segmentation branch in the above multi-branch element segmentation structure show similarity in task nature, i.e. both need to accurately segment the target arranged densely and possibly with adhesion phenomenon. The above target refers to the cells or texts closely distributed in the table image layout, which may have boundary ambiguity, overlap or partial nesting, increasing the complexity of segmentation.

[0173] In a possible implementation, the plurality of branches includes a first branch for segmenting a target element, the target element including a text element or a cell element, and the method further includes: inputting the high-dimensional feature map into the first branch to predict an overall region and a center region of the target element; determining a center point and a target number of the target element based on the center region of the target element; and taking the center point of the target element as an aggregation center to aggregate pixels in the overall region of the target element to obtain a plurality of sub-regions, where the number of the sub-regions is the target number, and each of the sub-regions corresponds to a target element; and obtaining the target element in the table image based on the plurality of sub-regions.

[0174] It can be understood that, when the table element is a cell element or a text element, the first branch is the cell segmentation branch or the text segmentation branch in FIG. 5. In FIG. 5, the multi-scale fusion feature, that is, the high-dimensional feature map, is input into the cell segmentation branch or the text segmentation branch, and the cell element or the text element is obtained, corresponding to the cell segmentation result or the text segmentation result output by the cell segmentation branch or the text segmentation branch in FIG. 5.

[0175] As can be known from the foregoing, the two branches of the cell segmentation branch and the text segmentation branch each face the problem of how to accurately extract an independent cell or text from a dense and complex table image layout. As described above, due to the high density and mutual adhesion between targets (that is, each cell or text), a traditional single-target segmentation method may not effectively distinguish the boundaries of adjacent targets, resulting in problems such as omission, excessive fusion, or missegmentation in the segmentation result.

[0176] Based on this, in the embodiments of the present application, a multi-task prediction strategy can be used, that is, two related but different prediction tasks are performed at the same time.

[0177] Specifically, task one can be: overall region prediction of a target element, which aims to predict the actual position of the target element to be detected, that is, to predict the bounding box of all cell elements or text elements in the entire table image. It should be understood that the overall region prediction can determine the approximate outline and space range occupied by all target elements.

[0178] The second task can be: center region prediction of the target element, which aims to determine the center point of each target element and predict the region within a certain range around the center of the target element, i.e., determine the center point of each cell element or text element and predict the region within a certain range around the center of each cell element or text element. It should be understood that the above center region prediction helps to determine the specific number of target elements and the core position of each target element.

[0179] Further, after obtaining the prediction results of the above two tasks, the center points of each target element (i.e., each cell element or text element) and the positions of each target element in the table image can be identified by analyzing the prediction results of the center region of the target, and the total number of target elements can be counted.

[0180] Subsequently, the pixels within the overall region of the target element are aggregated using the prediction results of the overall region of the target and the centers of the target elements that have been determined. Specifically, the pixels within the overall region of each target element are reasonably attributed according to the center of each target element, i.e., the pixels within the overall region are assigned to the cell or text corresponding to the closest center point according to their distance relationship with the center points.

[0181] It can be understood that after the above pixel aggregation and distribution process, each target element (i.e., each cell element or text element) finally obtains a corresponding predicted region, which contains not only the actual position information of the target but also ensures that the boundaries between targets can be accurately distinguished.

[0182] Further, after the segmentation of the cell elements and text elements is achieved, the frame line elements in the table image also need to be segmented.

[0183] It can be understood that the purpose of frame line segmentation is to segment the lines (i.e., frame lines) in the table, which have the characteristics of being thin and long and spanning the entire table image. This means that the lines occupy a small local space of the image, but their extension range can be very large and can span multiple receptive fields (i.e., the image area covered by a convolution kernel) of the convolutional neural network. However, the conventional convolutional neural network (CNN) may have difficulty effectively capturing the global or overall features of the lines when processing such thin and long targets that span a wide range, especially when the lines are very long and span multiple receptive fields of the convolutional layers.

[0184] Based on this, the embodiments of the present application can use a spatial attention network to segment the frame lines in the table image. Specifically, the spatial attention network can be used to extract the horizontal and vertical features corresponding to the table frame lines.

[0185] In a possible implementation manner, the plurality of branches include a second branch for segmenting the frame line element, the second branch includes a spatial attention network, and the spatial attention network includes a frame line feature extraction module, a horizontal fusion module, and a vertical fusion module; the inputting of the each feature point into the branch corresponding to the type of the feature point to segment the high-dimensional feature map to obtain the table element of the table image includes: inputting the high-dimensional feature map into the frame line feature extraction module to obtain a frame line fusion feature map; inputting feature values corresponding to each row of feature points in the frame line fusion feature map into the horizontal fusion module as a sequence to perform feature fusion to obtain a horizontal frame line element in the frame line element; and inputting feature values corresponding to each column of feature points in the frame line fusion feature map into the vertical fusion module as a sequence to perform feature fusion to obtain a vertical frame line element in the frame line element.

[0186] It can be understood that when the table element is a frame line element, the second branch is the frame line segmentation branch in FIG. 5. In FIG. 5, the multi-scale fusion feature, that is, the high-dimensional feature map, is input into the frame line segmentation branch, and the horizontal frame line element and the vertical frame line element, that is, the frame line element, are obtained, which correspond to the frame line segmentation result output by the frame line segmentation branch in FIG. 5.

[0187] The spatial attention network is a network structure that can enhance the sensitivity of a model to a specific spatial region. It can dynamically allocate attention weights when processing an image, focus on important regions related to the current task, and suppress the influence of irrelevant or secondary regions. In the embodiment of the present application, the spatial attention network can effectively focus on the horizontal and vertical features of the table frame line, and can more effectively capture the characteristics of the table frame line extending in a specific direction in the image. Even if the line spans a large spatial range, the attention mechanism can still concentrate on processing the features in the relevant direction, so that the overall features of the line can be better extracted.

[0188] For example, as shown in FIG. 6, the high-dimensional feature map generated by the feature extraction module is input into the spatial attention network, and the high-dimensional feature map is processed by the frame line feature extraction module (such as a convolutional neural network) to obtain a multi-scale fusion feature map. Each light gray block in the feature map represents a neuron and contains feature information of a local region.

[0189] As shown in FIG. 6, the spatial attention network is divided into horizontal and vertical dimensions according to the direction of feature extraction, which can correspond to the characteristics that the table frame line may extend in the row or column direction.

[0190] As shown in the left part of FIG. 6, the feature values (also referred to as neurons) corresponding to each row of feature points in the feature map are regarded as a group, and all the row groups are input into a horizontal fusion module, such as a Bi-LSTM (Bidirectional Long Short-Term Memory) module, so that each neuron performs bidirectional feature fusion through the Bi-LSTM, and its receptive field (i.e., the image region affecting the neuron) extends to the entire row region. This means that each neuron not only considers the local features in which it is located, but also fuses the features of all the neurons in the row to obtain global information in the row direction, i.e., the dark gray block in the left part of FIG. 6, which fuses the features of all the neurons in each row of the feature map, can obtain the horizontal frame lines in the table elements.

[0191] Similarly, as shown in the right part of FIG. 6, the feature values (also referred to as neurons) corresponding to each column of feature points in the feature map are regarded as a group, and all the column groups are input into a vertical fusion module, such as a Bi-LSTM module. Each neuron also performs bidirectional feature fusion through the Bi-LSTM, and its receptive field extends to the entire column region. In this way, each neuron fuses the features of all the neurons in the column to obtain global information in the column direction, i.e., the dark gray block in the right part of FIG. 6, which fuses the features of all the neurons in each column of the feature map, can obtain the vertical frame lines in the table elements.

[0192] The above scheme enables each neuron to sufficiently fuse the global information in the row or column through the spatial attention network, which is crucial for recognizing and segmenting the table frame lines extending along the rows or columns, and helps to improve the accuracy of frame line segmentation.

[0193] In some embodiments, the horizontal fusion module or the vertical fusion module, i.e., the Bi-LSTM module, can be replaced by other structures with global feature transmission properties, such as Attention (attention mechanism) or SCNN (spatial convolutional neural network).

[0194] The above technical scheme generates a high-dimensional feature map by performing multi-level feature extraction and fusion on the table image, which can capture the complex structural information of the table, including row and column division, cell boundary, merged cell, etc., ensuring accurate recognition and restoration of the table structure. The element segmentation module performs feature analysis on the high-dimensional feature map to determine the type information of each feature point (which may correspond to a pixel or region in the table image), achieving automatic classification of table elements and greatly improving processing efficiency and accuracy. Inputting the high-dimensional feature map into the multi-branch structure corresponding to the type information enables targeted segmentation of different types of table elements. This design enables the model to learn multiple table element segmentation tasks simultaneously during the training process, improving learning efficiency and model performance.

[0195] As described above, the trained table element segmentation model can not only segment each table element in the table image, but also determine the position information of each table element, i.e., the coordinate information of each table element in the table image.

[0196] Generally, the table element segmentation result obtained by performing table element segmentation on the table image can include the coordinate information of the table element in the table image. However, in the case where there is an overlapping area between any two table elements, it can be difficult to accurately determine the coordinate information of the table element in the table image.

[0197] In a possible implementation, after performing table element segmentation on the table image to obtain the table elements in the table image, the method further includes: determining whether the table elements include a critical text element; wherein the critical text element is a text element having an overlapping area with at least two cell elements; in a case where it is determined that the critical text element exists, determining time sequence information of a drawing track of the critical text element; based on the time sequence information, determining a target cell to which the critical text element belongs among the at least two cell elements; and based on the target cell, determining coordinate information of the critical text element in the table image.

[0198] It can be understood that after obtaining the table elements in the segmented table image, it can be determined whether the table elements include a critical text element.

[0199] Specifically, if there is an intersection area between the text region and the cell region, it can be determined whether the critical text element exists based on a ratio value between an area of the intersection area and an area of the cell.

[0200] For example, as shown in FIG. 7(a), there is an intersection area Text Box between the text region A and the cell Tabel Cell. The ratio value between the area of the intersection area Text Box and the area of the cell Tabel Cell can be calculated, and it can be determined whether the ratio value is less than a preset ratio threshold. If the ratio value is less than the preset ratio threshold, the time sequence information of the drawing track of the critical text element inside the intersection area Text Box can be obtained to determine the ownership of the critical text.

[0201] It can be understood that the preset ratio threshold can be set according to actual needs, for example, set to 0.8.

[0202] Further, as described above, if the ratio value is less than the preset ratio threshold, the time sequence information of the drawing track of the critical text element inside the intersection area Text Box can be obtained, and the ownership of the critical text can be determined based on the time sequence information.

[0203] For example, as shown in FIG. 7(b), the 7th track and the 8th track are located at the intersection of a cell and a text area, and the position information of the 7th track and the 8th track can be determined based on the position information of the last track of the 7th track. For example, if the last track of the 7th track, i.e., the 6th track, is located in the Tabel Cell, the 7th track and the 8th track can be merged into the Tabel Cell.

[0204] Further, after determining the target cell to which the critical text element belongs, the coordinate information of the critical text element in the table image can be determined based on the target cell.

[0205] For S103, it can be understood that after obtaining the text elements and the frame line elements in the table image, the coordinate information of the text elements and the coordinate information of the frame line elements in the table image can be determined, so as to subsequently beautify and adjust the table image based on the coordinate information of the text elements and the coordinate information of the frame line elements.

[0206] For example, the beautification of the table image can be self-beautification after the user completes drawing the table, or can be triggered by a functional control, such as a virtual or physical button, and the like.

[0207] In order to retain the personalized features of the user's hand-drawn table as much as possible, the embodiments of the present application will not directly recognize and convert the content of the text track drawn by the user, but only normalize the size of the text track, so that the text track looks more beautiful.

[0208] For S104, it can be understood that based on the coordinate information of the text in the table element, the position of each text in the table image can be determined, and by analyzing the coordinate information of the text in the same row, the overall inclination angle of the text in the row can be calculated.

[0209] The reference line represents the bottom boundary of the text, that is, the line formed by the lowest points of the handwriting of the text in each row. The inclination angle generally refers to the inclination angle of a single character or the text in a row relative to the reference line. The reference line can be an invisible line in the actual handwritten text, or a horizontal line artificially set during analysis.

[0210] For example, the commonly used method for calculating the overall inclination angle of the text in a row can include fitting a straight line model, calculating the first principal direction of principal component analysis, and the like. Through these methods, the inclination angle of the text in a row relative to the horizontal direction can be calculated.

[0211] In a possible implementation, the adjusting the tilt angle of the text in the table image to the target tilt angle based on the tilt angle includes: performing affine transformation on the text based on the tilt angle to adjust the height of the text to the target height and adjust the tilt angle of the text in the table image to the target tilt angle.

[0212] It can be understood that the text elements in the text line are subjected to affine transformation based on the tilt angle calculated above.

[0213] The affine transformation is a geometric transformation that keeps the parallel lines and the distance between the parallel lines of a two-dimensional figure unchanged, and includes basic operations such as translation, scaling, rotation, and shearing (i.e., tilting). In image processing, affine transformation is often used to adjust the position, size, direction, and other attributes of objects in an image.

[0214] Through the affine transformation described above, the text elements of each text line are adjusted to a uniform target height, and the rotation angle difference to be performed is calculated according to the target tilt angle and the actual tilt angle of each text line, and then the text elements are subjected to rotation transformation so that the tilt angle is adjusted to the target tilt angle.

[0215] Further, after the affine transformation described above is completed, the adjusted text elements are redrawn to the corresponding positions in the table image to replace the original tilted text. In this way, the text in the table image reaches the target height and tilt angle. In addition, the coordinates of the bounding rectangle of the adjusted text can be combined to adjust the table area, such as "center alignment" adjustment, and the adjusted effect is shown in FIG. 8(a), which can achieve beautification of the table image.

[0216] In addition, in addition to beautifying the text drawn by the user, the style of the table frame line drawn by the user can also be beautified.

[0217] For S105 described above, it can be understood that the coordinate point data of the table frame line is subjected to curve fitting based on the coordinate information of the frame line element in the table element, to obtain a continuous and smooth curve to replace the original frame line.

[0218] In a possible implementation, the fitting of the coordinate points of each frame line element in the table image based on the coordinate information of the frame line element in the table element to obtain a beautified frame line includes: the fitting includes at least one of alignment, completion, and smoothing operation of the frame line in the table image.

[0219] It can be understood that the alignment operation of the frame line in the table image described above means that the frame line in the table is adjusted to ensure that the frame line is vertically or horizontally aligned.

[0220] In some embodiments, the table image can be a hand-drawn table image. In some embodiments, the table image can be a table image drawn by a user using a stylus or a finger.

[0221] Due to hand shaking, touch precision limitation or lack of drawing skills when drawing the table, the lines in the table can be rough and discontinuous. The smoothing operation on the lines in the table image can reduce the roughness and discontinuity of the lines by an algorithm, so that the lines become smooth and continuous.

[0222] In some embodiments, the table line can be aligned when it is determined that the table line in the table image is misaligned. The table line can be completed when it is determined that the table line in the table image is missing. The table line can be smoothed when it is determined that the table line in the table image is rough and discontinuous.

[0223] It can be understood that the alignment operation, the completion operation and the smoothing operation on the table line can be collectively referred to as beautifying the table line.

[0224] For example, as shown in FIG. 8(b), the table line in FIG. 2 is beautified to obtain FIG. 8(b). The fitted line in FIG. 8(b) is similar to the original writing trajectory of the user, but is smoother than the original line. The fitted line is vertically aligned and horizontally aligned, and the fitted line has no missing.

[0225] For example, the smoothing operation method can be a Bezier curve, a spline curve or the like. The method can generate a smooth curve according to the determined coordinate points.

[0226] Further, the smoothness generally refers to the smoothness of the curve, which can be measured by the curvature change of the curve, the size of the second derivative, the angle difference between adjacent points, etc. The smoothness of the curve can be measured by setting a preset threshold, for example, the beautified line can be a line with a smoothness greater than or equal to the preset threshold.

[0227] For example, the preset threshold can be set according to actual needs, and the preset threshold can be used as a standard to determine whether the fitted curve meets the smoothness requirement.

[0228] It can be understood that if the smoothness of the fitted curve is lower than the preset threshold, the fitting process can be further optimized, such as increasing the fitting parameters, changing the fitting function type or adjusting the data point weight, until the smoothness of the curve reaches or exceeds the preset threshold.

[0229] Further, in addition to adjusting the tilt angle of the text in the table image, in order to ensure the uniformity of the font format of the text elements, the font format of the text elements can be converted.

[0230] In a possible implementation, before the electronic device displays the hand-drawn table after the adjustment and beautification, the method further includes: converting the font format corresponding to the text elements into a preset font format.

[0231] It can be understood that, by converting the font format corresponding to the text elements into a preset font format, the text in the hand-drawn table displayed on the electronic device can look more standardized and uniform, and the overall visual effect of the hand-drawn table can be improved.

[0232] Optionally, the preset font format can be a system default font format easy to read, or can be a certain font format preset by a user. The embodiments of the present application do not limit this.

[0233] For example, as shown in FIG. 8(c), if the preset font format is “font 1”, after converting the font format corresponding to the text elements in the hand-drawn table into “font 1”, the hand-drawn table in FIG. 8(c) can be obtained.

[0234] In some embodiments, in the case where the text elements in the table image include character elements and literal elements, the font format of the character elements can be converted into a first preset format corresponding to the character elements, and the font format of the literal elements can be converted into a second preset format corresponding to the literal elements.

[0235] It can be understood that the character elements can be the smallest individually recognizable symbols, including letters, numbers, punctuation marks, spaces, and various special symbols, etc., such as “14, 20, 10, 8” in FIG. 8(c). The literal elements can generally be understood as text in pure literal form, such as the literal “county name, rich, Longjiang” in FIG. 8(c).

[0236] Optionally, the first preset format and the second preset format can be a system default font format easy to read, or can be a certain font format preset by a user. The first preset format and the second preset format can be the same format, or can be different formats. The embodiments of the present application do not limit this.

[0237] For example, if the text elements in the table image include character elements and literal elements, and the first preset format corresponding to the character elements is set as “font 2” and the second preset format corresponding to the literal elements is set as “font 1”, the font format of the character elements is converted into “font 2”, and the font format of the literal elements is converted into “font 1”.

[0238] For S106, it can be understood that after adjusting the tilt angle of the text elements in the table image, beautifying the table frame lines in the table image, and converting the font format of the text elements in the table image, the adjusted and beautified hand-drawn table is displayed on the electronic device.

[0239] The above scheme adjusts the tilt angle of the text in the table image to the target tilt angle, which can ensure that all text lines in the table image maintain a consistent target tilt angle, so that the final displayed table is more regular and professional, facilitating reading and quick information extraction. By fitting the coordinate points of the frame line elements to generate a beautified frame line, the original frame line can be eliminated to make the table border clearer and smoother, significantly improving the overall aesthetics of the table image. In addition, by converting the font format of the text elements to a preset font format, all text in the table image can look more standardized, coordinated and consistent, improving the overall visual effect of the table image.

[0240] FIG. 9 is a structural schematic diagram of a display device of a hand-drawn table according to an embodiment of the present application.

[0241] For example, as shown in FIG. 9, the device 900 includes:

[0242] The acquisition module 901 is configured to acquire a hand-drawn table drawn by a user and convert the hand-drawn table into a table image.

[0243] The segmentation module 902 is configured to perform table element segmentation on the table image to obtain table elements in the table image.

[0244] The table elements in the table image include text elements and frame line elements.

[0245] The determination module 903 is configured to determine coordinate information of the text elements and coordinate information of the frame line elements in the table image.

[0246] The beautification module 904 is configured to determine a tilt angle of each row of text in the table image based on the coordinate information of the text elements, and adjust the tilt angle of the text in the table image to a target tilt angle based on the tilt angle; based on the coordinate information of the frame line elements, fit coordinate points of each frame line element in the table image to obtain a beautified frame line, and replace the frame line in the table image with the beautified frame line.

[0247] The tilt angle is a tilt angle of the each row of text relative to a reference line.

[0248] The display module 905 is configured to display the adjusted and beautified hand-drawn table on an electronic device.

[0249] In a possible implementation, the acquisition module is specifically configured to: acquire trajectory data of the user when the user draws the hand-drawn table; obtain a plurality of drawing trajectories in the hand-drawn table based on the trajectory data; acquire time sequence information of each of the plurality of drawing trajectories; and render the plurality of drawing trajectories according to the time sequence information of each of the plurality of drawing trajectories to obtain the table image.

[0250] In a possible implementation, the device further includes a judgment module, which is specifically configured to: determine whether the table element contains a critical text element; the critical text element is a text element having a position overlap area with at least two cell elements; in a case where it is determined that the critical text element exists, determine time sequence information of a drawing trajectory of the critical text element according to the table image; determine a target cell to which the critical text element belongs among the at least two cell elements based on the time sequence information; and determine coordinate information of the critical text element in the table image based on the target cell.

[0251] In a possible implementation, the acquisition module includes a rendering unit, which is specifically configured to: determine a gray value of a plurality of drawing trajectories in the table image according to the time sequence information of each of the plurality of drawing trajectories; and render the plurality of drawing trajectories based on the gray value to obtain the table image.

[0252] In a possible implementation, the segmentation module is specifically configured to: input the table image into a table element segmentation model to obtain table elements in the table image; the table element segmentation model is used to segment the input table image into a plurality of table elements, and the table element segmentation model is obtained based on a table database.

[0253] In a possible implementation, the segmentation module further includes a table database generation module, which is specifically configured to: determine a table matrix of a target table to be generated according to a preset number of rows and a preset number of columns; each element in the table matrix corresponds to a cell in the target table; traverse each element in the table matrix, and set a feature attribute of the cell corresponding to the element; determine a target frame line matched with the feature attribute of the cell according to a preset frame line library; determine a target text used to fill in the cell in a preset text library according to the feature attribute of the cell; generate the target table according to the target frame line and the target text; take position information of the target frame line and a filling area of the target text as annotation information of the target table; and generate the table database based on the target table and the annotation information.

[0254] In a possible implementation, the characteristic attribute of the cell includes a size attribute and a coordinate attribute; the table database generation module includes a selection unit, which is specifically configured to: in the preset frame line library, select the target frame line that matches the size attribute of the cell and matches the coordinate attribute of the cell; or, in the preset frame line library, select a random frame line, and adjust the random frame line to obtain the target frame line based on the size attribute and the coordinate attribute; wherein the target frame line matches the size attribute of the cell and matches the coordinate attribute of the cell; and the target text used for filling into the cell is determined from the preset text library according to the characteristic attribute of the cell, including: in the preset text library, selecting the target text that matches the coordinate attribute of the cell.

[0255] In a possible implementation, the table element segmentation model includes a feature extraction module and an element segmentation module, the element segmentation module includes a plurality of branches for segmenting different types of table elements, the same type of table element corresponds to the same branch, and the segmentation module is specifically configured to: perform multi-level feature extraction on the table image through the feature extraction module, fuse the extracted multi-level features, and obtain a high-dimensional feature map; perform feature analysis on the high-dimensional feature map through the element segmentation module, determine the type of each feature point in the high-dimensional feature map; and input each feature point into a branch corresponding to the type of the feature point, to segment the high-dimensional feature map and obtain the table elements of the table image.

[0256] In a possible implementation, the plurality of branches include a first branch for segmenting a target element, the target element including a text element or a cell element, and the segmentation module includes a first segmentation unit, which is specifically configured to: input the high-dimensional feature map into the first branch, predict an overall region and a center region of the target element; determine a center point and a target number of the target element based on the center region of the target element; and take the center point of the target element as an aggregation center, aggregate pixels in the overall region of the target element to obtain a plurality of sub-regions; wherein the number of the sub-regions is the target number, and each of the sub-regions corresponds to a target element; and obtain the target element in the table image based on the plurality of sub-regions.

[0257] In a possible implementation, the plurality of branches include a second branch for segmenting the frame line element, the second branch includes a spatial attention network, and the spatial attention network includes a frame line feature extraction module, a horizontal fusion module, and a vertical fusion module; the segmentation module includes a second segmentation unit, and the second segmentation unit is specifically configured to: input the high-dimensional feature map into the frame line feature extraction module to obtain a frame line fusion feature map; input feature values corresponding to each row of feature points in the frame line fusion feature map into the horizontal fusion module as a sequence for feature fusion to obtain a horizontal frame line element in the frame line element; and input feature values corresponding to each column of feature points in the frame line fusion feature map into the vertical fusion module as a sequence for feature fusion to obtain a vertical frame line element in the frame line element.

[0258] In a possible implementation, the beautifying module is specifically configured to: based on the inclination angle, perform affine transformation on the text to adjust a height of the text to a target height and adjust an inclination angle of the text in the table image to a target inclination angle.

[0259] In a possible implementation, the beautifying module is further configured to: the fitting includes at least one of frame line alignment, completion, and smoothing operation in the table image.

[0260] Optionally, the apparatus further includes a conversion module, which is specifically configured to: convert a font format corresponding to the text element into a preset font format.

[0261] FIG. 10 is a structural schematic diagram of an electronic device provided in an embodiment of the present application.

[0262] For example, as shown in FIG. 10, the electronic device 1000 includes a memory 1001 and a processor 1002, wherein the memory 1001 stores executable program code 1011, and the processor 1002 is configured to invoke and execute the executable program code 1011 to execute a display method of a hand-drawn table.

[0263] In addition, an apparatus is also protected in an embodiment of the present application, and the apparatus can include a memory and a processor, wherein the memory stores executable program code, and the processor is configured to invoke and execute the executable program code to execute a display method of a hand-drawn table provided in an embodiment of the present application.

[0264] The present embodiment can divide the apparatus into functional modules according to the above method examples, for example, each functional module can be corresponding, or two or more functions can be integrated in one processing module, and the integrated module can be implemented in the form of hardware. It should be noted that the division of modules in the present embodiment is illustrative, and is only a logical function division, and another division mode can be used in actual implementation.

[0265] In the case of adopting the respective functional modules corresponding to the respective functions, the apparatus can further include an obtaining module, a splitting module, a determining module, a beautifying module, and the like. It should be noted that all the related content of the respective steps involved in the above method embodiments can be referred to the function description of the corresponding functional modules, and will not be repeated here.

[0266] It should be understood that the apparatus provided by the embodiment is used to execute the above-mentioned display method of a hand-drawn table, and thus can achieve the same effects as the above-mentioned implementation method.

[0267] In the case of adopting the integrated unit, the apparatus can include a processing module, a storage module. When the apparatus is applied to an electronic device, the processing module can be used to control and manage the actions of the electronic device. The storage module can be used to support the electronic device to execute the related program code and the like.

[0268] The processing module can be a processor or a controller, which can realize or execute various exemplary logical blocks, modules and circuits shown in combination with the disclosure of the present application. The processor can also be a combination of computing functions, such as one or more microprocessor combinations, digital signal processing (DSP) and microprocessor combinations, and the like. The storage module can be a memory.

[0269] In addition, the apparatus provided by the embodiments of the present application can be a chip, an assembly or a module, the chip can include a connected processor and a memory; wherein the memory is used to store instructions, when the processor calls and executes the instructions, the chip can execute the display method of a hand-drawn table provided by the above-mentioned embodiments.

[0270] The embodiment also provides a computer readable storage medium, the computer readable storage medium stores computer program code, when the computer program code runs on the computer, the computer executes the related method steps to realize the display method of a hand-drawn table provided by the above-mentioned embodiments.

[0271] The embodiment also provides a computer program product, when the computer program product runs on the computer, the computer executes the related steps to realize the display method of a hand-drawn table provided by the above-mentioned embodiments.

[0272] The apparatus, computer readable storage medium, computer program product or chip provided by the embodiment are used to execute the corresponding method provided above, and thus the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, and will not be repeated here.

[0273] Through the description of the above embodiments, those skilled in the art can understand that, for the convenience and brevity of description, only the division of the above functional modules is exemplified, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0274] In the embodiments provided in the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules or units is only a logical function division, and in actual implementation, there can be another division manner, for example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed ones can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0275] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A display method of a hand-drawn table applied to an electronic device, comprising: The method comprises: obtaining a hand-drawn table drawn by a user, and converting the hand-drawn table into a table image; performing table element segmentation on the table image to obtain table elements in the table image; wherein the table elements in the table image comprise text elements and frame line elements; determining coordinate information of the text elements and coordinate information of the frame line elements in the table image; based on the coordinate information of the text elements, determining an inclination angle of each row of text in the table image, and based on the inclination angle, adjusting the inclination angle of the text in the table image to a target inclination angle; wherein the inclination angle is an inclination angle of each row of text relative to a reference line; based on the coordinate information of the frame line elements, fitting coordinate points of each frame line element in the table image to obtain a beautified frame line, and replacing the frame line in the table image with the beautified frame line; displaying the hand-drawn table after the adjustment and beautification on an electronic device.

2. The method of claim 1, wherein, The conversion of the hand-drawn table into a table image comprises: obtaining trajectory data when the user draws the hand-drawn table; based on the trajectory data, obtaining a plurality of drawing trajectories in the hand-drawn table; obtaining time sequence information of each of the plurality of drawing trajectories; based on the time sequence information of each of the plurality of drawing trajectories, rendering the plurality of drawing trajectories to obtain the table image.

3. The method of claim 2, wherein, After the table element segmentation on the table image to obtain the table elements in the table image, the method further comprises: determining whether the table elements contain a critical text element; wherein the critical text element is a text element having a positional overlap area with at least two cell elements; in a case where it is determined that the critical text element exists, determining time sequence information of a drawing trajectory of the critical text element according to the table image; based on the time sequence information, determining a target cell to which the critical text element belongs among the at least two cell elements; based on the target cell, determining coordinate information of the critical text element in the table image.

4. The method of claim 2, wherein, The rendering of the plurality of drawing trajectories based on the time sequence information of each of the plurality of drawing trajectories to obtain the table image comprises: determining gray values of the plurality of drawing trajectories in the table image according to the time sequence information of each of the plurality of drawing trajectories; based on the gray values, rendering the plurality of drawing trajectories to obtain the table image.

5. The method of claim 1, wherein, The table element segmentation on the table image to obtain the table elements in the table image comprises: inputting the table image into a table element segmentation model to obtain the table elements in the table image; wherein the table element segmentation model is used to segment the input table image into a plurality of table elements, and the table element segmentation model is obtained based on a table database.

6. The method of claim 5, wherein, The table database is generated by the following method: determining a table matrix of a target table to be generated according to a preset number of rows and a preset number of columns; wherein each element in the table matrix corresponds to a cell in the target table; Traverse each element in the table matrix, set the characteristic attribute of the cell corresponding to each element; According to the preset frame line library, determine the target frame line matched with the characteristic attribute of the cell; According to the characteristic attribute of the cell, determine the target text for filling into the cell in the preset text library; According to the target frame line and the target text, generate the target table; The position information of the target frame line and the filling area of the target text are used as the annotation information of the target table; Based on the target table and the annotation information, generate the table database.

7. The method of claim 6, wherein, The characteristic attribute of the cell includes a coordinate attribute; According to the preset frame line library, determine the target frame line matched with the characteristic attribute of the cell, including: In the preset frame line library, select the target frame line matched with the coordinate attribute of the cell; Or, select a random frame line in the preset frame line library, and adjust the random frame line based on the coordinate attribute to obtain the target frame line; wherein the target frame line matches the coordinate attribute of the cell; According to the characteristic attribute of the cell, determine the target text for filling into the cell in the preset text library, including: In the preset text library, select the target text matched with the coordinate attribute of the cell.

8. The method of claim 5, wherein, The table element segmentation model includes a feature extraction module and an element segmentation module, the element segmentation module includes a plurality of branches for segmenting different types of table elements, the same type of table element corresponds to the same branch, and the table image is input into the table element segmentation model to obtain the table elements in the table image, including: Through the feature extraction module, a plurality of levels of features are extracted from the table image, and the extracted plurality of levels of features are fused to obtain a high-dimensional feature map; Through the element segmentation module, the high-dimensional feature map is analyzed to determine the type of each feature point in the high-dimensional feature map; Each feature point is input into the branch corresponding to the type of the feature point to segment the high-dimensional feature map to obtain the table elements of the table image.

9. The method of claim 8, wherein, The plurality of branches include a first branch for segmenting a target element, the target element includes a text element or a cell element, and the each feature point is input into the branch corresponding to the type of the feature point to segment the high-dimensional feature map to obtain the table elements of the table image, including: The high-dimensional feature map is input into the first branch to predict the overall region and the center region of the target element; Based on the center region of the target element, determine the center point and the target number of the target element; Taking the center point of the target element as the aggregation center, the pixels in the overall region of the target element are aggregated to obtain a plurality of sub-regions; wherein the number of sub-regions is the target number, and each sub-region corresponds to a target element; Based on the plurality of sub-regions, obtain the target element in the table image.

10. The method of claim 8, wherein, The plurality of branches includes a second branch for segmenting the frame line element, and the second branch includes a spatial attention network, which includes a frame line feature extraction module, a horizontal fusion module, and a vertical fusion module. The inputting of the each feature point into the branch corresponding to the type of the feature point respectively to segment the high-dimensional feature map to obtain the table element of the table image comprises: The high-dimensional feature map is input into the frame line feature extraction module to obtain a frame line fusion feature map; The feature values corresponding to each row of feature points in the frame line fusion feature map are input into the horizontal fusion module as a sequence for feature fusion to obtain a horizontal frame line element in the frame line element; The feature values corresponding to each column of feature points in the frame line fusion feature map are input into the vertical fusion module as a sequence for feature fusion to obtain a vertical frame line element in the frame line element.

11. The method of claim 1, wherein, The adjusting of the inclination angle of the text in the table image to a target inclination angle based on the inclination angle comprises: The text is subjected to affine transformation based on the inclination angle to adjust the height of the text to a target height and adjust the inclination angle of the text in the table image to a target inclination angle.

12. The method of claim 1, wherein, The fitting of the coordinate points of each frame line element in the table image based on the coordinate information of the frame line element in the table element to obtain a beautified frame line comprises: The fitting comprises at least one of alignment, completion, and smoothing operation of the frame line in the table image.

13. The method of claim 1, wherein, Before the display of the hand-drawn table after the adjustment and beautification on the electronic device, the method further comprises: The font format corresponding to the text element is converted into a preset font format.

14. A display device of a hand-drawn table, applied to an electronic device, characterized by, The device comprises: An acquisition module configured to acquire a hand-drawn table drawn by a user and convert the hand-drawn table into a table image; A segmentation module configured to segment table elements in the table image to obtain the table elements in the table image, wherein the table elements in the table image comprise text elements and frame line elements; A determination module configured to determine coordinate information of the text elements and coordinate information of the frame line elements in the table image; A beautification module configured to determine an inclination angle of each row of text in the table image based on the coordinate information of the text elements, adjust the inclination angle of the text in the table image to a target inclination angle based on the inclination angle, wherein the inclination angle is an inclination angle of the each row of text relative to a reference line, fit coordinate points of each frame line element in the table image based on the coordinate information of the frame line element to obtain a beautified frame line, and replace the frame line in the table image with the beautified frame line; A display module configured to display the hand-drawn table after the adjustment and beautification on an electronic device. The electronic device comprises:

15. An electronic device, comprising: A memory configured to store executable program codes; A processor configured to call and run the executable program codes from the memory, so that the electronic device performs the method according to any one of claims 1 to 13. The electronic device comprises: A memory configured to store executable program codes; A processor configured to call and run the executable program codes from the memory, so that the electronic device performs the method according to any one of claims 1 to 13.

16. A computer readable storage medium characterized by: The computer readable storage medium stores a computer program which, when executed, implements the method of any one of claims 1 to 13.

Images