A method and system for adaptive alignment of table data rows and columns based on edge visual computing

Abstract

The application discloses a kind of based on edge visual computing's table data row and column self-adapting alignment method and system, the method includes creating table video stream self-adapting extractor, real-time intercepts video frame containing table data, identifies target table position information, extracts table image to be detected;For target table image, construct the self-adapting alignment model based on visual computing, splice multiple sets of table profile of video frame, align table row and column data;Based on table row and column data, construct deep reinforcement learning model, fine adjustment table structured information;Based on table structure information, customization identification and processing cell text data;According to table structure and text data, carry out structured encoding and storage.The application realizes the automatic identification, storage and alignment of table data in video stream, improves the acquisition efficiency and accuracy of table data, greatly saves manpower cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of visual computing and automatic recognition technology, in particular to a method and system for adaptive alignment of table data rows and columns based on edge visual computing. BACKGROUND

[0002] Visual computing is a technology that simulates the human visual system to interpret and understand images and videos. With the rapid development of computer vision and machine learning, visual computing technology has made significant progress, especially with the introduction of deep learning, enabling computers to recognize and process complex patterns in images with unprecedented accuracy. With the deepening development of digitization and automation, a large amount of information and data is disseminated through video forms, especially in online education, remote meetings, data analysis reports, and multimedia content. In these fields, table data in videos often carries key information, such as important statistical data, relational data, and complex data sets. Therefore, the ability to automatically and accurately identify and extract text, numbers, and other data from videos not only greatly improves the efficiency of information processing, but also reduces the error rate of manual entry, further promoting the immediate sharing and utilization of data. In addition, automatic recognition technology has broad application prospects in big data analysis, machine learning, artificial intelligence, and other fields, helping to mine valuable information from massive video content, providing support for decision-making, and promoting the development of intelligent management and scientific research. Therefore, automatic recognition of table data in videos has become a key technology link connecting data flow and intelligent processing in the information age.

[0003] Edge computing, on the other hand, is an open platform that integrates network, computing, storage, and application capabilities, and meets real-time, data optimization, privacy protection, and other needs through localized data processing. Unlike traditional cloud computing, edge computing emphasizes processing data near the source of generation rather than relying on remote data centers.

[0004] Therefore, using edge computing to recognize table data in videos is a challenging task. Video streams often contain dynamically changing scenes and different perspectives, which can affect the visibility and recognition accuracy of tables. In addition, the size, style, and complexity of tables in videos can vary greatly, which requires the recognition system to have high flexibility and adaptability.

[0005] Based on the above situation, in order to accurately extract table data from videos, it is particularly important to design a method and system for adaptive alignment of table data rows and columns based on edge visual computing. SUMMARY

[0006] This invention aims to provide a method for adaptive alignment of rows and columns of table data based on edge visual computing. By designing an adaptive keyframe extractor, a video table edge detection method, an adaptive alignment model based on visual computing, and a table structure fine-tuning technique based on deep reinforcement learning, it achieves efficient recognition of video tables using edge computing.

[0007] One of the objectives of this invention is achieved through the following technical solution:

[0008] A method for adaptive row and column alignment of tabular data based on edge visual computation includes the following steps:

[0009] Step S1: Create a table video stream adaptive extractor to capture video frames containing table data in real time, identify the target table location information, and extract the image of the table to be detected;

[0010] Step S2: For the target table image, construct an adaptive alignment model based on visual computing, stitch together the table outlines of multiple video frames, and align the table row and column data;

[0011] Step S3: Based on the table row and column data, construct a deep reinforcement learning model and fine-tune the table's structured information;

[0012] Step S4: Based on the table structure information, customize the identification and processing of cell text data;

[0013] Step S5: Perform structured encoding and storage based on the table structure and text data.

[0014] Furthermore, step S1 specifically includes:

[0015] Step S101: For each video stream to be processed, the video is broken down into individual frames;

[0016] Step S102: For each video frame, construct an adaptive extractor based on changes in video content to extract keyframes from the video stream. The specific steps for constructing the adaptive extractor based on changes in video content are as follows:

[0017] First, select the first frame of the video stream as the keyframe;

[0018] Then, calculate the current frame f. i Compared to the previous keyframe f i-1 Similarity S f Finally, compare the similarity of the current frame with the similarity threshold S. t The size of S t > S f Then, the current frame f is extracted as the keyframe, and the similarity to the next frame is calculated based on the current frame. The similarity calculation formula is as follows:

[0019]

[0020] wherein H1 and H2 are color histogram vectors of two video frames calculated by cosine similarity, E1 and E2 are image edge feature vectors of two video frames, T1’ and T2’ are texture feature vectors of two video frames, cov(•) is a covariance function, and σ is a standard deviation;

[0021] Step S103: performing table edge detection on each key frame to intelligently identify a table region in the video stream and mark a table frame contour;

[0022] Further, in the step S103, the method of table edge detection is specifically as follows:

[0023] First, for each key frame, a table filter F(x, y) is created to perform convolution operation with each pixel point P(x, y) in the image, and the specific formula is as follows:

[0024]

[0025] P’(x, y) = P(x, y)*G(x, y)

[0026] wherein σ is a standard deviation, exp(•) is an exponential function with natural constant e as the base, l and w are the length and width of the image respectively, x and y are the coordinates of the pixel point of the key frame, and P’(x, y) is the pixel point after filtering.

[0027] For each pixel point P’(x, y), the gradient amplitude K(x, y) thereof is calculated by the following method:

[0028]

[0029] According to the gradient amplitude of each pixel point, the table boundary gray value G(x, y) is calculated, and the table frame line edge is judged and marked, and the specific method is as follows:

[0030] G(x, y) = [K(x, y) – K(x+1, y+1)] 2 – [K(x, y) – K(x+1, y)] 2 – [K(x, y)- K(x, y+1)] 2

[0031] When G(x, y) < 0.5, the pixel point P(x, y) is marked as a table contour point; if G(x, y) > 0.5, it is marked as a noise point.

[0032] Further, the step S2 specifically includes:

[0033] Step S201: constructing a table feature point intelligent detection algorithm to identify k stable feature points Q(x, y) in each table image, and the table feature point intelligent detection algorithm is specifically:

[0034] Calculating the visual feature value E(x, y) of each pixel point, and the calculation formula is:

[0035]

[0036]

[0037] Wherein, T x R , T x G , T x B represent the partial derivative of the pixel point on the x coordinate of the red, green and blue three primary colors respectively; T y R , T y G , T y B represent the partial derivative of the pixel point on the y coordinate of the red, green and blue three primary colors respectively; is a table visual feature coefficient, which is an empirical value; K is an expected feature value, when the expected feature value K(x, y) of the coordinate point (x, y) is greater than 0, it is marked as a table feature point;

[0038] Step S202: constructing an adaptive alignment model based on visual calculation according to the position of the feature points, to automatically adjust and align the row and column information of the table images in different frames;

[0039] Step S203: based on the feature points and the row and column information, splicing multiple aligned table images into a complete table.

[0040] Further, in the step S202, the adaptive alignment model based on visual calculation is specifically:

[0041] Based on the jth feature point Q i j (x, y) in the ith table image, the visual feature center point (g x , g y ) of each frame table image and the table visual feature vector U are calculated, and the calculation formula of the visual feature center point is specifically:

[0042]

[0043] Wherein, Q j x and Q jy respectively represent the x-coordinate and y-coordinate of the jth feature point, I j R , I j G , I j B respectively represent the red, green and blue three-channel color values corresponding to the jth feature point;

[0044] The calculation formula of the above table visual feature vector U is:

[0045]

[0046] Wherein, u xx , u yy , u xy The value calculation method is as follows:

[0047]

[0048] Wherein, x j , y j respectively represent the horizontal and vertical coordinates of the center point of the table image, and the row and column characteristic values B(Q j x , Q j y , Q x j+1 , Q x j+1 ) < 0.5, which represents the table row and column to be aligned, wherein the calculation method of the row and column characteristic values is:

[0049]

[0050] Wherein Q j x , Q j y respectively represent the horizontal and vertical coordinates of the jth feature point, U i , U i+1 respectively represent the visual feature vectors of the i-th frame and the i+1-th frame table.

[0051] Further, the step S3 specifically comprises:

[0052] Step S301: Based on the existing table information, the row and column data are tree structure coded as the state space of deep reinforcement learning;

[0053] Step S302: Define the merging cell and the splitting cell operation as the action of deep reinforcement learning, which is used to change the state of the current cell;

[0054] Step S303: design a table action reward model for evaluating the adjustment effect of each action on the accurate table structure, if the action causes the table structure to be closer to the real structure, give a positive reward; if the action causes the structure to deviate from the real structure, give a negative reward; the table action reward model is specifically:

[0055]

[0056] Wherein, T1 and T2 respectively represent the tree coding representation of the target table structure and the current table state, s i And s j Respectively represent the i-th and j-th sub-tree of the above coding; k(·) is used to judge whether the structures of the two sub-trees are the same, 1 if the same, and -1 if different;

[0057] Step S304: implement a table image sequence encoder, which inputs a table image sequence and outputs a table image feature, and the table image sequence encoder is specifically:

[0058] Z t = W t ⊙conv(v t-1 , N Z )) + ReLU(conv(c t , M Z )

[0059] W t = conv(v t-1 , N W )) + ELU(conv(c t , M W )

[0060] O t = c t-1 ⊙(W t + 1)* Z t ⊙(W t – 1)

[0061] g t = ReLU(c t × R)

[0062] Wherein, c is a table image sequence, c t is a current table image, c t-1 is a previous table image, Z t is the information loss degree of the table image at t and t-1, W t is the information retention degree of the table image at t and t-1, M W and N W are table information weight matrices, MZ and N Z is the weight matrix with table missing information, O t is the feature matrix containing the key information of the table image at time t, is the Hadamard product, which means multiplying the elements at the corresponding positions of the two matrices, g t is the feature vector containing the key information of the video frame at time t, conv(•) is the convolution layer, ReLU(•) is the linear activation function, ELU(•) is the exponential linear activation function, and R is the linear transformation weight matrix.

[0063] Step S305: Based on the table image features, a deep reinforcement learning model is designed and trained to fine-tune the row and column structure of the table and output a tree-like code representing the table structure.

[0064] Further, in the step S305, the deep reinforcement learning model is specifically:

[0065] (1) A policy network is constructed to guide table adjustment actions, and its network structure is specifically:

[0066] π(a t |s t , θ) = softmax(Dense(Dense(LSTM(conv(O t ))))

[0067] where O t represents the input of the neural network, i.e., the feature matrix containing the key information of the table image at time t, conv(•) is the convolution layer, Dense(•) is the fully connected layer, LSTM(•) is the long short-term memory network layer, a t and s t are the action and state at time t, respectively, and θ represents the parameters of the policy network; the network output is the probability distribution of selecting each possible action.

[0068] (2) A value network is constructed to estimate the expected return of action a t in state s t , and its network structure is specifically:

[0069]

[0070] where ReLU(•) is the linear activation function, MaxPooling(•) is the maximum pooling layer, Flatten(•) is the flattening layer, Linear(•) is the linear layer, ;

[0071] (3) A policy loss function is designed to update the parameters of the policy network , and its specific formula is:

[0072]

[0073] wherein, is the entropy of the policy, which measures the randomness or diversity of the policy, β is the entropy regularization coefficient, A(s, a) is the advantage function, T s is the number of sub-trees in the tree coding, and R is the total reward value.

[0074] (4) Design a value loss function to update the parameters of the policy network , and the formula is specifically:

[0075]

[0076] wherein, , γ is the discount factor, is the regularization coefficient.

[0077] Further, the step S4 specifically comprises:

[0078] Step S401: According to the identified table structure information, the image is cut into individual cells;

[0079] Step S402: Design a customized cell character recognition method based on visual computing to recognize font, font size, color and other information, and encode them;

[0080] Step S403: For the character encoding information of each cell, a new tree node is constructed and inserted into the table structure coding tree.

[0081] Further, in the step S402, the customized cell character recognition method based on visual computing is specifically:

[0082] h0 = PreLU(v) × W 1 + b 1

[0083]

[0084] d = ReLU(h1) × W 2 + b 2

[0085]

[0086] p = softmax(h2) × W 3 + b 3

[0087]

[0088] q = Maxout(h3) x W 4 + b 4

[0089] wherein h0 is a character embedding visual vector, PreLU(•) is a parametric rectified activation function, v is a visual feature vector of the character, W 1 and b 1 are weight parameters and bias parameters of the visual feature embedding model; ln() is a logarithmic function, () T is a transpose function, h1 is a first dimension visual integrated vector, W c1 and b c1 are weight parameters and bias parameters of the self-attention mechanism layer c in the first dimension visual feature analysis model, L1 is a dimension of the embedding vector h1, W u1 and b u1 are weight parameters and bias parameters of the self-attention mechanism layer u in the first dimension visual feature analysis model; ReLU(•) is a linear activation function, W 2 and b 2 are weight parameters and bias parameters of the table character color recognition, d is a character color classification; h2 is a second dimension visual integrated vector, W c2 and b c2 are weight parameters and bias parameters of the self-attention mechanism layer c in the second dimension visual feature analysis model, L2 is a dimension of the embedding vector h2, W u2 and b u2 are weight parameters and bias parameters of the self-attention mechanism layer u in the second dimension visual feature analysis model; softmax(•) is a classification function, W 3 and b 3 are weight parameters and bias parameters of the table character font size recognition, p is a character font size; h3 is a third dimension visual integrated vector, W c3 and b c3 are weight parameters and bias parameters of the self-attention mechanism layer c in the third dimension visual feature analysis model, L3 is a dimension of the embedding vector h3, W u3 and b u3 are weight parameters and bias parameters of the self-attention mechanism layer u in the third dimension visual feature analysis model; Maxout(•) is a maximum output activation function, W 4 and b 4 are weight parameters and bias parameters of the table character font recognition, q is a classification result of the character font.

[0090] Further, the step S5 specifically comprises:

[0091] Step S501: encoding the extracted text data into a structured format according to the table data model;

[0092] Step S502: cleaning and formatting the table data, and formatting the specific data types;

[0093] Step S503: storing the structured data into the database, and selecting a suitable storage type according to the data structure and query requirements.

[0094] The second object of the present application is to provide a table data row and column adaptive alignment system based on edge visual computing, which comprises a memory, an edge processing device, and a computer program stored in the memory and capable of running on the edge processing device, wherein the edge processing device implements the method as described above when executing the computer program.

[0095] The beneficial effects of the present application include:

[0096] (1) By adaptively extracting key frames in the video stream, the required computing power and storage space of the edge computing system can be greatly reduced, making the table data extraction process more energy-efficient and efficient; at the same time, focusing on key frames can improve the accuracy of table data recognition and reduce errors caused by poor image quality or scene changes;

[0097] (2) Precise edge detection helps to accurately determine the outline and structure of the table, and by accurately identifying the boundaries of the table, it can effectively distinguish between table and non-table content, improving the accuracy of data extraction; at the same time, table edge detection is beneficial to more finely process various different table row and column structures, avoiding the negative impact of irregular shapes caused by camera angles, light or other factors on subsequent processing, and improving the universality and flexibility of the computing method;

[0098] (3) In the splicing process of multiple video key frames, the feature point detection algorithm can accurately locate the key features of each table, thereby providing accurate alignment reference points during the splicing process, ensuring the accuracy of the splicing result and the consistency of the data; in addition, the intelligent detection algorithm can adapt to the dynamic changes of the table in the video, ensuring timely and accurate capture of the table feature points at each stage;

[0099] (4) The adaptive alignment model based on visual computing can accurately identify and match the alignment points of different tables, ensuring the accuracy in the splicing process, and preserving the integrity and accuracy of the table structure; in addition, the adaptive alignment model can effectively deal with the distortion of the table in the video due to the shooting angle or other factors, ensuring that all table segments are correctly aligned and spliced, which is beneficial to subsequent data analysis and processing;

[0100] (5) In the process of aligning table rows and columns, the use of the three primary color help algorithm can more accurately identify color information, better distinguish tables from backgrounds, and avoid visual distortions caused by lighting and shooting angles, which helps improve the robustness of the algorithm under different lighting conditions and ensures the accuracy of table row and column alignment;

[0101] (6) In processing the merging and splitting tasks of table cells, the use of deep reinforcement learning methods can enable the algorithm to have adaptive learning ability, constantly optimize its strategy to achieve better performance, and help identify and process various complex cell merging and splitting patterns. At the same time, deep reinforcement learning methods can consider long-term benefits when making decisions, not just short-term results, which is particularly important for processing complex table structures that require considering multi-step operations and long-term impacts. In addition, the deep reinforcement learning model can continuously learn from new table videos to continuously improve its performance, learn how to recover from erroneous cell operations, and avoid similar errors in future splitting and merging cell decisions, thereby improving the robustness of the overall table recognition algorithm;

[0102] (7) In deep reinforcement learning, designing an accurate and effective reward model helps the algorithm quickly learn cell splitting and merging strategies, enhances the stability and robustness of the algorithm, reduces fluctuations and uncertainties in the learning process, and makes the algorithm's behavior more stable and predictable. At the same time, a good reward model will punish incorrect decisions, preventing the algorithm from learning incorrect or undesirable behavior patterns;

[0103] (8) The table image sequence encoder can process time-series table data, which helps improve the accuracy and efficiency of the algorithm in processing and analyzing video table data. In addition, this encoder can effectively extract key information from video frame sequences, which helps capture and understand the dynamic changes of tables in videos, filter noise and redundant information in videos, and significantly improve the efficiency of subsequent processing steps;

[0104] (9) In the training process of deep reinforcement learning, a more complex loss function is designed, which not only considers the effect of the strategy and the accuracy of the value estimate, but also adds entropy regularization to promote the exploratory nature of the strategy and weight regularization to prevent overfitting. In addition, such design can help balance exploration and utilization, prevent the model from being too greedy in pursuing short-term returns, and ensure the generalization ability of the model;

[0105] (10) The table in the video is converted into structured code and stored in the database, so that the query and retrieval of data become more efficient and convenient, users can quickly query to find the required information without manually browsing the entire video, which greatly saves the labor cost; at the same time, compared with storing the entire video, the database provides a stable and durable way to save important data, converting key data into structured format and storing in the database can greatly save storage space.

[0106] Other advantages, objects, and features of the application will be apparent to those skilled in the art from the following specification, and it is intended to cover any alternatives, modifications, or equivalents included within the scope of the application. BRIEF DESCRIPTION OF DRAWINGS

[0107] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings, in which:

[0108] Figure 1 The overall workflow of the present application is shown in the figure;

[0109] Figure 2 The key frame table edge detection of the present application is shown in the figure;

[0110] Figure 3 The table feature point intelligent detection algorithm of the present application is shown in the figure;

[0111] Figure 4 The adaptive alignment model based on visual computing of the present application is shown in the figure;

[0112] Figure 5 The reward model in deep reinforcement learning of the present application is shown in the figure;

[0113] Figure 6 The agent action execution in deep reinforcement learning of the present application is shown in the figure;

[0114] Figure 7 The table character structured coding of the present application is shown in the figure. DETAILED DESCRIPTION

[0115] The preferred embodiments of the present application will be described in detail below with reference to the accompanying drawings. It should be understood that the preferred embodiments are only for illustration of the present application, and are not intended to limit the protection scope of the present application.

[0116] A table data row and column adaptive alignment method based on edge visual computing of the present application, as shown in Figure 1 The method comprises the following steps:

[0117] Step S1: Create a table video stream adaptive extractor to capture video frames containing table data in real time, identify the target table location information, and extract the image of the table to be detected;

[0118] Step S2: For the target table image, construct an adaptive alignment model based on visual computing, stitch together the table outlines of multiple video frames, and align the table row and column data;

[0119] Step S3: Based on the table row and column data, construct a deep reinforcement learning model and fine-tune the table's structured information;

[0120] Step S4: Based on the table structure information, customize the identification and processing of cell text data;

[0121] Step S5: Perform structured encoding and storage based on the table structure and text data.

[0122] The specific steps of the above method will be further explained below through a specific embodiment.

[0123] In this embodiment, step S1 specifically includes the following steps:

[0124] Step S101: For each video stream to be processed, the video is broken down into individual frames;

[0125] Step S102: For each video frame, construct an adaptive extractor based on changes in video content to extract keyframes from the video stream. The specific steps for constructing the adaptive extractor based on changes in video content are as follows:

[0126] First, select the first frame of the video stream as the keyframe;

[0127] Then, calculate the current frame f. i Compared to the previous keyframe f i-1 Similarity S f ;

[0128] Finally, the similarity of the current frame is compared with the similarity threshold S. t The size of S t > S f Then, the current frame f is extracted as the keyframe, and the similarity to the next frame is calculated based on the current frame. The similarity calculation formula is as follows:

[0129]

[0130] Where H1 and H2 are the color histogram vectors of the two video frames calculated using cosine similarity, E1 and E2 are the image edge feature vectors of the two video frames, T1' and T2' are the texture feature vectors of the two video frames, cov(•) is the covariance function, and σ is the standard deviation.

[0131] In this embodiment, the values of the color histogram vectors of certain video frames are H1 = [1, 3, 2] and H2 = [2, 2, 1]; the values of the edge feature vectors are E1 = [1, 0, 0] and E2 = [0, 1, 0]; and the values of the texture feature vectors are T1 = [1, 3, 7, 8] and T2 = [12, 15, 16, 18], and the covariance cov(T1, T2) = 7.75 can be calculated; the standard deviations of T1 and T2 are σ T1 = 3.304 and σ T2 = 2.5; and the final calculation can obtain S f = 0.6908.

[0132] Step S103: performing table edge detection on each key frame to intelligently identify the table region in the video stream and mark the table frame contour.

[0133] In the step S103, the method of table edge detection is specifically as follows.

[0134] First, for each key frame, a table filter F(x, y) is created to perform convolution operation with each pixel point P(x, y) in the image, and the specific formula is as follows:

[0135]

[0136] P'(x, y) = P(x, y)*G(x, y)

[0137] wherein, is the standard deviation of the Gaussian distribution, exp(•) is the exponential function with the natural constant e as the base, l and w are the length and width of the image respectively, x and y are the coordinates of the key frame pixel point, and P'(x, y) is the pixel point after filtering.

[0138] In this embodiment, the coordinates of the key frame pixel point are (3, 4), the length and width of the image are l = 20 and w = 10 respectively, and the value of σ is 5, and F(3, 4) = 1 / 50π × exp(-0.360555127546) ≈ 0.0044 can be calculated.

[0139] For each pixel point P'(x, y), the gradient amplitude K(x, y) is calculated as follows:

[0140]

[0141] According to the gradient amplitude of each pixel point, the table boundary gray value G(x, y) is calculated, and the table frame line edge is judged and marked, and the specific method is as follows:

[0142] G(x, y) = [K(x, y) – K(x+1, y+1)] 2 – [K(x, y) – K(x+1, y)] 2 – [K(x, y)– K(x, y+1)] 2

[0143] When G(x, y) < 0.5, then mark the pixel point P(x, y) as a table outline point; if G(x, y) > 0.5, then mark as a noise point;

[0144] In this embodiment, the boundary gray values of the pixel point (542, 408) and the pixel point (465, 505) are G(542, 408) = 0.839 and G(465, 505) = 0.249 respectively, so that the pixel point (542, 408) is judged as a table outline point and the pixel point (465, 505) is a noise point.

[0145] Step S2 specifically comprises:

[0146] Step S201: constructing a table feature point intelligent detection algorithm to identify k stable feature points Q(x, y) in each table image;

[0147] In step S201, the table feature point intelligent detection algorithm specifically comprises:

[0148] Calculate the visual feature value E(x, y) of each pixel point, and the calculation formula is:

[0149]

[0150]

[0151] Wherein, T x R , T x G , T x B represent the partial derivatives of the pixel points on the x coordinate of the red, green and blue three primary colors respectively; T y R , T y G , T y B represent the partial derivatives of the pixel points on the y coordinate of the red, green and blue three primary colors respectively; is a table visual feature coefficient, which is an empirical value; K is an expected feature value, when the expected feature value K(x, y) of the coordinate point (x, y) is greater than 0, it is marked as a table feature point;

[0152] In the embodiment, as shown in Figure 3 the identified table feature points are (30, 252), (123, 244), (588, 234), (27, 304), (125, 298), (589, 291), (28, 361), (123, 353), (591, 340), (28, 415), (125, 406), (596, 397), (26, 472), (125, 466), (598, 457), (20, 528), (125, 523), (599, 513), (24, 587), (123, 582), (607, 570), (23, 625), (127, 626), (608, 625).

[0153] Step S202: According to the positions of the feature points, an adaptive alignment model based on visual computing is constructed to automatically adjust and align the row and column information of the table images in different frames;

[0154] In step S202, the adaptive alignment model based on visual computing is specifically:

[0155] Based on the jth feature point Q i j (x, y) in the ith table image, the visual feature center points (g x , g y ) of each table image and the table visual feature vector U are calculated, and the calculation formula of the visual feature center points is specifically:

[0156]

[0157] wherein Q j x and Q j y are the x coordinate and y coordinate of the jth feature point respectively, I j R , I j G , and I j B represent the red, green and blue three-channel color values corresponding to the jth feature point respectively.

[0158] The calculation formula of the above table visual feature vector U is:

[0159]

[0160] wherein u xx , uyy xy The value is calculated as follows:

[0161]

[0162] wherein x j , y j represent the horizontal and vertical coordinates of the center point of the table image, and the row and column characteristic values B(Q j x , Q j y , Q x j+1 , Q x j+1 < 0.5, then it represents a table row and column to be aligned, wherein the row and column characteristic values are calculated as follows:

[0163]

[0164] wherein Q j x , Q j y are the horizontal and vertical coordinates of the jth feature point, and U i , U i+1 are the visual feature vectors of the i-th frame and the i+1-th frame table.

[0165] In this embodiment, the tables in the two video frames shown in Figure 3 are aligned based on visual calculation, and the result is shown in Figure 4 .

[0166] Step S203: based on the feature points and row and column information, the plurality of aligned table images are spliced into a complete table.

[0167] Step S3 specifically includes:

[0168] Step S301: based on the existing table information, the row and column data are tree structure coded as a state space of deep reinforcement learning;

[0169] Step S302: the merging cell and the segmented cell operation are defined as actions of deep reinforcement learning, for changing the state of the current cell;

[0170] Step S303: a table action reward model is designed for evaluating the adjustment effect of each action on the accurate table structure, if the action causes the table structure to be closer to the real structure, a positive reward is given, if the action causes the structure to deviate from the real structure, a negative reward is given; the table action reward model is specifically:

[0171] ​In step S303, the table action reward model is specifically:

[0172]

[0173] wherein T1 and T2 represent the tree coding of the target table structure and the current table state respectively, s i and s j represent the i-th and j-th sub-tree of the above coding respectively; k(•) is used to determine whether the structures of the two sub-trees are the same, and is 1 if the same, and -1 if different;

[0174] In this embodiment, the coding of the target table and the current table state is as shown in Figure 5 The value of each node in the tree coding is calculated, and the result is k(G, G) = 0, k(F, F) = 1, k(E, E) = 1, k(D, D) = 1, k(C, C) = 1, k(B, B) = 3, k(A, A) = 7, and finally the final reward value r = 7 can be obtained.

[0175] In this embodiment, the specific action execution of the agent is as shown in Figure 6 For the table structure coding state shown in Figure 6 (a), the following actions are performed: action a1: merge cell 25 and cell 26, action a2: merge cell 29 and cell 30, action a3: split cell 1, action a4: merge cell 25 and cell 27, action a5: merge cell 29 and cell 31, and action a6: merge cell 25 and cell 29; finally, the table structure coding shown in Figure 6 (b) is obtained.

[0176] Step S304: implement a table image sequence encoder, which inputs a table image sequence and outputs table image features.

[0177] In step S304, the table image sequence encoder is specifically:

[0178] Z t = W t ⊙conv(v t-1 , N Z )) + ReLU(conv(c t , M Z )

[0179] W t = conv(v t-1 , N W )) + ELU(conv(c t , M W )

[0180] O t = c t-1 ⊙(W t + 1)* Z t ⊙(W t – 1)

[0181] g t = ReLU(c t × R)

[0182] Where c is the table image sequence, c t is the current table image, c t-1 is the previous table image, Z t is the information loss degree of the table image at t and t-1, W t is the information retention degree of the table image at t and t-1, M W and N W are table information weight matrices, M Z and N Z are weight matrices with table loss information, O t is a feature matrix containing key information of the table image at t, is the Hadamard product, which means multiplying the elements at corresponding positions of two matrices, g t is a feature vector containing key information of the video frame at t, conv(•) is a convolution layer, ReLU(•) is a linear activation function, ELU(•) is an exponential linear activation function, and R is a linear transformation weight matrix.

[0183] Step S305: Based on the table image features, a deep reinforcement learning model is designed and trained to fine-tune the row and column structure of the table and output a tree-like code representing the table structure.

[0184] In step S305, the deep reinforcement learning model is specifically:

[0185] (1) A policy network is constructed to guide table adjustment actions, and its network structure is specifically:

[0186]

[0187] Where O t represents the input of the neural network, i.e., a feature matrix containing key information of the table image at t, conv(•) is a convolution layer, Dense(•) is a fully connected layer, LSTM(•) is a long short-term memory network layer, a t and s t are the action and state at t, respectively, and represents the parameters of the policy network; the network output is a probability distribution of selecting each possible action.

[0188] (2) Construct a value network to estimate the expected return of action a t in state s t , and its network structure is specifically as follows:

[0189]

[0190] where ReLU(•) is a linear activation function, MaxPooling(•) is a maximum pooling layer, Flatten(•) is a flattening layer, Linear(•) is a linear layer, ;

[0191] (3) Design a policy loss function to update the parameters of the policy network , and its specific formula is:

[0192]

[0193] where H(π(a t |s t , θ)) is the entropy of the policy, which is used to measure the randomness or diversity of the policy, β is the entropy regularization coefficient, A(s, a) is the advantage function, T s is the number of sub-trees in the tree encoding, and R is the total reward value.

[0194] (4) Design a value loss function to update the parameters of the policy network , and its formula is specifically as follows:

[0195]

[0196] where , γ is the discount factor, and λ is the regularization coefficient.

[0197] Step S4 specifically includes:

[0198] Step S401: According to the identified table structure information, the image is cut into individual cells;

[0199] Step S402: Design a customized cell character recognition method based on visual computing to recognize font, font size, color and other information, and encode them;

[0200] Step S403: For the character encoding information of each cell, construct a new tree node and insert it into the table structure encoding tree.

[0201] In step S402, the customized cell character recognition method based on visual computing is specifically as follows:

[0202] h0 = PreLU(v) × W 1 + b 1

[0203]

[0204] d = ReLU(h1) × W 2 + b 2

[0205]

[0206] p = softmax(h2) × W 3 + b 3

[0207]

[0208] q = Maxout(h3) × W 4 + b 4

[0209] wherein h0 is a character embedding visual vector, PreLU(•) is a parametric rectified activation function, v is a visual feature vector of a character, W 1 and b 1 are weight parameters and bias parameters of a visual feature embedding model; ln() is a logarithmic function, () T is a transpose function, h1 is a first-dimensional visual integrated vector, W c1 and b c1 are weight parameters and bias parameters of a self-attention mechanism layer c in a first-dimensional visual feature analysis model, L1 is a dimension of the embedding vector h1, W u1 and b u1 are weight parameters and bias parameters of a self-attention mechanism layer u in the first-dimensional visual feature analysis model; ReLU(•) is a linear activation function, W 2 and b 2 are weight parameters and bias parameters of table character color recognition, d is a character color classification; h2 is a second-dimensional visual integrated vector, W c2 and b c2 are weight parameters and bias parameters of a self-attention mechanism layer c in a second-dimensional visual feature analysis model, L2 is a dimension of the embedding vector h2, W u2 and b u2 are weight parameters and bias parameters of a self-attention mechanism layer u in the second-dimensional visual feature analysis model; softmax(•) is a classification function, W 3 and b 3 are weight parameters and bias parameters of table character font size recognition, p is a character font size; h3 is a third-dimensional visual integrated vector, W c3 and b c3Let L3 be the weight and bias parameters of the self-attention mechanism layer c in the third-dimensional visual feature analysis model, and let W be the dimension of the embedding vector h3. u3 and b u3 Here are the weight and bias parameters of the self-attention mechanism layer u in the third-dimensional visual feature analysis model; Maxout(•) is the maximum output activation function, W 4 and b 4 The weight and bias parameters are used for character font recognition in the table, and q is the character font classification result.

[0210] In this embodiment, for example, Figure 7 The table shown uses adaptive character encoding. In the cell containing "Goods Status", the font is identified as bold, size 19, and color close to black. Therefore, its final encoding is {[Xe8xb4xa7,f9d, 19, 090909], [Xe5x86xb5, f9d, 19, 0a0a0a]}. In the cell containing "Spot Goods", the font is identified as bold, size 23, and color close to red. Therefore, its final encoding is {[xe7x8exb0, 9fd, 23,c71e28], [xe8xb4xa7,f9d, 23, ca1922]}.

[0211] Step S5 specifically includes:

[0212] Step S501: Based on the tabular data model, encode the extracted text data into a structured format, such as JSON, XML, etc.

[0213] Step S502: Clean and format the table data. Cleaning includes removing extra spaces and correcting data errors. Formatting involves formatting specific data types (such as dates, numbers, and times).

[0214] Step S503: Store the structured data in the database, and select the appropriate storage type according to the data structure and query requirements.

[0215] Based on the characteristics of the edge computing system itself, the application provides a table data row and column adaptive alignment method based on edge visual computing. This technology can intelligently identify and adjust the rows and columns of the table to ensure accurate data extraction. By adaptively extracting key frames in the video stream, the required computing power and storage space of the edge computing system can be greatly reduced, making the table data extraction process more energy-efficient and efficient. Precise edge detection helps to accurately determine the outline and structure of the table. By accurately identifying the boundaries of the table, the table and non-table content can be effectively distinguished, improving the accuracy of data extraction. In the splicing process of multiple video key frames, the feature point detection algorithm can accurately locate the key features of each table, providing accurate alignment reference points during the splicing process to ensure the accuracy of the splicing result and the consistency of the data. At the same time, in the process of aligning the rows and columns of the table, the use of three primary color help algorithms can more accurately identify color information, better distinguish the table from the background, and avoid visual distortion of the image caused by factors such as lighting and shooting angle, which is beneficial to improving the robustness of the algorithm under different lighting conditions and ensuring the accuracy of the table row and column alignment. Finally, the table data in the video is converted into structured code and stored in the database, which is more suitable for various analysis and processing operations on the data, helps to extract valuable knowledge from the table data, and improves work efficiency. At the same time, compared with storing the entire video, converting the key data into a structured format and storing it in the database can greatly save storage space.

[0216] It should be appreciated that embodiments of the application can be realized or implemented by computer hardware, a combination of hardware and software, or through computer instructions stored in a non-transitory computer readable memory. The method can be implemented in a computer program configured with a non-transitory computer readable storage medium, using standard programming techniques, including the configuration of the storage medium with the computer program, wherein the storage medium thus configured causes the computer to operate in a specific and predefined manner according to the methods described in the specific embodiments and the accompanying drawings. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, if necessary, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. In addition, the program can be run on a programmed application-specific integrated circuit for this purpose.

[0217] Further, the operations of the processes described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and / or combinations thereof) can be implemented under the control of one or more computer systems configured with executable instructions (e.g., computer programs, one or more computer programs, or one or more applications), by hardware, or combinations thereof. The computer programs include machine- readable instructions that can be executed by one or more processors.

[0218] Further, the methods can be implemented in any suitable type of computing platform operatively connected to, including but not limited to, a personal computer, mini-computer, mainframe, workstation, networked or distributed computing environment, separate or integrated computer platforms, or in communication with charged particle tools or other imaging devices, and the like. Aspects of the present application can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated to the computing platform, such as a hard disk, optical read and / or write storage media, RAM, ROM, and the like, such that it can be read by a programmable computer to configure and operate the computer to perform the processes described herein when the storage medium or device is read by the computer. Further, the machine-readable code, or portions thereof, can be transmitted over wired or wireless networks. The present application described herein includes these and other different types of non-transitory computer-readable storage media when including instructions or programs that implement the steps described above in conjunction with microprocessors or other data processors. The present application also includes the computer itself when programmed according to the methods and techniques described herein.

[0219] Finally, it is to be understood that the above-described embodiments are merely meant as illustrative of the technical solutions of the present application and not limiting. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical solutions, and all should be included in the scope of the claims of the present application.

Claims

1. A method for adaptive row and column alignment of tabular data based on edge visual computation, characterized in that: The method includes the following steps: Step S1: Create a table video stream adaptive extractor to capture video frames containing table data in real time, identify the target table location information, and extract the image of the table to be detected; Step S2: For the target table image, construct an adaptive alignment model based on visual computing, stitch together the table outlines of multiple video frames, and align the table row and column data; Step S3: Based on the table's row and column data, construct a deep reinforcement learning model and fine-tune the table's structured information; Step S3 specifically includes: Step S301: Based on the existing table information, perform tree structure encoding on the row and column data to serve as the state space for deep reinforcement learning; Step S302: Define the operations of merging and splitting cells as actions of deep reinforcement learning, which are used to change the state of the current cell; Step S303: Design a table action reward model to evaluate the effect of each action on the precise table structure. A positive reward is given if the action makes the table structure closer to the true structure; a negative reward is given if the action causes the structure to deviate from the true structure. The table action reward model is as follows: Where T1 and T2 represent the tree-structured representation of the target table and the current table state, respectively, s i With s j These represent the i-th and j-th subtrees of the above encoding, respectively; k(•) is used to determine whether the structures of the two subtrees are the same, with a value of 1 if they are the same and a value of -1 if they are different; Step S304: Implement a table image sequence encoder, whose input is a sequence of table images and whose output is table image features. The table image sequence encoder is specifically as follows: WITH t = In t ⊙conv(v t-1 , N Z )) + ReLU(conv(c t , M Z ) W t = conv(v t-1 , N W )) + TWO(conv(c t ,M W ) ABOUT t = c t-1 ⊙(W t + 1)* WITH t ⊙(W t – 1) g t = ReLU(c t × R) Where c is the table image sequence, c t For the current table image, c t-1 For the image in the previous table, Z t W represents the degree of information loss in the table images at times t and t-1. t M represents the degree of information retention of the table images at times t and t-1. W and N W M is the table information weight matrix. Z and N Z For a weight matrix containing missing table information, O t Let g be the feature matrix containing key information from the table image at time t, ⊙ be the Hadamard product, which represents the element-wise multiplication of two matrices, and g be the feature matrix containing key information from the table image at time t. t Let be the feature vector containing key information of the video frame at time t, conv(•) be the convolutional layer, ReLU(•) be the linear activation function, ELU(•) be the exponential linear activation function, and R be the linear transformation weight matrix; Step S305: Based on the table image features, design and train a deep reinforcement learning model, finely adjust the row and column structure of the table, and output a tree-like code representing the table structure; Step S4: Based on the table structure information, customize the identification and processing of cell text data; Step S5: Perform structured encoding and storage based on the table structure and text data.

2. The method for adaptive row and column alignment of tabular data based on edge visual computing according to claim 1, characterized in that: Step S1 specifically includes: Step S101: For each video stream to be processed, the video is broken down into individual frames; Step S102: For each video frame, construct an adaptive extractor based on changes in video content to extract keyframes from the video stream. The specific steps for constructing the adaptive extractor based on changes in video content are as follows: First, select the first frame of the video stream as the keyframe; Then, calculate the current frame f. i Compared to the previous keyframe f i-1 Similarity S f ; Finally, the similarity of the current frame is compared with the similarity threshold S. t The size of S t > S f Then, the current frame f is extracted as the keyframe, and the similarity to the next frame is calculated based on the current frame. The similarity calculation formula is as follows: Where H1 and H2 are the color histogram vectors of the two video frames calculated using cosine similarity, E1 and E2 are the image edge feature vectors of the two video frames, T1' and T2' are the texture feature vectors of the two video frames, and cov(•) is the covariance function. ; Step S103: Perform table edge detection on each keyframe, intelligently identify table regions in the video stream, and mark the outline of the table frame.

3. The method for adaptive row and column alignment of tabular data based on edge visual computing according to claim 2, characterized in that: In step S103, the method for table edge detection is specifically as follows: First, for each keyframe, a table filter F(x, y) is created and convolved with each pixel P(x, y) in the image. The specific formula is as follows: in, exp(•) is an exponential function with the natural constant e as its base, l and w are the length and width of the image, respectively, and x and y are the coordinates of the keyframe pixels. These are the pixels after the filter has been applied; For each pixel The gradient magnitude K(x, y) is calculated as follows: Based on the gradient magnitude of each pixel, the grayscale value G(x, y) of the table boundary is calculated, and the table border edges are determined and marked. The specific method is as follows: G(x, y) = [K(x, y) – K(x+1, y+1)] 2 – [K(x, y) – K(x+1, y)] 2 – [K(x, y) – K(x, y+1)] 2 When G(x, y) < 0.5, the pixel point P(x, y) is marked as a table outline point; if G(x, y) > 0.5, it is marked as a noise point.

4. The method for adaptive row and column alignment of tabular data based on edge visual computing according to claim 1, characterized in that: Step S2 specifically includes: Step S201: Construct an intelligent detection algorithm for table feature points, identifying k stable feature points Q(x, y) in each table image. The specific intelligent detection algorithm for table feature points is as follows: The visual feature value E(x, y) of each pixel is calculated using the following formula: Among them, T x R T x G T x B T represents the partial derivatives of the pixel with respect to the x-coordinate in the three primary colors of red, green, and blue; y R T y G T y B This represents the partial derivatives of the pixel with respect to the y-coordinate in the three primary colors of red, green, and blue; , where is the visual feature coefficient of the table, and is an empirical value; K is the expected feature value. When the expected feature value K(x, y) of the coordinate point (x, y) is > 0, it is marked as a table feature point. Step S202: Based on the position of feature points, construct an adaptive alignment model based on visual computing to automatically adjust and align the row and column information of the table image in different frames; Step S203: Based on feature points and row and column information, stitch together multiple aligned table images into a complete table.

5. The method for adaptive row and column alignment of tabular data based on edge visual computing according to claim 4, characterized in that: In step S202, the adaptive alignment model based on visual computing is specifically as follows: Based on the j-th feature point Q in the i-th frame of the table image i j (x, y), calculate the visual feature center point (g) of each frame of the table graphic. x , g y And the visual feature vector U of the table, the specific formula for calculating the visual feature center point is as follows: Among them, Q j x With Q j y Let I be the x and y coordinates of the j-th feature point, respectively. j R I j G I j B These represent the red, green, and blue channel color values ​​corresponding to the j-th feature point, respectively. The formula for calculating the visual feature vector U in the above table is: Among them, u xx u yy u xy The method for obtaining the value is as follows: Where, x j y j The x and y coordinates of the center point of the table image are represented by B(Q), and the row and column feature values ​​of the previous and next table image frames are represented by Q. j x Q j y Q x j+1 Q x j+1 When the eigenvalue is less than 0.5, it indicates that the table rows and columns are to be aligned. The method for calculating the row and column eigenvalues ​​is as follows: Q j x Q j y U represents the x and y coordinates of the j-th feature point, respectively. i U i+1 These are the visual feature vectors of the tables in frame i and frame (i+1), respectively.

6. The method for adaptive row and column alignment of tabular data based on edge visual computing according to claim 1, characterized in that: In step S305, the deep reinforcement learning model is specifically as follows: (1) Construct a strategy network to guide table adjustment actions. The specific network structure is as follows: Among them, O t The input to the neural network is represented by the feature matrix containing key information from the table image at time t. `conv(•)` represents a convolutional layer, `Dense(•)` represents a fully connected layer, and `LSTM(•)` represents a long short-term memory network layer. t With s t These represent the action and state at time t, respectively. The network output is the probability distribution for selecting each possible action. (2) Construct a value network to estimate action a t In state s t The expected return is given by the following network structure: Among them, MaxPooling(•) is the max pooling layer, Flatten(•) is the flattening layer, and Linear(•) is the linear layer. ； (3) Design the policy loss function to update the parameters θ of the policy network. The specific formula is as follows: in, It is the entropy of the policy, used to measure the randomness or diversity of the policy, β is the entropy regularization coefficient, A(s, a) is the advantage function, and T s R represents the number of subtrees in the tree-based coding scheme, and R represents the total reward value. (4) Design a value loss function to update the parameters of the policy network. The specific formula is as follows: in, γ is the discount factor, and λ is the regularization coefficient.

7. The method for adaptive row and column alignment of tabular data based on edge visual computation according to claim 1, characterized in that: Step S4 specifically includes: Step S401: Based on the identified table structure information, the image is divided into individual cells; Step S402: Design a customized cell character recognition method based on visual computing to recognize information such as font, font size, and color, and encode it; Step S403: For the character encoding information of each cell, construct a new tree node and insert it into the table structure encoding tree.

8. The method for adaptive row and column alignment of tabular data based on edge visual computation according to claim 7, characterized in that: In step S402, the customized cell character recognition method based on visual computing is specifically as follows: h0 = PreLU(v) × W 1 + b 1 d = ReLU(h1) × W 2 + b 2 p = softmax(h2) × W 3 + b 3 q = Maxout(h3) × W 4 + b 4 Where h0 is the character embedding visual vector, PreLU(•) is the parameter-corrected activation function, v is the visual feature vector of the character, and W 1 and b 1 The weights and biases of the visual feature embedding model are given; ln() is the logarithmic function. T Let h1 be the transpose function, h1 be the first-dimensional visual integration vector, and W be the transpose function. c1 and b c1 Let L1 be the weight and bias parameters of the self-attention mechanism layer c in the first-dimensional visual feature analysis model, and W be the dimension of the embedding vector h1. u1 and b u1 Here are the weight and bias parameters of the self-attention mechanism layer u in the first-dimensional visual feature analysis model; ReLU(•) is the linear activation function, W 2 and b 2 Here, d represents the weight and bias parameters for character color recognition in the table, h2 represents the character color classification, and W represents the second-dimensional visual ensemble vector. c2 and b c2 Here, L2 represents the weight and bias parameters of the self-attention mechanism layer c in the second-dimensional visual feature analysis model, and W represents the dimension of the embedding vector h2. u2 and b u2 Here are the weight and bias parameters of the self-attention mechanism layer u in the second-dimensional visual feature analysis model; softmax(•) is the classification function, W 3 and b 3 The weight and bias parameters for character size recognition in tables are p, where p is the character size; h3 is the third-dimensional visual ensemble vector, and W is the weight and bias parameters. c3 and b c3 Let L3 be the weight and bias parameters of the self-attention mechanism layer c in the third-dimensional visual feature analysis model, and let W be the dimension of the embedding vector h3. u3 and b u3 Here are the weight and bias parameters of the self-attention mechanism layer u in the third-dimensional visual feature analysis model; Maxout(•) is the maximum output activation function, W 4 and b 4 The weight and bias parameters are used for character font recognition in the table, and q is the character font classification result.

9. A system for adaptive row and column alignment of tabular data based on edge visual computing, comprising a memory, an edge processing device, and a computer program stored in the memory and capable of running on a processor, characterized in that: When the edge processing device executes the computer program, it implements the method as described in any one of claims 1-8.

Images