Single table structure identification method, device and equipment based on end-to-end complex relay protection setting value and medium

Abstract

The invention relates to the technical field of image data processing of a power system, in particular to an end-to-end complex relay protection setting value single table structure recognition method, device and equipment and a medium, and the method comprises the steps: carrying out the preprocessing of the existing relay protection setting value single complex table image data, and constructing a training and verification data set for supervision; introducing a dynamic form to adapt to a convolution module, and carrying out feature extraction on irregular deformation in the complex table; constructing a multi-scale attention module; designing a dynamic calibration module for cross-layer feature fusion, and constructing an end-to-end table structure recognition model by integrating a dynamic form adaptation convolution module and a multi-scale attention module; inputting the training data set into a table structure recognition model for training to obtain the table structure recognition model; new to-be-detected relay protection setting value single complex table image data are input into the table structure recognition model to obtain a table structure detection result of the image, a high-accuracy output result can be obtained, and complex table structure recognition is rapidly achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing of power systems, and in particular to a method, device, equipment and medium for identifying a single table structure based on end-to-end complex relay protection setting values. Background Art

[0002] Tables play a fundamental role in expressing and conveying information due to their inherent efficiency in data organization and information representation. Given the need for table recognition in multiple scenarios, there is an urgent need for end-to-end automatic complex table structure recognition technology, which aims to transform tables from visually rich documents into structured formats that cleverly capture data information, thereby facilitating numerous downstream applications such as information retrieval and question answering.

[0003] However, accurately identifying table structures remains a difficult task for the following reasons: 1) Complex structure. Tables have complex hierarchical header structures with a large number of empty cells or spanned cells, and the spacing between adjacent columns may vary greatly. At the same time, the size of tables varies, sometimes only a few rows, sometimes spanning multiple pages; in addition, table cells contain a variety of content, from single characters to more complex page objects such as paragraphs, graphics, and formulas. The above complexity makes it difficult for existing neural networks to accurately capture key features and distinguish between positive and negative samples, resulting in errors in table structure detection. 2) Irregular deformation. Many documents captured by cameras have poor image quality, deformed tables, or contain artifacts and noise, which complicates table detection and structure recognition because visual deformation may mislead the model to obtain negative matches, resulting in error accumulation, leading to incorrect text extraction. Therefore, overcoming these challenges and realizing end-to-end fully automatic complex table structure recognition technology is a difficult and urgent problem to be solved.

[0004] The information disclosed in this background technology section is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as acknowledging or suggesting in any form that the information constitutes the prior art already known to those skilled in the art. Summary of the invention

[0005] The present invention provides a method, device, equipment and medium for identifying a single table structure based on end-to-end complex relay protection settings, thereby effectively solving the problems in the background technology.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is: a method for identifying a single table structure of an end-to-end complex relay protection setting value, comprising the following steps:

[0007] S10: Preprocessing the existing complex table image data of the relay protection setting list; the preprocessing includes: marking the key table area structure, obtaining the real structure label of the table, and constructing a training data set and a verification data set for supervision;

[0008] S20: Introducing the dynamic morphological adaptation convolution module to extract features of irregular deformations in complex tables;

[0009] S30: Construct a multi-scale attention module to perceive multi-scale semantic features while reducing computational burden;

[0010] S40: designing a dynamic calibration module for cross-layer feature fusion, and constructing an end-to-end table structure recognition model by integrating the dynamic morphological adaptation convolution module and the multi-scale attention module;

[0011] S50: inputting the training data set into the table structure recognition model for training to obtain the table structure recognition model;

[0012] S60: Inputting the complex table image data of the new relay protection setting value list to be detected into the table structure recognition model to obtain the table structure detection result of the image.

[0013] Furthermore, in step S10, the key table area structure is marked, the true structure label of the table is obtained, and a training data set and a verification data set for supervision are constructed, which specifically include:

[0014] S11: Get the table structure of the area of ​​interest and containing all key information on the complex table image of the relay protection setting list;

[0015] S12: According to the table structure, use annotation software to mark at least four key points of the table header, sub-table header, horizontal cell, column cell, and cross-row and cross-column cells in the table, namely, the upper left, lower left, upper right, and lower right, to locate the bounding box;

[0016] S13: Refine the bounding box, obtain the refined key point coordinates and record them as labels, and compare them with the original Figure 1 To form a supervised training data set;

[0017] S14: Data augmentation of supervised training datasets using flipping, mirroring, random cropping, adding Gaussian and salt and pepper noise, and grayscale nonlinear change methods.

[0018] Furthermore, in step S20, the dynamic morphological adaptation convolution module is introduced to extract features of irregular deformations in complex tables, including:

[0019] The dynamic morphological adaptation convolutional module is constructed to include a bias learning convolutional layer, an interpolation module and a feature extraction convolutional layer;

[0020] Inputting the upper layer feature map into the bias learning convolutional layer to obtain a bias;

[0021] Inputting the bias into the interpolation module to obtain an integer bias;

[0022] The integerized bias and the upper-layer feature map are input into the feature extraction convolution layer to obtain the lower-layer feature map, and the network is trained through back propagation of the entire network to form a bias calculation model.

[0023] Furthermore, the bias learning convolution layer and the feature extraction convolution layer include two convolution layers, a group normalization layer and a Relu activation layer; the bias calculation model includes:

[0024]

[0025] Where C represents the position coordinate of the convolution kernel after deformation, i and j are the current horizontal and vertical directions respectively, m is the current iteration to the mth position, x and y are the horizontal and vertical coordinates of the specific position coordinates respectively, Δx and Δy represent the horizontal and vertical offsets learned at the current convolution kernel position respectively, and the formula uses an iterative method to calculate the potential position coordinates to be learned.

[0026] Furthermore, the interpolation module uses a bilinear interpolation method, which is expressed as:

[0027]

[0028] B(C,C')=b(C x ,C y ')·b(C y ,C y ');

[0029] Where C represents the decimal position of the learned coordinates, C′ lists all integer spatial positions, B is the bilinear interpolation kernel, which is divided into two one-dimensional kernels, and b(·) represents the one-dimensional kernel.

[0030] Furthermore, in step S30, a multi-scale attention module is constructed to perceive multi-scale semantic features while reducing the computational burden, including:

[0031] The multi-scale attention module is constructed to include a feed-forward network, an attention mechanism, and a multi-head parallel strategy;

[0032] The forward feature map is input into the feedforward network to obtain multi-scale features and output them after straightening, the straightened multi-scale features are merged, and the multi-scale features are obtained by using the attention mechanism and the multi-head parallel strategy.

[0033] Furthermore, the multi-scale attention model includes:

[0034] MAM(F)=φ o (Concat(head 0 ,...,head n ));

[0035] In the formula, φ o represents a linear mapping, n represents the number of heads, and Concat is a concatenation operation based on the channel dimension.

[0036] Furthermore, the feed-forward network includes layer normalization and depth-wise separable convolution.

[0037] Furthermore, the attention mechanism model includes:

[0038]

[0039] Where, d k Represents the dimension of each head, q, k, and v represent query, key, and value respectively.

[0040] Furthermore, the multi-head parallel strategy model includes:

[0041]

[0042] In the formula, Represents a linear map.

[0043] Furthermore, in step S40, the dynamic calibration module is designed for cross-layer feature fusion, and the specific steps include:

[0044] S41: The upper layer feature F a and the lower-level features F b Concatenate and fuse to obtain feature F cat ;

[0045] S42: The feature F cat Input a standard convolution unit, which consists of a 3×3 convolution kernel, batch normalization and ReLU activation function, to obtain F that encapsulates context feature information. fuse ;

[0046] S43: The F encapsulating the context feature information fuse Input the global average pooling layer GAP(·), then input a fully connected neural network FC(·), and finally input the Sigmoid activation function to obtain the weight Weight. The model includes:

[0047] Weight = Sigmoid(FC(GAP(F fuse)));

[0048] In the formula, GAP(·) represents the global average pooling operation, and FC(·) represents the fully connected neural network;

[0049] S44: Use the weight Weight to update the context representation to obtain feature F update , the model includes:

[0050] F update =Weight·F a +(1-Weight)·F b ;

[0051] In the formula, F a represents the upper layer features, F b Represents the lower layer features; Weight represents the obtained weight;

[0052] S45: Using residual fusion technology, the updated feature F update and the F that encapsulates the context feature information fuse Fusion, input into the standard convolution unit CBR(·) consisting of 3×3 convolution kernel, batch normalization and ReLU activation function, to obtain the final feature F dcm , the model of the dynamic calibration module includes:

[0053] F dcm =CBR(Concat(F update ,F fuse ));

[0054] In the formula, F dcm Represents the context representation after calibration. CBR(·) is a convolution unit composed of a 3×3 convolution kernel, which consists of a convolution layer, batch normalization, and ReLU. update represents the updated features, F fuse It indicates that the context feature information is encapsulated, and Concat is a concatenation operation performed according to the channel dimension.

[0055] The present invention also includes a device for identifying a single table structure of an end-to-end complex relay protection setting value, using the above method, including:

[0056] The data preprocessing module is used to preprocess the existing complex table image data of the relay protection setting value list; the preprocessing includes: marking the key table area structure, obtaining the real structure label of the table, and constructing a training data set and a verification data set for supervision;

[0057] Dynamic morphological adaptation convolution module, used to extract features from irregular deformations in complex tables;

[0058] Multi-scale attention module, which is used to perceive multi-scale semantic features while reducing computational burden;

[0059] A dynamic calibration module for cross-layer feature fusion, which builds an end-to-end table structure recognition model by integrating the dynamic morphological adaptation convolution module and the multi-scale attention module;

[0060] A model training module, used for inputting the training data set into the table structure recognition model for training to obtain the table structure recognition model;

[0061] The table structure detection module is used to input the complex table image data of the new relay protection setting value list to be detected into the table structure recognition model to obtain the table structure detection result of the image.

[0062] The present invention also includes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. Furthermore, when the processor executes the computer program, the method described above is implemented.

[0063] The present invention also includes a storage medium on which a computer program is stored. Further, when the computer program is executed by a processor, the method as described above is implemented.

[0064] The beneficial effects of the present invention are:

[0065] The present invention utilizes the constructed existing relay protection setting value single complex table image data set, and inputs it into an end-to-end table structure recognition model composed of the designed dynamic morphological adaptive convolution, multi-scale attention module and dynamic calibration module to obtain accurate table structure recognition results; through the integration of various design modules, the present invention utilizes the constructed existing relay protection setting value single complex table image data set, and inputs it into an end-to-end table structure recognition model composed of the designed dynamic morphological adaptive convolution, multi-scale attention module and dynamic calibration module to obtain accurate table structure recognition results; through the integration of various design modules, even in the case of occlusion, overlap, small objects and complex patterns, significant improvements are made in accurately capturing the shapes and features of complex objects, and high-accuracy output results can be obtained, and complex table structure recognition can be quickly realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0067] Figure 1It is a flow chart of a method for identifying a single table structure of an end-to-end complex relay protection setting value;

[0068] Figure 2 This is a deep learning network architecture diagram based on the end-to-end complex relay protection setting single table structure recognition method;

[0069] Figure 3 It is the structured data graph after recognition.

[0070] Figure 4 It is a structural diagram of a single-meter structure identification device based on end-to-end complex relay protection setting values;

[0071] Figure 5 A schematic diagram of the structure of a computer device. DETAILED DESCRIPTION

[0072] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0073] like Figure 1 As shown: A method for identifying a single table structure of an end-to-end complex relay protection setting value includes the following steps:

[0074] S10: Preprocess the existing complex table image data of relay protection setting list; the preprocessing includes: marking the key table area structure, obtaining the real structure label of the table, and constructing the training data set and verification data set for supervision; this step provides the necessary input data and labels for the model, which is the basis of model training;

[0075] S20: Introduce a dynamic morphological adaptation convolution module to extract features of irregular deformations in complex tables, thereby ensuring the accuracy of table structure detection. This step effectively extracts table features and solves the impact of irregular deformation on recognition accuracy.

[0076] S30: Construct a multi-scale attention module to perceive multi-scale semantic features while reducing the computational burden; the purpose of this step is to improve the model's ability to perceive features of different scales, thereby enhancing the understanding of complex table structures;

[0077] S40: Design a dynamic calibration module for cross-layer feature fusion, and build an end-to-end table structure recognition model by integrating the dynamic morphological adaptation convolution module and the multi-scale attention module. This step combines the previous modules to form a complete end-to-end model, realizes the fusion and calibration of cross-layer features, and further improves the recognition accuracy.

[0078] S50: inputting the training data set into the table structure recognition model for training to obtain the table structure recognition model;

[0079] S60: Input the complex table image data of the new relay protection setting value list to be detected into the table structure recognition model to obtain the table structure detection result of the image. This step is the stage of using the model for practical application, and the model results are applied to new data for prediction.

[0080] The present invention utilizes the constructed existing relay protection setting value single complex table image data set, and inputs it into an end-to-end table structure recognition model composed of the designed dynamic morphological adaptive convolution, multi-scale attention module and dynamic calibration module to obtain accurate table structure recognition results; through the integration of various design modules, the present invention utilizes the constructed existing relay protection setting value single complex table image data set, and inputs it into an end-to-end table structure recognition model composed of the designed dynamic morphological adaptive convolution, multi-scale attention module and dynamic calibration module to obtain accurate table structure recognition results; through the integration of various design modules, even in the case of occlusion, overlap, small objects and complex patterns, significant improvements are made in accurately capturing the shapes and features of complex objects, and high-accuracy output results can be obtained, and complex table structure recognition can be quickly realized.

[0081] By introducing the dynamic morphological adaptation convolution module, it can effectively handle the irregular deformation in complex tables, dynamically adjust the position and shape of the convolution kernel, adapt to different table structures, and ensure that the key areas of the table can be captured more accurately during the feature extraction process, thereby significantly improving the accuracy of table structure detection; the use of the multi-scale attention module enables the model to perceive multi-scale semantic features, especially when dealing with tables of various sizes and complexities, it can maintain efficient feature extraction capabilities. The multi-scale attention module can also reduce the computational burden, which means that even in resource-limited environments, the model can maintain efficient operation. Through data preprocessing steps (such as data enhancement) and the use of cross-layer feature fusion (such as dynamic calibration module), the generalization ability of the model is significantly improved. The data enhancement method can generate diverse training samples, so that the model can better adapt to different types of tabular data, and the cross-layer feature fusion technology can effectively combine feature information at different levels to improve the stability and robustness of the model in practical applications;

[0082] As a preferred embodiment of the above embodiment, in step S10, the key table area structure is marked, the real structure label of the table is obtained, and a training data set and a verification data set for supervision are constructed. The specific steps include:

[0083] S11: obtaining the table structure of the area of ​​interest and containing all key information on the complex table image of the relay protection setting sheet; in this step, the area of ​​interest is first obtained on the complex table image of the relay protection setting sheet. These areas contain the table structure of all key information, which can ensure that the data set only contains valid information that needs to be paid attention to and processed, thereby improving the training efficiency and accuracy of the model;

[0084] S12: Based on the table structure, use annotation software to mark at least the top left, bottom left, top right and bottom right key points of the table header, sub-header, horizontal cell, column cell and cross-row and cross-column cells in the table, and locate the bounding box; this step provides accurate label data for supervised learning to ensure that the model can learn the real structure of the table;

[0085] S13: Refine the bounding box, obtain the refined key point coordinates and record them as labels, and compare them with the original Figure 1 Together with the original Figure 1 Together, they form a supervised training data set. These precise label information can significantly improve the learning effect of the model and ensure the accuracy of the training process.

[0086] S14: Data enhancement of supervised training data sets using flipping, mirroring, random cropping, adding Gaussian and salt-and-pepper noise, and grayscale nonlinear change methods. Diverse training samples can be generated. Data enhancement technology enables the model to maintain good recognition performance in the face of a variety of complex situations (such as different table layouts, table contents, etc.), significantly improving the generalization ability of the model.

[0087] Considering that the traditional standard convolution kernel has a fixed shape, it will produce a fixed receptive field when processing table images and extracting key features. However, as the various types of table organizational structures in the challenge have targets of different scales or deformations that may correspond to different positions in the feature map, this requires the model to dynamically adjust its receptive field. Therefore, the following improvements are made:

[0088] In step S20, a dynamic morphological adaptation convolution module is introduced to extract features of irregular deformations in complex tables, including:

[0089] The dynamic morphological adaptation convolution module is constructed to include a bias learning convolution layer, an interpolation module, and a feature extraction convolution layer;

[0090] Input the upper feature map into the bias learning convolution layer to obtain the bias;

[0091] Input the bias into the interpolation module to obtain the integer bias;

[0092] The integerized bias and the upper-layer feature map are input into the feature extraction convolution layer to obtain the lower-layer feature map, and the network is trained through back propagation of the entire network to form a bias calculation model.

[0093] Compared with the standard convolution kernel with fixed shape, this method allows the convolution kernel to be adaptively adjusted according to the actual content in the feature map, expanding or reducing the receptive field, so as to better adapt to targets of different scales and deformations, and is particularly effective for various structures, cells of different sizes and deformations that may appear in complex table images; through interpolation (such as bilinear interpolation), the module ensures that the conversion of bias values ​​between continuous space and discrete space is smooth and distortion-free, and the integerized bias helps to accurately locate feature points in actual convolution operations and improve the accuracy of feature extraction; by combining the adjusted bias, the feature extraction convolution layer can extract more discriminative features while adapting to deformation, and the convolution operation is performed at multiple feature scales. At the same time, through back-propagation training, the entire network can adaptively adjust parameters, further improving the effectiveness and efficiency of feature learning.

[0094] In this embodiment, the bias learning convolution layer and the feature extraction convolution layer include two convolution layers (convolution layers with a convolution kernel size of 3), a group normalization layer and a Relu activation layer, the maximum pooling layer kernel size is 2, the stride is 2, the convolution kernel size of the rice convolution layer is 3, and the stride is 2; the bias calculation model includes:

[0095]

[0096] Where C represents the position coordinate of the convolution kernel after deformation, i and j are the current horizontal and vertical directions respectively, m is the current iteration to the mth position, x and y are the horizontal and vertical coordinates of the specific position coordinates respectively, Δx and Δy represent the horizontal and vertical offsets learned at the current convolution kernel position respectively, and the formula uses an iterative method to calculate the potential position coordinates to be learned.

[0097] The introduction of bias learning convolutional layers and bias calculation models enables the position of the convolution kernel to be dynamically adjusted according to actual features. This flexible receptive field adjustment method enables the model to more accurately extract the key features of the table when faced with complex and changeable table structures, thereby improving the accuracy of table structure recognition. The bias calculation model reduces the errors that may occur in different types of tables by learning position bias and dynamically adjusting the position of the convolution kernel, thereby improving the recognition accuracy of the model.

[0098] Among them, the interpolation module uses bilinear interpolation method, which is expressed as:

[0099]

[0100] B(C,C')=b(Cx ,C y ')·b(C y ,C y ');

[0101] Where C represents the decimal position of the learned coordinates, C′ lists all integer spatial positions, B is the bilinear interpolation kernel, which is divided into two one-dimensional kernels, and b(·) represents the one-dimensional kernel.

[0102] The purpose of the interpolation module is to process the bias generated by the bias learning convolution layer, converting it from a floating-point position to an integer position so that the feature extraction convolution layer can perform more accurate feature extraction operations; since convolution operations are usually performed on an integer grid, it is necessary to interpolate the learned floating-point bias; by using bilinear interpolation, the interpolation module can accurately map the learned floating-point coordinates to the nearest integer coordinates. This accuracy is particularly important for the recognition of complex table structures, because there may be many tiny details and features in the table that need to be accurately located and extracted.

[0103] Considering the self-attention mechanism in Transformer, due to its powerful long-distance modeling ability, it can effectively capture global information, but it is limited by its high computational and memory costs.

[0104] In this embodiment, in step S30, a multi-scale attention module is constructed to perceive multi-scale semantic features while reducing the computational burden, including:

[0105] The multi-scale attention module is constructed to include a feed-forward network, an attention mechanism, and a multi-head parallel strategy;

[0106] The forward feature map is input into the feedforward network to obtain multi-scale features and output them after straightening. The straightened multi-scale features are merged, and the multi-scale features are obtained by using the attention mechanism and multi-head parallel strategy. Through the extraction and fusion of multi-scale features, the model's ability to recognize table structures of different scales and complex deformations is improved. Multi-scale features help the model focus on details and global information at the same time, so as to more accurately detect and identify each element in the table.

[0107] As a preferred embodiment of the above, the multi-scale feature model includes:

[0108] MAM(F)=φ o (Concat(head 0 ,...,head n ));

[0109] In the formula, φ o represents a linear mapping, n represents the number of heads, and Concat is a concatenation operation based on the channel dimension.

[0110] Multiple heads are used, each head corresponds to an independent linear mapping operation. This design is similar to the multi-head self-attention mechanism in Transformer, which enables the model to process input features from different angles and different subspaces, thereby enhancing the model's expressiveness and feature capture capabilities.

[0111] Among them, the feedforward network includes layer normalization and depthwise separable convolution. Specifically, combining layer normalization with depthwise separable convolution can effectively improve the expressiveness and computational efficiency of the feedforward network. Layer normalization ensures the stability of features and the effectiveness of model training, while depthwise separable convolution reduces computational complexity while maintaining feature extraction capabilities.

[0112] As a preferred embodiment of the above, the attention mechanism model includes:

[0113]

[0114] Where, d k Represents the dimension of each head, q, k, and v represent query, key, and value respectively.

[0115] The multi-head attention mechanism allows the model to focus on the input data from multiple different angles and subspaces. Each head focuses on different features. Combining the information of all heads can more comprehensively capture the important features and structures in complex table images.

[0116] In this embodiment, the multi-head parallel strategy model includes:

[0117]

[0118] In the formula, Represents a linear map.

[0119] Since high-level features contain semantic information, while low-level features capture detailed information, their contributions to downstream tasks vary. Simply fusing contextual features may dilute semantic features, thereby reducing the performance of downstream tasks. To alleviate this problem, the dynamic calibration module in step S40 performs dynamic weight correction on contextual information. This method can effectively utilize cross-level contextual semantics to improve table structure recognition performance.

[0120] As a preference for the above embodiment, in step S40, a dynamic calibration module is designed for cross-layer feature fusion, specifically including:

[0121] S41: The upper layer feature F a and the lower-level features F b Concatenate and fuse to obtain feature F cat; By splicing and fusing the upper and lower layer features, it is possible to comprehensively utilize feature information at different levels, thereby improving the model's ability to understand complex table structures;

[0122] S42: Feature F cat Input a standard convolution unit, which consists of a 3×3 convolution kernel, batch normalization, and ReLU activation function, to obtain F that encapsulates context feature information. fuse ; The use of standard convolutional units effectively extracts and enhances the contextual information of features, providing rich contextual background for subsequent feature updates;

[0123] S43: encapsulate the context feature information fuse Input the global average pooling layer GAP(·), then input a fully connected neural network FC(·), and finally input the Sigmoid activation function to obtain the weight Weight. The model includes:

[0124] Weight = Sigmoid(FC(GAP(F fuse )));

[0125] In the formula, GAP(·) represents the global average pooling operation, and FC(·) represents the fully connected neural network;

[0126] The weights calculated by global average pooling and fully connected neural networks can dynamically adjust the weights of features according to contextual features, improving the model's adaptability and feature utilization efficiency;

[0127] S44: Use the weight Weight to update the context representation and obtain the feature F update , the model includes:

[0128] F update =Weight·F a +(1-Weight)·F b ;

[0129] In the formula, F a represents the upper layer features, F b Represents the lower layer features; Weight represents the obtained weight;

[0130] Through weight updating and residual fusion technology, the model can effectively integrate feature information, reduce information loss, and enhance the representation ability of the final features, thereby improving the accuracy and robustness of table structure recognition;

[0131] S45: Using residual fusion technology, the updated feature F update and F which encapsulates the context feature information fuseFusion, input into the standard convolution unit CBR(·) consisting of 3×3 convolution kernel, batch normalization and ReLU activation function, to obtain the final feature F dcm , the model of the dynamic calibration module includes:

[0132] F dcm =CBR(Concat(F update ,F fuse ));

[0133] In the formula, F dcm Represents the context representation after calibration. CBR(·) is a convolution unit composed of a 3×3 convolution kernel, which consists of a convolution layer, batch normalization, and ReLU. update represents the updated features, F fuse It indicates that the context feature information is encapsulated, and Concat indicates the concatenation operation based on the channel dimension.

[0134] The combination of residual fusion and standard convolutional units improves the expressiveness of features, enabling the model to better capture key features and structural information in table images.

[0135] like Figure 2 As shown in the figure, the deep learning network architecture based on the end-to-end complex relay protection setting single table structure recognition method is designed to process unstructured complex table documents. The recognition and detection accuracy of the table structure is improved through the collaboration of multiple modules. Document" (unstructured document) represents the original input data, "Conv, K = 3" (convolutional layer, K = 3) represents the extraction of features through the convolutional layer in the data preprocessing step, and the preprocessing and feature extraction of complex table data; the "DCM" (dynamic calibration module) in the figure processes complex table images through bias learning convolutional layers, interpolation modules and feature extraction convolutional layers to extract irregular deformation features; the "MAM" (multi-scale attention module) in the figure is used to perceive multi-scale semantic features and reduce the computational burden. This module improves the model's perception of features of different scales through feedforward networks, attention mechanisms and multi-head parallel strategies; the application of the "DCM" module in the figure and multiple feature fusion operations (such as upsampling and C2F modules) are related to the dynamic calibration module and the cross-layer feature fusion process, which are used to fuse information from different feature layers and optimize the detection accuracy of the table structure; the "Detect" (detection) module in the figure represents the application part of the model, and the trained model is used to detect the table structure of new input data.

[0136] Based on the method of the present invention, the following Table 1 shows the advantages of the method adopted by the present invention under three public data sets and various parameters.

[0137] The advantages of the table structure recognition network proposed in this invention for each indicator are described in the table. Compared with other methods, the table structure recognition network proposed in this invention achieves the best detection. The Precision, Recall and F1 coefficients in the ICDAR 2013 dataset are 96.2%, 98.1% and 97.1% respectively. The ablation experiment in Table 2 verifies the importance of each module.

[0138] Table 1 Advantages of the method used in the present invention under three public data sets and various parameters

[0139]

[0140] Table 2 Ablation experiments verify the importance of each module

[0141]

[0142]

[0143] In summary, the present invention designs a method for identifying a single table structure based on end-to-end complex relay protection settings, such as Figure 3 As shown, the structured data graph after the customized single recognition table recognition is input into the end-to-end table structure recognition model composed of the designed dynamic morphological adaptation convolution, multi-scale attention module and dynamic calibration module by using the constructed existing relay protection setting value single complex table image data set to obtain accurate table structure recognition results. Through the integration of various design modules, we can achieve significant improvements in accurately capturing the shapes and features of complex objects even in cases involving occlusion, overlap, small objects and complex patterns. The present invention can be well applied to the power industry, especially to solve the complexity of the table structure in the power relay protection system. The present invention can directly obtain accurate table structure recognition results.

[0144] The present invention also includes a device for identifying a single table structure of an end-to-end complex relay protection setting value, using the above method, such as Figure 4 As shown, including:

[0145] The data preprocessing module is used to preprocess the existing complex table image data of relay protection setting list; the preprocessing includes: marking the key table area structure, obtaining the real structure label of the table, and constructing the training data set and verification data set for supervision;

[0146] Dynamic morphological adaptation convolution module, used to extract features from irregular deformations in complex tables;

[0147] Multi-scale attention module, which is used to perceive multi-scale semantic features while reducing computational burden;

[0148] Dynamic calibration module for cross-layer feature fusion, which builds an end-to-end table structure recognition model by integrating dynamic morphological adaptation convolution module and multi-scale attention module;

[0149] A model training module is used to input a training data set into a table structure recognition model for training to obtain a table structure recognition model;

[0150] The table structure detection module is used to input the complex table image data of the new relay protection setting value list to be detected into the table structure recognition model to obtain the table structure detection result of the image.

[0151] See also Figure 5 A computer device 400 provided in an embodiment of the present application includes: a processor 410 and a memory 420, wherein the memory 420 stores a computer program executable by the processor 410, and when the computer program is executed by the processor 410, the above method is executed.

[0152] The embodiment of the present application further provides a storage medium 430 on which a computer program is stored. When the computer program is run by the processor 410, the above method is executed.

[0153] Among them, the storage medium 430 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Red-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic storage, flash memory, disk or optical disk.

[0154] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. "Multiple" means two or more, unless otherwise clearly and specifically defined.

[0155] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0156] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0157] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0158] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0159] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0160] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0161] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for identifying a single table structure of an end-to-end complex relay protection setting value is characterized in that: The steps include: S10: Preprocessing the existing complex table image data of the relay protection setting list; the preprocessing includes: marking the key table area structure, obtaining the real structure label of the table, and constructing a training data set and a verification data set for supervision; S20: Introducing the dynamic morphological adaptation convolution module to extract features of irregular deformations in complex tables; S30: Construct a multi-scale attention module to perceive multi-scale semantic features while reducing computational burden; S40: designing a dynamic calibration module for cross-layer feature fusion, and constructing an end-to-end table structure recognition model by integrating the dynamic morphological adaptation convolution module and the multi-scale attention module; S50: inputting the training data set into the table structure recognition model for training to obtain the table structure recognition model; S60: Inputting the complex table image data of the new relay protection setting value list to be detected into the table structure recognition model to obtain the table structure detection result of the image.

2. The method for identifying a single table structure based on end-to-end complex relay protection setting values ​​according to claim 1 is characterized in that: In step S10, the key table area structure is marked, the true structure label of the table is obtained, and a training data set and a verification data set for supervision are constructed, which specifically include: S11: Get the table structure of the area of ​​interest and containing all key information on the complex table image of the relay protection setting list; S12: According to the table structure, use annotation software to mark at least four key points of the table header, sub-table header, horizontal cell, column cell, and cross-row and cross-column cells in the table, namely, the upper left, lower left, upper right, and lower right, to locate the bounding box; S13: Refine the bounding box, obtain the refined key point coordinates and record them as labels, and form a supervised training data set together with the original image; S14: Data augmentation of supervised training datasets using flipping, mirroring, random cropping, adding Gaussian and salt and pepper noise, and grayscale nonlinear change methods.

3. The method for identifying a single table structure based on end-to-end complex relay protection setting values ​​according to claim 1 is characterized in that: In step S20, the dynamic morphological adaptation convolution module is introduced to extract features of irregular deformations in complex tables, including: The dynamic morphological adaptation convolutional module is constructed to include a bias learning convolutional layer, an interpolation module and a feature extraction convolutional layer; Inputting the upper layer feature map into the bias learning convolutional layer to obtain a bias; Inputting the bias into the interpolation module to obtain an integer bias; The integerized bias and the upper-layer feature map are input into the feature extraction convolution layer to obtain the lower-layer feature map, and the network is trained through back propagation of the entire network to form a bias calculation model.

4. The method for identifying a single table structure based on end-to-end complex relay protection settings according to claim 3 is characterized in that: The bias learning convolution layer and the feature extraction convolution layer include two convolution layers, a group normalization layer and a Relu activation layer; the bias calculation model includes: Where C represents the position coordinate of the convolution kernel after deformation, i and j are the current horizontal and vertical directions respectively, m is the current iteration to the mth position, x and y are the horizontal and vertical coordinates of the specific position coordinates respectively, Δx and Δy represent the horizontal and vertical offsets learned at the current convolution kernel position respectively, and the formula uses an iterative method to calculate the potential position coordinates to be learned.

5. The method for identifying a single table structure based on end-to-end complex relay protection setting values ​​according to claim 3 is characterized in that: The interpolation module uses a bilinear interpolation method, which is expressed as: B(C,C')=b(C x ,C y ')·b(C y ,C y '); Where C represents the decimal position of the learned coordinates, C′ lists all integer spatial positions, B is the bilinear interpolation kernel, which is divided into two one-dimensional kernels, and b(·) represents the one-dimensional kernel.

6. The method for identifying a single table structure based on end-to-end complex relay protection settings according to claim 1 is characterized in that: In step S30, a multi-scale attention module is constructed to perceive multi-scale semantic features while reducing the computational burden, including: The multi-scale attention module is constructed to include a feed-forward network, an attention mechanism, and a multi-head parallel strategy; The forward feature map is input into the feedforward network to obtain multi-scale features and output them after straightening, the straightened multi-scale features are merged, and the multi-scale features are obtained by using the attention mechanism and the multi-head parallel strategy.

7. The method for identifying a single table structure based on end-to-end complex relay protection settings according to claim 6 is characterized in that: The multi-scale attention model includes: MAM(F)=φ o (Concat(head 0 ,...,head n )); In the formula, φ o represents a linear mapping, n represents the number of heads, and Concat is a concatenation operation based on the channel dimension.

8. The method for identifying a single table structure based on end-to-end complex relay protection settings according to claim 6 is characterized in that: The feed-forward network includes layer normalization and depth-wise separable convolutions.

9. The method for identifying a single table structure based on end-to-end complex relay protection settings according to claim 6 is characterized in that: The attention mechanism model includes: Where, d k Represents the dimension of each head, q, k, and v represent query, key, and value respectively.

10. The method for identifying the structure of a single table based on end-to-end complex relay protection settings according to claim 6, characterized in that: The multi-head parallel strategy model includes: In the formula, Represents a linear map.

11. The method for identifying a single table structure based on end-to-end complex relay protection setting values ​​according to claim 1, characterized in that: In step S40, the design of the dynamic calibration module for cross-layer feature fusion specifically includes: S41: The upper layer feature F a and the lower-level features F b Concatenate and fuse to obtain feature F cat ; S42: The feature F cat Input a standard convolution unit, which consists of a 3×3 convolution kernel, batch normalization and ReLU activation function, to obtain F that encapsulates context feature information. fuse ; S43: The F encapsulating the context feature information fuse Input the global average pooling layer GAP(·), then input a fully connected neural network FC(·), and finally input the Sigmoid activation function to obtain the weight Weight. The model includes: Weight=Sigmoid(FC(GAP(F fuse ))); In the formula, GAP(·) represents the global average pooling operation, and FC(·) represents the fully connected neural network; S44: Use the weight Weight to update the context representation to obtain feature F update , the model includes: F update =Weight·F a +(1-Weight)·F b ; In the formula, F a represents the upper layer features, F b Represents the lower layer features; Weight represents the obtained weight; S45: Using residual fusion technology, the updated feature F update and the F that encapsulates the context feature information fuse Fusion, input into the standard convolution unit CBR(·) consisting of 3×3 convolution kernel, batch normalization and ReLU activation function, to obtain the final feature F dcm , the model of the dynamic calibration module includes: F dcm =CBR(Concat(F update ,F fuse )); In the formula, F dcm Represents the context representation after calibration. CBR(·) is a convolution unit composed of a 3×3 convolution kernel, which consists of a convolution layer, batch normalization, and ReLU. update represents the updated features, F fuse It indicates that the context feature information is encapsulated, and Concat is a concatenation operation performed according to the channel dimension.

12. A device for identifying single-table structure of end-to-end complex relay protection settings, characterized in that: Use of the method according to any one of claims 1 to 11, comprising: The data preprocessing module is used to preprocess the existing complex table image data of the relay protection setting value list; the preprocessing includes: marking the key table area structure, obtaining the real structure label of the table, and constructing a training data set and a verification data set for supervision; Dynamic morphological adaptation convolution module, used to extract features from irregular deformations in complex tables; Multi-scale attention module, which is used to perceive multi-scale semantic features while reducing computational burden; A dynamic calibration module for cross-layer feature fusion, which builds an end-to-end table structure recognition model by integrating the dynamic morphological adaptation convolution module and the multi-scale attention module; A model training module, used for inputting the training data set into the table structure recognition model for training to obtain the table structure recognition model; The table structure detection module is used to input the complex table image data of the new relay protection setting value list to be detected into the table structure recognition model to obtain the table structure detection result of the image.

13. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 11 is implemented.

14. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images