A method and system for converting an excel file for a complex structure table image

Abstract

The application discloses a method and system for converting an Excel file from a complex structure table image, and the method comprises the following steps: acquiring image data, performing feature extraction on the image data by using a first trunk network to obtain feature information; performing first transformation on the feature information to obtain first feature information, inputting the first feature information into a first branch to obtain positioning information; inputting the first feature information into a second branch based on the positioning information to obtain a first sequence; replacing the first sequence according to the positioning information and the first sequence to obtain a first code; and converting the first code by using a first tool to obtain an Excel file, which has good performance for complex structure table images.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of document image conversion, and particularly relates to a method and system for converting an Excel file from a complex structure table image. BACKGROUND

[0002] With the rapid development of digital media, network and hardware technology, the number of images generated in the office scene has increased dramatically, and image data has become an important source of work data. A large number of paper documents, especially table documents, need to be digitized to achieve archiving and convenient retrieval. How to quickly extract the required information from the numerous images has become a key challenge. In the office environment, extracting information from table images is a common task, such as invoice reimbursement entry, medical reimbursement verification, cargo inventory, etc. However, this work is often tedious and prone to error, resulting in high labor costs. One of the key challenges in the field of document image recognition is the accurate identification of table structure. As a special type of document image, the combination of rows and columns and the characteristics of cell merging make it difficult for integrated processing of document images, thus becoming a research hotspot and difficulty in the industry and academia. Table image recognition technology has important significance in data reduction, office process optimization, and informatization and intelligent development. Specifically, the following points are made: Convert data format and reduce storage space. Reconstructing the content of the table image and converting it from image data to structured sequential text data can significantly reduce data storage. Taking the common value-added tax invoice image as an example, 10,000 images in JPEG compression format with a resolution of 1059x693 pixels have a storage size of about 1.70GB. After image reconstruction and conversion to an electronic table, only the key text information is stored, with a storage size of about 240MB, a reduction of about 80% in data storage; Portable office process. With the advent of the information age, whether it is an enterprise, government agency or individual, will generate a large amount of documents, such as contracts, reports, invoices, receipts, etc. The popularity of smartphones and digital cameras has made it a convenient and common practice to convert documents into digital images. Compared with traditional questionnaire data filling, shooting or scanning images have the characteristics of easy operation, fast and efficient, and are suitable for all ages, etc., which better meet the needs of modern office. Converting document images to searchable electronic formats can significantly improve the efficiency and accuracy of information search and retrieval, and thus improve work efficiency and decision-making quality; Realize the intelligentization of business. Unstructured image data usually needs to be preprocessed and feature extracted before valuable information can be extracted and converted into a structured form. Convolutional Neural Network (CNN) has shown great ability in image processing and can be used for feature extraction, target detection and image classification. Through the processing of CNN, image information can be effectively converted into structured data, which is more convenient for subsequent data analysis, mining and application. In the downstream tasks related to table images, structured data can significantly improve processing efficiency and simplify operation processes, such as analyzing the financial statements of listed companies, counting the in-out of goods, and extracting paper invoice information, etc.

[0003] The prior art has problems of being unable to recognize complex table images (with merged cells, large-span row cells, etc.) with high accuracy, requiring numerous post-processing steps and high computing resource requirements, etc. The technical design of the present application designs a lightweight ConvStem backbone network and a double-branch detection and recognition strategy, which can embed any OCR tool, and the entire model realizes end-to-end conversion of table images into HTML / Excel / Latex code. SUMMARY

[0004] In view of the above existing problems, the present application is proposed.

[0005] Therefore, the present application provides a method and system for converting an Excel file for a complex structure table image to solve the problems of the prior art, such as being unable to recognize complex table images (with merged cells, large-span row cells, etc.) with high accuracy, requiring numerous post-processing steps and high computing resource requirements, etc.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides a method for converting an Excel file for a complex structure table image, comprising:

[0008] Obtaining image data, performing feature extraction on the image data using a first backbone network to obtain feature information;

[0009] Performing a first transformation on the feature information to obtain first feature information, inputting the first feature information into a first branch to obtain positioning information;

[0010] Based on the positioning information, inputting the first feature information into a second branch to obtain a first sequence;

[0011] According to the positioning information and the first sequence, replacing the first sequence to obtain a first code;

[0012] Converting the first code using a first tool to obtain an Excel file.

[0013] As a preferred scheme of the method for converting an Excel file for a complex structure table image according to the present application, wherein: inputting the first feature information into the second branch comprises:

[0014] The first backbone network and the encoding head are combined to form a first encoder, and the image is input into the first encoder;

[0015] The information output by the first encoder is subjected to first vocabulary embedding.

[0016] As a preferred scheme of the method for converting an Excel file of a complex structure table image, the method further comprises:

[0017] The first decoded information after the first vocabulary embedding is obtained as a feature tensor;

[0018] The feature tensor is mapped to a vocabulary space through a linear layer to generate an output first sequence.

[0019] As a preferred scheme of the method for converting an Excel file of a complex structure table image, the method further comprises:

[0020] The table key field information is weighted and reorganized to obtain feature data of the key field information;

[0021] The feature data of the key field information is early fused and matched with the image dimension.

[0022] As a preferred scheme of the method for converting an Excel file of a complex structure table image, the method further comprises:

[0023] The feature data is sampled using a convolution kernel, and a segmentation mapping of the set convolution kernel output is utilized;

[0024] The positioning information is obtained according to the segmentation mapping, the positioning information is input to the first recognition tool, and the output text is stored.

[0025] As a preferred scheme of the method for converting an Excel file of a complex structure table image, the method further comprises:

[0026] The target sequence is subjected to position embedding and fused with the output feature of the first encoder to obtain the input of the first decoder;

[0027] The fused feature and the mask tensor are input into the first decoder to obtain the feature tensor.

[0028] As a preferred scheme of the method for converting an Excel file of a complex structure table image, the method further comprises:

[0029] The word embedding operation is performed on each word in the target vocabulary, and the sequence reference position embedding operation is performed on the word embedded sequence;

[0030] For each position and dimension, if the dimension is even, the value of the position embedding is calculated using a sine function;

[0031] If the dimension is odd, the value of the position embedding is calculated using a cosine function;

[0032] The first word table embedded information is obtained by adding the word embedding and the position embedding.

[0033] In a second aspect, the present application provides a system for converting an Excel file from a complex structure table image, comprising:

[0034] A feature extraction module acquires image data, and extracts features from the image data using a first backbone network to obtain feature information.

[0035] A transformation module is configured to perform a first transformation on the feature information to obtain first feature information, and input the first feature information into a first branch to obtain positioning information.

[0036] An input module is configured to input the first feature information into a second branch based on the positioning information to obtain a first sequence.

[0037] A replacement module is configured to replace the first sequence based on the positioning information and the first sequence to obtain a first code.

[0038] A conversion module is configured to convert the first code using a first tool to obtain an Excel file.

[0039] In a third aspect, the present application provides a computing device, comprising:

[0040] A memory and a processor.

[0041] The memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions, so as to implement the steps of the method for converting an Excel file from a complex structure table image.

[0042] In a fourth aspect, the present application provides a computer readable storage medium storing computer executable instructions, which, when executed by a processor, implement the steps of the method for converting an Excel file from a complex structure table image.

[0043] Compared with the existing technology, the beneficial effects of the present invention are as follows: the present invention realizes the full task chain of table image recognition, namely table detection, table structure recognition, and table content recognition, and can be converted into an actually usable and editable Excel file, which can effectively promote the automation and intelligence of business; it has commercial-level high precision, and on the currently most widely used PubTabNet and TableBank data sets (over 100,000 images), the average recognition accuracy exceeds 85% (TDES indicator), especially for table images with complex structures; it has low computing resource requirements and adopts a lightweight backbone network design. The backbone has only 24.08 million parameters, and even if the overall network model has only 100 million parameters, it can be deployed on a single machine. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 A schematic diagram of the overall process logic of a method for converting an Excel file into a complex structure table image according to an embodiment of the present invention;

[0046] Figure 2 A structural diagram of a table structure recognition model of the ConvStem backbone and encoder of a method for converting complex structure table images into Excel files according to an embodiment of the present invention;

[0047] Figure 3 This is a demonstration diagram of the recognition effect of Chinese statistical table images with header merging in a method for converting complex structure table images into Excel files according to an embodiment of the present invention;

[0048] Figure 4 This is a demonstration diagram of the recognition effect of a large table image with multiple rows and columns merged in a method for converting complex structure table images into Excel files described in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, the following detailed description of the specific embodiments of the present invention is given in conjunction with the accompanying drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in this field without creative work should fall within the scope of protection of the present invention.

[0050] Embodiment 1

[0051] Referring to Figures 1-2 For an embodiment of the present application, a method for converting an Excel file for a complex structure table image is provided, comprising:

[0052] S100: Obtain image data, and perform feature extraction on the image data by using a first backbone network to obtain feature information;

[0053] In an optional embodiment, the first backbone network can include a YOLO backbone network, and the feature extraction on the image data by using the YOLO backbone network includes identifying cells in a table by training a YOLO model, predicting the position and size of each cell, and the features extracted by YOLO in the detection process are used for subsequent text recognition or structure analysis, and are combined with other networks to realize end-to-end training from an image to structured data; the first backbone network can also include a convolutional neural network backbone network, and the feature extraction on the image data by using the convolutional neural network backbone network includes extracting features of a table image, and identifying the structure and content of the table;

[0054] In an optional embodiment, the first backbone network can also include a ResNet backbone network, and the feature extraction on the image data by using the ResNet backbone network includes that the feature extraction capability of ResNet can be used to identify complex patterns and structures in a table image, ResNet can learn more abstract features by increasing the depth of the network, and a ResNet pre-training model can be used for transfer learning to migrate the features learned from a large-scale data set to a table image recognition task.

[0055] In the embodiment of the present application, the first backbone network includes:

[0056] As shown in Figure 2 As shown in FIG. 1, CovnStem is used as a backbone network, the image information of the document table is processed by perceiving table information to obtain multi-level feature information Finter, and a multi-layer step convolution stem is designed as the ConvStem backbone network. The ConvStem backbone is designed as a combination of a plurality of 3×3 convolution modules, a batch normalization layer (BatchNorm), an activation layer (Activation), and the like, and finally a 1×1 convolution is used for transformation to match the d-dimensional features of the Transformer;

[0057] Specifically, the document table image data information is [H, W, D], which respectively represents the height, width and depth of the input image;

[0058] It should be noted that the ConvStem backbone network adopts a lightweight design, which requires less computing resources while maintaining high efficiency, which is particularly important for deploying models in resource-constrained environments. Through multiple 3x3 convolution modules, ConvStem can extract multi-level feature information (Finter) of images, which helps to capture subtle structures and patterns in table images. Combined with BatchNorm (normalization layer) and Activation (activation layer), it helps the network to learn more stable feature representations, while increasing nonlinearity, enabling the network to learn more complex function mappings. The final 1x1 convolution transformation ensures that the feature dimensions output by ConvStem match the input dimensions of Transformer, which helps to implement an end-to-end table image recognition process. ConvStem backbone network can adapt to different sizes of input images, and by processing data of [H, W, D] dimensions, it can flexibly process table images of different resolutions.

[0059] S200: performing a first transformation on the feature information to obtain first feature information, inputting the first feature information into a first branch to obtain positioning information;

[0060] S300: based on the positioning information, inputting the first feature information into a second branch to obtain a first sequence;

[0061] S400: replacing the first sequence according to the positioning information and the first sequence to obtain a first code;

[0062] S500: converting the first code using a first tool to obtain an Excel file.

[0063] In an optional embodiment, the first tool can include Table2Excel, which analyzes the structure of the HTML table, captures the row, column and cell data of the table, and formats these data into a format recognizable by Excel. Users can trigger the conversion process by calling the function provided by Table2Excel, such as convertToExcel, and passing in the ID of the HTML table or the table element itself; the first tool can also include SheetJS, which parses HTML table data, creates a new workbook, and fills table data into the workbook, and saves the workbook as an Excel file format.

[0064] In an optional embodiment, the first tool can also include TableExport, which triggers the export operation by listening to the click event on the table. When the user clicks a button or link, TableExport captures the table data and converts it to an Excel file.

[0065] In the embodiments of the present application, the first tool includes: js-export-excel;

[0066] The script file of the js-export-excel library is introduced in the HTML page, the HTML table data is prepared, the table structure is clear, the data format is correct, through the JavaScript code, an event listener can be added to the export button or link, which triggers the export operation when the user clicks, the function provided by js-export-excel is used, the ID of the HTML table or the table element itself is input, and the name of the exported file is input, the input HTML table is analyzed, the row, column and cell data of the table are extracted, and the extracted data is converted into Excel file format.

[0067] It should be noted that using js-export-excel as a tool to realize the conversion from HTML to Excel provides an intuitive user interface, allowing users to export table data by clicking buttons, enhancing user experience, automating the conversion process from HTML to Excel, reducing the need for manual copying and pasting of data, improving efficiency, ensuring data integrity and accuracy during the conversion process, and avoiding errors that may be introduced by manual operation.

[0068] It should be noted that the full task chain processing from the table image to the Excel file is realized, including table detection, structure recognition and content recognition, which promotes the automation and intelligentization of business processes, adopts a lightweight ConvStem backbone network design, the backbone network parameter is only 24.08 million, and the overall model parameter is controlled within 100 million, so that the model can be deployed on a single machine, reducing the hardware requirements, and by converting the table image into structured sequence text data, the data storage amount is significantly reduced.

[0069] In the embodiments of the present application, the above step S200 includes the following sub-steps A1-A10;

[0070] In A1, the key field information is weighted and adjusted to obtain feature data of the key field information;

[0071] In A2, the feature data of the key field information is early fused and matched with the image dimension.

[0072] In A3, the feature data is sampled using a convolution kernel, and a segmentation mapping of the set convolution kernel output is utilized;

[0073] In A4, the positioning information is obtained according to the segmentation mapping, the positioning information is input to the first recognition tool, and the output text is stored.

[0074] In an optional embodiment, the first recognition tool may include Baidu OCR, which integrates the Baidu OCR service into the system. By calling the API, the table image is sent to Baidu's server for text recognition. The recognized text data can be returned and used for subsequent table structure reconstruction and data filling. The first recognition tool may also include iFlytek OCR, which integrates the iFlytek OCR into the system and calls its service through the API interface for text recognition. The recognized text can be integrated into the table data to complete the conversion of the table image into structured data.

[0075] In an optional embodiment, the first recognition tool may also include self-deployed OCR, which is tightly integrated with the table image recognition system to achieve end-to-end automated processing.

[0076] In the embodiment of the present application, the first recognition tool includes: any OCR;

[0077] Input cell positioning information to OCR and store the output text in the dictionary. The model recognizes from top to bottom and from left to right, so the index of the dictionary is the sequential ID of the cell. Input cell positioning information to OCR and store the output text in the dictionary.

[0078] Specifically, the output of the backbone network is first transformed into a dimension transformation and then input into a first branch as a cell positioning branch, which obtains the location information of the cell;

[0079] Use axis attention to weight key domain information of the table, including rows, columns, and cells;

[0080] For example, row-wise attention performs linear layer, binarization, and normalization on the feature values ​​on the X-axis to highlight the quantitative information of the "row" structure in the image.

[0081] Specifically, the feature data of rows, columns, and cells are early fused to match the image dimensions, which can be expressed as:

[0082] f Spatial =Concat(f cell ,f row ,f coll )

[0083] Among them, f cell ,f row ,f coll They are the feature data of row, column and cell respectively, and Concat is early fusion;

[0084] The convolution kernel is further sampled to 64, and the predicted segmentation map is obtained using the set 1×1 convolution kernel output, which is expressed as:

[0085]

[0086] wherein K is a prediction classification type of "background", "boundary", "cell", and is a real set, H is the height of the input image, and W is the width of the input image;

[0087] The recognized cells are stored in the form of a dictionary respectively indicate the X value, Y value, width value, and height value of the cell center point.

[0088] It should be noted that in addition to using an OCR tool, this function can be directly embedded in the model, but it will increase the model and reduce the accuracy. The axis attention mechanism is used to weight the features of the row direction, column direction, and cell, improve the accuracy of table structure recognition, especially when processing complex tables. By storing the recognized text in a dictionary and using the cell order ID as the index, a structured output can be generated, which facilitates subsequent data processing and analysis. Early fusion of row direction, column direction, and cell feature data can more effectively match the image dimension and improve the efficiency of feature processing. Using a convolution kernel to sample to 64 and then using a 1x1 convolution kernel for prediction can reduce the amount of calculation while maintaining the accuracy of recognition.

[0089] In the above steps S300 and S400, the following sub-steps B1-B6 are included in the embodiments of the present application.

[0090] In B1, the first encoder is composed of the first backbone network and the encoding head, and the image is input to the first encoder.

[0091] In B2, the information output by the first encoder is subjected to first vocabulary embedding.

[0092] In B3, word embedding operation is performed on each word in the target vocabulary, and position embedding operation is performed on the sequence after word embedding.

[0093] In B4, for each position and dimension, if the dimension is even, the value of the position embedding is calculated using a sine function.

[0094] In B5, if the dimension is odd, the value of the position embedding is calculated using a cosine function.

[0095] In B6, the word embedding and the position embedding are added to obtain the information after the first vocabulary embedding.

[0096] Specifically, the output of the backbone network is first transformed to a dimension transformation and then input to a second branch, which is a sequence recognition branch. This branch converts the image information into an HTML sequence through a visual encoder, word embedding, and text decoder.

[0097] The ConvStem backbone network combines a 1x1 layer and an encoder (encoding head) to form a visual encoder, which has an output channel of 192, a down-sampling factor of 8, a convolution kernel of 5*5, an encoding layer of 4, a decoding layer of 4, and an input image size of 504*504.

[0098] The first vocabulary embedding includes embedding the information output by the visual encoder, embedding the label vocabulary data using a position embedding layer and a Token Embed (token embedding), and representing as:

[0099] T pos = PositionEmbedding(TokenEmbedding(Y))

[0100] where Y is a target HTML library, PositionEmbedding is a position embedding operation, and the output target sequence T pos ;

[0101] The position embedding is used to introduce position information in the sequence model, so that the model can perceive the position of each element in the input sequence. The definition of the position embedding is represented as:

[0102]

[0103] where pos is the position in the sequence, n is the index in the embedding dimension, and d is the size of the embedding dimension.

[0104] In the embodiments of the present application, after the steps B1-B6 in the above steps S300 and S400 are completed, the following steps B7-B10 are further included;

[0105] In B7: the first decoded information after the first vocabulary embedding is obtained, and a feature tensor is obtained;

[0106] In B8: the feature tensor is mapped to the vocabulary space through a linear layer to generate an output first sequence;

[0107] In B9: the target sequence is subjected to position embedding, and is fused with the output feature of the first encoder to obtain the input of the first decoder;

[0108] In B10: the fused feature and the mask tensor are input into the first decoder to obtain the feature tensor.

[0109] In an alternative embodiment, the first decoder can include a recurrent neural network (RNN) first decoder, inputting the text sequence recognized by OCR into the RNN, the RNN learning the context information of the text sequence through its hidden state, thereby generating or predicting the final text output; the first decoder can also include a self-encoder, inputting the features extracted by the encoder into the first decoder, the first decoder attempting to reconstruct the original table image or generate an HTML representation of the table.

[0110] In an alternative embodiment, the first decoder can also include a convolutional neural network (CNN) first decoder, inputting the feature maps extracted by the CNN encoder into transpose convolutional layers, which gradually increase the spatial dimensions of the feature maps, finally generating an output matching the size of the original input image;

[0111] In the embodiments of the present application, the first decoder includes a Transformer first decoder;

[0112] The information after embedding the vocabulary is input into the Transformer decoding; the target sequence T pos is input together with the output M features as input, and the decoded output is obtained through the Transformer decoder, represented as:

[0113] D = Decoder (T pos , M, T mask , T pad_ask )

[0114] Wherein, T mask is a mask tensor corresponding to the target sequence, used to prevent the model from seeing future words during decoding, and such a mask ensures the causal relationship of prediction, T pad_ask is a padding position in the target sequence, in order to make the length of all sequences consistent, padding symbols are added at the end of short sequences when processing variable-length sequences;

[0115] Specifically, the output feature tensor D is mapped to the vocabulary space through a linear layer (generator) to generate the final output text sequence, represented as:

[0116] O = Linear (D)

[0117] In order to write subsequent content, a "placeholder" string is inserted in the first sequence as an HTML sequence to occupy the content;

[0118] The text dictionary output by the fusion positioning branch is processed in a loop, and the first replacement includes replacing the "placeholder" with the corresponding text in the dictionary, and the output is the first code with complete table structure and text content in HTML.

[0119] It should be noted that the Transformer decoder is good at processing sequence data and can capture long-distance dependencies. By using a mask tensor, the causal relationship in the decoding process is ensured, that is, when predicting the current word, the model cannot see the future word. The feature tensor is mapped to the vocabulary space through a linear layer to generate the final output text sequence, which helps to accurately reconstruct the text content in the table.

[0120] The above is a schematic scheme of the method for converting the complex structure table image into an Excel file according to the present embodiment. It should be noted that the technical scheme of the system for converting the complex structure table image into an Excel file belongs to the same concept as the technical scheme of the method for converting the complex structure table image into an Excel file described above. The technical scheme of the system for converting the complex structure table image into an Excel file in the present embodiment is not described in detail, and can be referred to the description of the technical scheme of the method for converting the complex structure table image into an Excel file.

[0121] Embodiment 2

[0122] With reference to Figures 3-4 The present embodiment differs from the first embodiment in that an application embodiment of the method for converting the complex structure table image into an Excel file is provided to verify and illustrate the technical effects adopted in the method.

[0123] Take a Chinese statistical table image with merged table headers and a large table image with multiple rows and columns merged as examples;

[0124] The network model is designed with a double-branch strategy. First, the table cell positioning branch detects the cell space position and segments the image combined with OCR for text recognition. Second, the table structure recognition branch. The features output by the backbone network enter the visual encoder, which first uses the image encoder to extract features from the input image, then uses PositionalEmbed to embed the features, and finally uses the Transformer encoder to encode the embedded features. Finally, the HTML structure tag sequence is obtained, combined with the text dictionary output by the table cell positioning branch, to realize the table image content reconstruction task.

[0125] As Figure 3As shown, the model as a whole is also adaptive to Chinese table, accurately identifying the row and column information of large table, and the detection model identifies the existence of merged table header, and the structure recognition model accurately outputs the cross-column format in sequence generation. For complex statistical data existing in the table, the OCR tool identifies statistical numbers, time format and financial currency symbols, but due to the performance limitation of the OCR tool, there are some errors in the output;

[0126] As shown in Figure 4 The detection model identifies the spatial meaning of the blank area and assigns it to the row and column attributes in the structure sequence generation. For complex table images, the key is to restore their proper row and column correspondence. The model has a certain understanding of cross-row and cross-column text, and the final output content is well reconstructed to fit the original image information.

[0127] The double-branch strategy can effectively handle complex table structures, including merged cells and multi-row and multi-column layouts, improving the accuracy of table structure recognition. The structure recognition model can accurately identify merged table headers and output the correct cross-column format in sequence generation, which is crucial for maintaining the integrity of the table. The model can identify blank areas and assign them row and column attributes, which helps to restore the regularity of the table, especially when dealing with irregular tables. For complex table images, the model can understand and restore their proper row and column correspondence, which is very important for accurate reconstruction of table content.

[0128] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit it. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application, and they should be covered in the scope of the claims of the present application.

[0129] Example 3

[0130] The system for converting Excel files for complex structure table images in this embodiment includes:

[0131] A feature extraction module acquires image data and extracts features from the image data using a first backbone network to obtain feature information.

[0132] A transformation module is configured to perform a first transformation on the feature information to obtain first feature information, and input the first feature information into a first branch to obtain positioning information.

[0133] An input module is configured to input the first feature information into a second branch based on the positioning information to obtain a first sequence.

[0134] The replacement module is configured to replace the first sequence according to the positioning information and the first sequence to obtain a first code.

[0135] The conversion module is configured to convert the first code by using a first tool to obtain an Excel file.

[0136] The embodiment further provides a computing device suitable for converting an Excel file from a complex structure table image, comprising:

[0137] The memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions to implement the method for converting an Excel file from a complex structure table image.

[0138] The embodiment further provides a storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the method for converting an Excel file from a complex structure table image.

[0139] The storage medium provided by the embodiment and the method for converting an Excel file from a complex structure table image provided by the above embodiment belong to the same inventive concept, and the technical details not described in the embodiment can be referred to the above embodiment, and the embodiment has the same beneficial effects as the above embodiment.

[0140] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be realized by means of software and necessary universal hardware, and of course can also be realized by hardware. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product can be stored in a computer readable storage medium, such as a floppy disk, a ROM, a RAM, a FLASH, a hard disk or an optical disk, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute the methods of various embodiments of the present application.

Claims

1. A method for converting Excel files into complex structure table images, characterized in that: include: Acquire image data, and perform feature extraction on the image data using a first backbone network to obtain feature information; Performing a first transformation on the feature information to obtain first feature information, and inputting the first feature information into a first branch to obtain positioning information; Based on the positioning information, inputting the first feature information into the second branch to obtain a first sequence; replacing the first sequence according to the positioning information and the first sequence to obtain a first code; Converting the first code using a first tool to obtain an Excel file; Inputting the first feature information into the second branch includes: The first backbone network and the encoding head form a first encoder, and the image is input to the first encoder; Embed the information output by the first encoder into the first vocabulary; Also includes: Performing a first decoding on the information embedded in the first vocabulary to obtain a feature tensor; Map the feature tensor to the vocabulary space through a linear layer to generate the first output sequence; Inputting the first feature information into the first branch includes: Weighting and integrating the key field information of the table to obtain the characteristic data of the key field information; Early fusion of feature data of key domain information and matching of image dimensions; Also includes: Use the convolution kernel to sample the feature data and use the set convolution kernel output and the segmentation map; Obtaining positioning information based on the segmentation map, inputting the positioning information into a first recognition tool, and storing the output text; The first decoding includes: The target sequence is positionally embedded and fused with the output features of the first encoder to obtain the input of the first decoder; The fused feature and mask tensors are input into the first decoder to obtain the feature tensor.

2. The method for converting Excel files into complex structure table images according to claim 1, characterized in that: The first vocabulary embedding includes: Perform word embedding on each word in the target vocabulary, and perform position embedding on the sequence after word embedding; For each position and dimension, if the dimension is even, the value of the position embedding is calculated using the sine function; If the dimension is odd, the cosine function is used to calculate the value of the position embedding; Add the word embedding and position embedding to get the information after the first vocabulary embedding.

3. A system for converting Excel files into complex structure table images, characterized by: include: A feature extraction module acquires image data and performs feature extraction on the image data using the first backbone network to obtain feature information; a transformation module, configured to perform a first transformation on the feature information to obtain first feature information, and input the first feature information into a first branch to obtain positioning information; an input module, configured to input the first feature information into the second branch based on the positioning information to obtain a first sequence; a replacing module, configured to replace the first sequence according to the positioning information and the first sequence to obtain a first code; a conversion module, configured to convert the first code using a first tool to obtain an Excel file; Inputting the first feature information into the second branch includes: The first backbone network and the encoding head form a first encoder, and the image is input to the first encoder; Embed the information output by the first encoder into the first vocabulary; Also includes: Performing a first decoding on the information embedded in the first vocabulary to obtain a feature tensor; Map the feature tensor to the vocabulary space through a linear layer to generate the first output sequence; Inputting the first feature information into the first branch includes: Weighting and integrating the key field information of the table to obtain the characteristic data of the key field information; Early fusion of feature data of key domain information and matching of image dimensions; Also includes: Use the convolution kernel to sample the feature data and use the set convolution kernel output and the segmentation map; Obtaining positioning information based on the segmentation map, inputting the positioning information into a first recognition tool, and storing the output text; The first decoding includes: The target sequence is positionally embedded and fused with the output features of the first encoder to obtain the input of the first decoder; The fused feature and mask tensors are input into the first decoder to obtain the feature tensor.

4. An electronic device comprising: memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method for converting Excel files for complex structure table images as described in any one of claims 1 to 2 are implemented.

5. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the method for converting an Excel file for a complex structure table image as described in any one of claims 1 to 2.

Images