Text image layout analysis method, device, electronic device and storage medium

Abstract

The embodiments of the present application provide a method, device, electronic device, and storage medium for analyzing the layout of a text image. The method for analyzing the layout of a text image includes: utilizing a feature extraction network in a parsing model to extract feature information of the text image to be processed; utilizing a region generation network in a parsing model to obtain the position and type of each layout region contained in the text image to be processed based on the feature information of the text image to be processed; utilizing a feature alignment network in a parsing model to obtain feature information of each layout region based on the feature information of the text image to be processed and the position of each layout region; for each layout region of a preset type, utilizing a content parsing network corresponding to the type of the current layout region in the parsing model to parse and obtain a parsing result of the current layout region based on the feature information of the current layout region. The embodiments of the present application can improve the accuracy of the parsing results, and the layout analysis process is more concise and efficient.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a text image layout analysis method, device, electronic device and storage medium. Background Art

[0002] Layout analysis of text images refers to the identification and analysis of different layout areas on a text image for subsequent OCR (Optical Character Recognition) processing. Layout analysis of text images has important applications in layout restoration.

[0003] Conventional methods for analyzing the layout of text images employ separate region detection and region parsing algorithms for different types of layout regions. For example, text regions require analysis using both a text region detection and a text region parsing algorithm, while table regions require analysis using both a table region detection and a table region parsing algorithm, and so on.

[0004] However, the above-mentioned independent region detection algorithm and region analysis algorithm do not utilize the global collaborative information of the region, resulting in poor analysis effect, and the processing process is complex and inefficient. Summary of the Invention

[0005] In view of the above problems, the embodiments of the present application propose a text image layout analysis method, device, electronic device and storage medium to improve the accuracy and efficiency of text image layout analysis.

[0006] According to a first aspect of an embodiment of the present application, a text image layout analysis method is provided, comprising:

[0007] Utilize the feature extraction network in the parsing model to extract the feature information of the text image to be processed;

[0008] Utilizing the region generation network in the analytical model, based on the feature information of the text image to be processed, to obtain the position and type of each layout region contained in the text image to be processed;

[0009] Utilizing a feature alignment network in the parsing model, based on the feature information of the text image to be processed and the position of each layout area, acquiring feature information of each layout area;

[0010] For each layout area of ​​a preset type, a content parsing network corresponding to the type of the current layout area in the parsing model is used to parse and obtain a parsing result of the current layout area based on feature information of the current layout area.

[0011] Optionally, the parsing model is trained in the following manner:

[0012] Utilize the feature extraction network in the parsing model to be trained to extract feature information of the sample text image;

[0013] Utilizing the region generation network in the to-be-trained parsing model, based on the feature information of the sample text image, to obtain the position and type of each sample layout region contained in the sample text image;

[0014] Utilizing the feature alignment network in the to-be-trained parsing model, based on the feature information of the sample text image and the position of each sample layout area, acquiring feature information of each sample layout area;

[0015] For each sample layout area of ​​a preset type, using the content parsing network corresponding to the type of the current sample layout area in the to-be-trained parsing model, parsing based on feature information of the current sample layout area to obtain a parsing result for the current sample layout area;

[0016] A model loss function is calculated, and whether training is complete is determined based on the model loss function.

[0017] Optionally, calculating the model loss function includes: calculating the loss function of the region generation network and the loss function of each content parsing network; and calculating the model loss function based on the loss function of the region generation network and the loss function of each content parsing network.

[0018] Optionally, the model loss function is calculated based on the loss function of the region generation network and the loss functions of each content parsing network, including: performing weighted calculation on the loss function of the region generation network and the loss function of each content parsing network to obtain the model loss function.

[0019] Optionally, the preset type includes at least one of the following: a table area, a text area, and a formula area.

[0020] According to a second aspect of an embodiment of the present application, there is provided a text image layout analysis device, comprising:

[0021] An extraction module is used to extract feature information of the text image to be processed using the feature extraction network in the parsing model;

[0022] A generation module, configured to utilize a region generation network in the parsing model to obtain the position and type of each layout region contained in the text image to be processed based on feature information of the text image to be processed;

[0023] an alignment module, configured to utilize a feature alignment network in the parsing model to obtain feature information of each layout area based on the feature information of the text image to be processed and the position of each layout area;

[0024] The parsing module is used to parse each layout area of ​​a preset type using a content parsing network corresponding to the type of the current layout area in the parsing model and obtain a parsing result of the current layout area based on feature information of the current layout area.

[0025] Optionally, the parsing model is trained modularly in the following manner:

[0026] The sample extraction module is used to extract feature information of the sample text image using the feature extraction network in the parsing model to be trained;

[0027] a sample generation module, configured to utilize the region generation network in the to-be-trained parsing model to obtain the position and type of each sample layout region contained in the sample text image based on feature information of the sample text image;

[0028] A sample alignment module, configured to utilize a feature alignment network in the to-be-trained parsing model to obtain feature information of each sample layout region based on the feature information of the sample text image and the position of each sample layout region;

[0029] A sample parsing module is configured to parse each sample layout area of ​​a preset type using a content parsing network corresponding to the type of the current sample layout area in the to-be-trained parsing model and obtaining a parsing result of the current sample layout area based on feature information of the current sample layout area;

[0030] The determination module is used to calculate the model loss function and determine whether the training is completed based on the model loss function.

[0031] Optionally, the determination module includes: a first calculation unit, used to calculate the loss function of the region generation network and the loss function of each content parsing network; a second calculation unit, used to calculate the model loss function based on the loss function of the region generation network and the loss function of each content parsing network.

[0032] Optionally, the second calculation unit is specifically configured to perform weighted calculation on the loss function of the region generation network and the loss functions of each content parsing network to obtain the model loss function.

[0033] Optionally, the preset type includes at least one of the following: a table area, a text area, and a formula area.

[0034] According to the third aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; and one or more computer-readable storage media having instructions stored thereon; when the instructions are executed by the one or more processors, the processors execute the text image layout analysis method as described in any one of the above items.

[0035] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the processor executes the text image layout analysis method as described in any one of the above items.

[0036] In the embodiment of the present application, the feature extraction network in the parsing model is used to extract the feature information of the text image to be processed; the region generation network in the parsing model is used to obtain the position and type of each layout area contained in the text image to be processed based on the feature information of the text image to be processed; the feature alignment network in the parsing model is used to obtain the feature information of each layout area based on the feature information of the text image to be processed and the position of each layout area; for each layout area of ​​the preset type, the content parsing network corresponding to the type of the current layout area in the parsing model is used to parse and obtain the parsing result of the current layout area based on the feature information of the current layout area. It can be seen from this that the embodiment of the present application can be trained end-to-end, effectively utilizing the shared feature extraction network and global collaborative information, so that the boundary division of the layout area is more accurate, thereby improving the accuracy of the parsing result, and the text image layout analysis process is more concise and efficient, which can improve the customer experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some drawings of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0038] Figure 1 This is a flowchart of the steps of the parsing model training process in an embodiment of the present application.

[0039] Figure 2 This is a flowchart of the steps of a text image layout analysis method according to an embodiment of the present application.

[0040] Figure 3 This is a structural block diagram of a text image layout analysis device according to an embodiment of the present application.

[0041] Figure 4It is a structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, rather than all the embodiments of this application. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0043] Layout analysis of text images refers to obtaining the attributes and analysis results of different layout areas in text images.

[0044] For example, the properties of a layout area may include but are not limited to: position, type, and the like.

[0045] For example, the types of layout areas may include, but are not limited to, text areas, table areas, formula areas, image areas, etc. The text area can be further divided into body areas, title areas, list areas, header and footer areas, etc. The table area can be further divided into wired table areas, wireless table areas, and mixed table areas, etc.

[0046] For example, the parsing results for a layout area may include, but are not limited to, content. Specifically, the parsing results for a text area may include, but are not limited to, the text content. The parsing results for a table area may include, but are not limited to, the position coordinates and logical row and column coordinates of each cell within the table area, as well as the content of each cell. The parsing results for a formula area may include, but are not limited to, the content of the formula. Image areas do not require further parsing of the image content.

[0047] In the embodiment of the present application, a parsing model is pre-trained, and then the pre-trained parsing model is used to collaboratively parse the text image, eliminating the need to use independent algorithms corresponding to each layout area to parse each layout area separately, making the processing process simpler and more efficient.

[0048] First, the training process of the parsing model is introduced.

[0049] Build a parsing model to be trained. A parsing model to be trained refers to a parsing model that has not been trained yet.

[0050] The parsing model to be trained may include, but is not limited to, a feature extraction network, a region generation network, a feature alignment network, and content parsing networks corresponding to various preset types. The "type" refers to the type of layout area within a text image. For example, the preset types may include, but are not limited to, at least one of the following: table area, text area, and formula area. Since image areas do not require content parsing, corresponding content parsing networks may not be required.

[0051] Obtain sample data. The sample data may include a sample text image and a label of the sample text image, wherein the label of the sample text image is used to indicate the actual position, actual type, and actual parsing result of each sample layout area contained in the sample text image.

[0052] For example, a large number of text images can be captured from the Internet as sample text images. For each sample text image, a Word parsing tool, a PDF parsing tool, etc. can be used to obtain the actual position, actual type, and actual parsing results of each layout area contained in the sample text image, so as to obtain a label for the sample text image.

[0053] In an optional implementation, all acquired sample data may be used to train the to-be-trained parsing model.

[0054] Reference Figure 1 , showing a flowchart of the steps of the parsing model training process in an embodiment of the present application.

[0055] like Figure 1 As shown, the parsing model training process may include the following steps:

[0056] Step 101: Extract feature information of a sample text image using a feature extraction network in a parsing model to be trained.

[0057] During the training process, the sample text image is input into the feature extraction network in the parsing model to be trained.

[0058] The feature extraction network is used to extract feature information from text images. Therefore, by using the feature extraction network in the training parsing model to perform operations such as convolution and pooling on the input sample text image, the feature information of the sample text image can be extracted. This feature information is shared feature information.

[0059] For example, the feature extraction network can employ the backbone network of CNN (Convolutional Neural Networks). The backbone is primarily provided at the front end of the CNN network. The backbone may include, but is not limited to, ResNet (Residual Network), VGGNet (Visual Geometry Group Network), DenseNet (Dense Convolutional Network), and the like.

[0060] The specific structure of the feature extraction network and the feature extraction process can be processed according to actual experience, and will not be discussed in detail in this embodiment.

[0061] Step 102: utilizing the region generation network in the to-be-trained parsing model to obtain the position and type of each sample layout region contained in the sample text image based on the feature information of the sample text image.

[0062] The feature information of the sample text image output by the feature extraction network in the parsing model to be trained is used as the input of the region generation network in the parsing model to be trained.

[0063] The region generation network is used to obtain the location and type of each layout region contained in the text image. Therefore, by using the region generation network in the training parsing model to perform regional analysis on the feature information of the input sample text image, the location and type of each sample layout region contained in the sample text image can be obtained. The location of the sample layout region can be the coordinates of the sample layout region in the sample text image. The location and type of each sample layout region serve as the output of the training parsing model.

[0064] For example, the region proposal network may be an RPN (Region Proposal Network).

[0065] Specifically, the feature information input to the RPN can be a feature map. The processing process of RPN may include: first, the RPN generates anchor boxes (Anchors) with different scales and aspect ratios for each point on the feature map (called an anchor point); then, the RPN inputs these anchor boxes (the anchor boxes can correspond to the sample layout area in the embodiment of the present application) into two network layers, one network layer is used for classification, that is, to determine whether the feature map in the anchor box belongs to the target (the target can correspond to the type of the sample layout area in the embodiment of the present application); the other network layer outputs the position coordinates of the anchor box (the position coordinates can correspond to the position of the sample layout area in the embodiment of the present application). Among them, the position coordinates (x, y, w, h) of the anchor box refer to the coordinates on the original image.

[0066] Step 103 , utilizing the feature alignment network in the to-be-trained parsing model, based on the feature information of the sample text image and the position of each sample layout area, obtains the feature information of each sample layout area.

[0067] The feature information of the sample text image output by the feature extraction network in the to-be-trained parsing model and the positions of the sample layout areas contained in the sample text image output by the area generation network in the to-be-trained parsing model are used as the input of the feature alignment network in the to-be-trained parsing model.

[0068] The feature alignment network is used to align the position of the layout area with the feature information of the text image in order to obtain the feature information of the layout area. Therefore, using the feature alignment network in the training parsing model, the feature information of the sample text image and the position of each sample layout area contained in the sample text image can be aligned to obtain the feature information of each sample layout area contained in the sample text image.

[0069] Exemplarily, for any sample layout area, based on the position of the sample layout area, feature information corresponding to the position of the sample layout area is extracted from the feature information of the sample text image, and the extracted feature information is used as the feature information of the sample layout area.

[0070] For example, the feature alignment network may adopt a ROI Align (Region of Interest Align) network or the like.

[0071] Specifically, the concept behind ROI Align is to eliminate quantization and use bilinear interpolation to obtain image values ​​at floating-point pixel coordinates, thereby transforming the entire feature aggregation process into a continuous operation. The process includes: traversing each candidate region, maintaining the floating-point boundaries without quantization; dividing the candidate region into k×k cells, with the boundaries of each cell also unquantized; calculating four fixed coordinates within each cell, using bilinear interpolation to calculate the values ​​of these four coordinates, and then performing a max pooling operation. These fixed positions are defined according to a fixed rule within each rectangular cell. For example, if the number of sampling points is 1, then the center point of the cell is used. If the number of sampling points is 4, then the center points of the four smaller squares after the cell is evenly divided. Obviously, the coordinates of these sampling points are usually floating-point numbers, so interpolation is required to obtain their pixel values.

[0072] Step 104 , for each sample layout area of ​​a preset type, using the content parsing network corresponding to the type of the current sample layout area in the parsing model to be trained, parses and obtains a parsing result of the current sample layout area based on the feature information of the current sample layout area.

[0073] Based on the type of each sample layout area output by the region generation network in the to-be-trained parsing model, it is determined whether each sample layout area is of a preset type. For each sample layout area of ​​the preset type, the feature information of the current sample layout area output by the feature alignment network in the to-be-trained parsing model is used as input to the content parsing network corresponding to the type of the current sample layout area in the to-be-trained parsing model.

[0074] The content parsing network is used to parse the content of a layout area in order to obtain a parsing result for that layout area. Therefore, for the current sample layout area, the content parsing network corresponding to the current sample layout area in the parsing model to be trained can be used to parse the feature information of the current sample layout area, thereby obtaining a parsing result for the current sample layout area. The parsing result of each sample layout area serves as the output of the parsing model to be trained.

[0075] Exemplarily, the content parsing network may adopt a Transformer network or the like.

[0076] The Transformer network consists of multiple encoders and multiple decoders, and the number of encoders and decoders is the same. The encoder includes a self-attention layer and a feed-forward neural network layer. The sentence input from the encoder will first pass through the self-attention layer. This layer pays attention to other words in the input sentence when encoding each word, and the output of the self-attention layer will be passed to the feed-forward neural network layer for processing. The decoder also has the encoder's self-attention layer and feed-forward neural network layer. In addition, there is an attention layer between these two layers to pay attention to the relevant parts of the input sentence. For the specific processing process, this embodiment will not be discussed in detail here.

[0077] Step 105: Calculate the model loss function, and determine whether the training is complete based on the model loss function.

[0078] In an optional implementation, the process of calculating the model loss function may include the following steps A1-A2:

[0079] Step A1: Calculate the loss function of the region generation network and the loss function of each content parsing network respectively.

[0080] The loss function of the region generation network can be calculated based on the position of the sample layout area output by the region generation network and the actual position of the sample layout area in the label, the type of the sample layout area output by the training parsing model and the actual type of the sample layout area in the label.

[0081] The loss function of any content parsing network can be calculated based on the parsing result of the sample layout area output by the content parsing network and the actual parsing result of the sample layout area in the label.

[0082] Exemplarily, the loss function may include but is not limited to: a cross entropy loss function, an exponential loss function, a square loss function, an absolute value loss function, and the like.

[0083] Step A2: Calculate the model loss function based on the loss function of the region generation network and the loss functions of each content parsing network.

[0084] For example, the process of calculating the model loss function based on the loss function of the region generation network and the loss functions of each content parsing network may include performing a weighted calculation on the loss function of the region generation network and the loss functions of each content parsing network to obtain the model loss function. The weights of the loss functions in the weighted calculation may be set based on actual circumstances and are not limited in this embodiment.

[0085] Exemplarily, the process of calculating the model loss function based on the loss function of the region generation network and the loss functions of each content parsing network may include: summing the loss function of the region generation network and the loss functions of each content parsing network to obtain the model loss function.

[0086] If the model loss function is less than a preset threshold, training can be determined to be complete. If training is determined to be complete based on the model loss function, the trained model can be used as the parsing model. If training is determined to be incomplete based on the model loss function, an appropriate optimizer is used to perform end-to-end network optimization on the parameters of the region generation network and each content parsing network, and training continues.

[0087] In an optional embodiment, a portion of the sample data can be selected from all the sample data obtained as a training set, and the remaining sample data can be selected as a test set. The training set is used to train the parsing model to be trained, and the test set is used to test and evaluate the trained parsing model.

[0088] Exemplarily, the number of sample data in the training set is greater than the number of sample data in the test set. Exemplarily, the percentage of the number of sample data in the training set to the total sample data is greater than or equal to a preset percentage. Optionally, the preset percentage may be 80%, 85%, 90%, etc. For example, if the total number of sample data is 500,000, the number of sample data in the training set may be 495,000, and the number of sample data in the test set may be 5,000.

[0089] In this case, after training is completed based on the model loss function, the trained parsing model can be further tested and evaluated. If the evaluation passes, the parsing model is obtained. If the evaluation fails, the parameters of the trained parsing model are further optimized.

[0090] For example, the accuracy of the analytical model is calculated based on the test results. The accuracy rate is the ratio of the number of samples with correct predictions divided by the total number of samples. The accuracy rate reflects the model's ability to make correct predictions. Therefore, the higher the accuracy rate, the greater the likelihood that the prediction model will make correct predictions. Therefore, if the calculated accuracy rate is greater than the preset accuracy rate, the evaluation can be determined to have passed.

[0091] Then, the process of text image layout analysis is introduced.

[0092] Reference Figure 2 , shows a step flow chart of a text image layout analysis method according to an embodiment of the present application.

[0093] like Figure 2 As shown, the text image layout analysis method may include the following steps:

[0094] Step 201: Using the feature extraction network in the parsing model, extract feature information of the text image to be processed.

[0095] Input the text image to be processed into the feature extraction network of the trained parsing model.

[0096] By using the feature extraction network in the parsing model to perform convolution, pooling and other operations on the input text image to be processed, feature information of the text image to be processed can be extracted. The feature information here is shared feature information.

[0097] Step 202 : utilizing the region generation network in the analytical model to obtain the position and type of each layout region contained in the text image to be processed based on the feature information of the text image to be processed.

[0098] The feature information of the text image to be processed output by the feature extraction network in the parsing model is used as the input of the region generation network in the parsing model.

[0099] The region generation network in the parsing model performs region analysis on the feature information of the input text image to be processed, and can obtain the location and type of each layout region contained in the text image to be processed. The location of the layout region can be the coordinates of the layout region in the text image to be processed. The location and type of each layout region serve as the output of the parsing model.

[0100] Step 203 : utilizing the feature alignment network in the parsing model, based on the feature information of the text image to be processed and the position of each layout area, obtain the feature information of each layout area.

[0101] The feature information of the text image to be processed output by the feature extraction network in the parsing model and the positions of each layout area contained in the text image to be processed output by the region generation network in the parsing model are used as inputs of the feature alignment network in the parsing model.

[0102] By utilizing the feature alignment network in the parsing model, the feature information of the text image to be processed and the positions of the various layout areas contained in the text image to be processed can be aligned, thereby obtaining the feature information of the various layout areas contained in the text image to be processed.

[0103] Step 204 : for each layout area of ​​the preset type, using the content parsing network corresponding to the type of the current layout area in the parsing model, and based on the feature information of the current layout area, obtain a parsing result of the current layout area.

[0104] Based on the type of each layout region output by the region generation network in the parsing model, it is determined whether each layout region is of a preset type. For each layout region of the preset type, the feature information of the current layout region output by the feature alignment network in the parsing model is used as input to the content parsing network corresponding to the type of the current layout region in the parsing model.

[0105] For the current layout area, the content parsing network corresponding to the current layout area in the parsing model can be used to parse the feature information of the current layout area, thereby obtaining the parsing result of the current layout area. The parsing result of each sample layout area is used as the output of the parsing model.

[0106] The embodiment of the present application can be trained end-to-end, effectively utilizing a shared feature extraction network and global collaborative information, so that the boundary division of the layout area is more precise, thereby improving the accuracy of the analysis results, and the text image layout analysis process is more concise and efficient, which can enhance the customer experience.

[0107] Reference Figure 3 , shows a structural block diagram of a text image layout analysis device according to an embodiment of the present application.

[0108] like Figure 3 As shown, the text image layout analysis device may include the following modules:

[0109] Extraction module 301, used to extract feature information of the text image to be processed using the feature extraction network in the parsing model;

[0110] A generation module 302 is configured to utilize the region generation network in the analytical model to obtain the position and type of each layout region contained in the text image to be processed based on the feature information of the text image to be processed;

[0111] An alignment module 303 is configured to utilize a feature alignment network in the parsing model to obtain feature information of each layout region based on the feature information of the text image to be processed and the position of each layout region;

[0112] The parsing module 304 is used to parse each layout area of ​​a preset type using the content parsing network corresponding to the type of the current layout area in the parsing model and based on the feature information of the current layout area to obtain a parsing result of the current layout area.

[0113] Optionally, the parsing model is trained modularly in the following manner:

[0114] The sample extraction module is used to extract feature information of the sample text image using the feature extraction network in the parsing model to be trained;

[0115] a sample generation module, configured to utilize the region generation network in the to-be-trained parsing model to obtain the position and type of each sample layout region contained in the sample text image based on feature information of the sample text image;

[0116] A sample alignment module, configured to utilize a feature alignment network in the to-be-trained parsing model to obtain feature information of each sample layout region based on the feature information of the sample text image and the position of each sample layout region;

[0117] A sample parsing module is configured to parse each sample layout area of ​​a preset type using a content parsing network corresponding to the type of the current sample layout area in the to-be-trained parsing model and obtaining a parsing result of the current sample layout area based on feature information of the current sample layout area;

[0118] The determination module is used to calculate the model loss function and determine whether the training is completed based on the model loss function.

[0119] Optionally, the determination module includes: a first calculation unit, used to calculate the loss function of the region generation network and the loss function of each content parsing network; a second calculation unit, used to calculate the model loss function based on the loss function of the region generation network and the loss function of each content parsing network.

[0120] Optionally, the second calculation unit is specifically configured to perform weighted calculation on the loss function of the region generation network and the loss functions of each content parsing network to obtain the model loss function.

[0121] Optionally, the preset type includes at least one of the following: a table area, a text area, and a formula area.

[0122] The embodiment of the present application can be trained end-to-end, effectively utilizing a shared feature extraction network and global collaborative information, so that the boundary division of the layout area is more precise, thereby improving the accuracy of the analysis results, and the text image layout analysis process is more concise and efficient, which can enhance the customer experience.

[0123] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0124] In an embodiment of the present application, an electronic device is further provided. The electronic device may include one or more processors and one or more computer-readable storage media storing instructions, such as application programs. When the instructions are executed by the one or more processors, the processors execute the text image layout analysis method described in any of the above embodiments.

[0125] Reference Figure 4 , shows a schematic diagram of an electronic device structure according to an embodiment of the present application. Figure 4 As shown, the electronic device includes a processor 401, a communication interface 402, a memory 403 and a communication bus 404. The processor 401, the communication interface 402 and the memory 403 communicate with each other via the communication bus 404.

[0126] The memory 403 is used to store computer programs.

[0127] The processor 401 is configured to implement the text image layout analysis method of any of the above embodiments when executing the program stored in the memory 403 .

[0128] The communication interface 402 is used for communication between the electronic device and other devices.

[0129] The communication bus 404 mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0130] The processor 401 mentioned above may include but is not limited to: a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0131] The memory 403 mentioned above may include but is not limited to: Read Only Memory (ROM), Random Access Memory (RAM), Compact Disc Read Only Memory (CD-ROM), Electronic Erasable Programmable Read Only Memory (EEPROM), hard disk, floppy disk, flash memory, etc.

[0132] In an embodiment of the present application, a non-temporary computer-readable storage medium is also provided, on which a computer program is stored. The computer program can be executed by a processor of an electronic device. When the computer program is executed by the processor, the processor executes the text image layout analysis method described in any of the above embodiments.

[0133] It should be noted that all actions of acquiring signals, information or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0134] The algorithm provided herein and demonstration are not inherently relevant to any particular computer, virtual system or other equipment. Various general purpose systems can also be used together with based on the teachings herein. According to the above description, it is apparent that the structure required for constructing this type of system. In addition, the embodiments of the present application are not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the embodiments of the present application described herein, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the embodiments of the present application.

[0135] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0136] Similarly, it should be understood that in order to streamline the present application and aid in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the embodiments of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the intention that the claimed embodiments of the present application require more features than those explicitly recited in each claim. More precisely, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims following the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the embodiments of the present application.

[0137] Those skilled in the art will appreciate that the modules in the devices of the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and further may be divided into a plurality of submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device so disclosed may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0138] The various component embodiments of the embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components of the device for generating dynamic images according to the embodiments of the present application. The embodiments of the present application can also be implemented as a device or apparatus program for performing part or all of the methods described herein. Such a program implementing the embodiments of the present application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0139] It should be noted that the above embodiments illustrate rather than limit the embodiments of the present application, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbols placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The embodiments of the present application may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

[0140] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0141] The above is only a specific implementation of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited to this. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in the embodiments of the present application, and they should all be covered by the protection scope of the embodiments of the present application.

Claims

1. A text image layout analysis method, characterized in that: The method comprises: Utilize the feature extraction network in the parsing model to extract the shared feature information of the text image to be processed; Utilizing the region generation network in the analytical model, based on the shared feature information of the text image to be processed, to obtain the position and type of each layout region contained in the text image to be processed; Utilizing the feature alignment network in the parsing model, based on the shared feature information of the text image to be processed and the positions of the respective layout regions, acquiring feature information of the respective layout regions, including: extracting feature information corresponding to the position of the text image to be processed from the shared feature information of the text image to be processed according to the positions of the respective layout regions, and using the extracted feature information corresponding to the position of the text image to be processed as the feature information of the text image to be processed; For each layout area of ​​a preset type, utilizing the content parsing network corresponding to the type of the current layout area in the parsing model, based on the feature information of the current layout area, a parsing result of the current layout area is obtained; the parsing result of the current layout area includes at least one of a parsing result of a text area, a parsing result of a table area, and a parsing result of a formula area; the parsing result of the text area includes the content of the text; the parsing result of the table area includes the position coordinates and logical row and column coordinates of each cell in the table area, as well as the content of each cell; the parsing result of the formula area includes the content of the formula; Wherein, the region generation network is RPN; The feature alignment network is a ROIAlign network.

2. The method according to claim 1, characterized in that The parsing model is trained as follows: Utilize the feature extraction network in the parsing model to be trained to extract feature information of the sample text image; Utilizing the region generation network in the to-be-trained parsing model, based on the feature information of the sample text image, to obtain the position and type of each sample layout region contained in the sample text image; Utilizing the feature alignment network in the to-be-trained parsing model, based on the feature information of the sample text image and the position of each sample layout area, acquiring feature information of each sample layout area; For each sample layout area of ​​a preset type, using the content parsing network corresponding to the type of the current sample layout area in the to-be-trained parsing model, parsing based on feature information of the current sample layout area to obtain a parsing result for the current sample layout area; A model loss function is calculated, and whether training is complete is determined based on the model loss function.

3. The method according to claim 2, characterized in that Calculating the model loss function includes: Calculating the loss function of the region generation network and the loss function of each content parsing network; The model loss function is calculated based on the loss function of the region generation network and the loss functions of each content parsing network.

4. The method according to claim 3, characterized in that Calculating the model loss function based on the loss function of the region generation network and the loss functions of each content parsing network includes: The loss function of the region generation network and the loss functions of each content parsing network are weightedly calculated to obtain the model loss function.

5. The method according to claim 1, wherein The preset type includes at least one of the following: a table area, a text area, and a formula area.

6. A text image layout analysis device, characterized in that: The device comprises: An extraction module, configured to extract shared feature information of the text image to be processed using a feature extraction network in the parsing model; A generation module, configured to utilize a region generation network in the parsing model to obtain the position and type of each layout region contained in the text image to be processed based on the shared feature information of the text image to be processed; an alignment module, configured to utilize a feature alignment network in the parsing model to obtain feature information of each layout area based on the shared feature information of the text image to be processed and the position of each layout area, including: extracting feature information corresponding to the position of the text image to be processed from the shared feature information of the text image to be processed according to the position of each layout area, and using the extracted feature information corresponding to the position of the text image to be processed as the feature information of the text image to be processed; A parsing module is configured to, for each layout area of ​​a preset type, utilize a content parsing network corresponding to the type of the current layout area in the parsing model, and parse the current layout area based on feature information of the current layout area to obtain a parsing result for the current layout area; the parsing result for the current layout area includes at least one of a parsing result for a text area, a parsing result for a table area, and a parsing result for a formula area; the parsing result for the text area includes the content of the text; the parsing result for the table area includes the position coordinates and logical row and column coordinates of each cell in the table area, as well as the content of each cell; and the parsing result for the formula area includes the content of the formula; Wherein, the region generation network is RPN; The feature alignment network is a ROIAlign network.

7. The device according to claim 6, characterized in that The parsing model is trained modularly in the following way: The sample extraction module is used to extract feature information of the sample text image using the feature extraction network in the parsing model to be trained; a sample generation module, configured to utilize the region generation network in the to-be-trained parsing model to obtain the position and type of each sample layout region contained in the sample text image based on feature information of the sample text image; A sample alignment module, configured to utilize a feature alignment network in the to-be-trained parsing model to obtain feature information of each sample layout region based on the feature information of the sample text image and the position of each sample layout region; A sample parsing module is configured to parse each sample layout area of ​​a preset type using a content parsing network corresponding to the type of the current sample layout area in the to-be-trained parsing model and obtaining a parsing result of the current sample layout area based on feature information of the current sample layout area; The determination module is used to calculate the model loss function and determine whether the training is completed based on the model loss function.

8. The device according to claim 7, characterized in that The determination module includes: A first calculation unit, configured to calculate the loss function of the region generation network and the loss functions of each content parsing network; The second calculation unit is used to calculate the model loss function based on the loss function of the region generation network and the loss functions of each content parsing network.

9. An electronic device, characterized in that: include: one or more processors; and one or more computer-readable storage media having instructions stored thereon; When the instructions are executed by the one or more processors, the processors are caused to execute the text image layout analysis method according to any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the processor is caused to execute the text image layout analysis method according to any one of claims 1 to 5.

Images